Q: A catastrophic production incident occurred on an active database node: the root filesystem reached 100% capacity. SREs ran `du -sh /var/lib/docker` which reported only 18GB used, yet `df -h` showed 180GB used (a 162GB discrepancy!). Running `docker system prune` reclaimed zero space. Meanwhile, containers began failing with I/O errors and databases entered read-only mode. You must author a postmortem root-cause analysis, explain the Linux filesystem discrepancy, and create an automated SRE operational triage runbook.

Understand Linux VFS mechanics: `du` (disk usage) traverses directory trees and calculates file sizes based on visible filenames. `df` (disk free) queries the filesystem superblock for allocated disk blocks. When a file is deleted (`rm`) while an active process still holds its file descriptor open, its directory entry is removed (invisible to `du`), but the kernel cannot free the disk blocks until the process closes the file descriptor or terminates.

<!-- du vs df Discrepancy Mechanics -->
Process (Log Writer) ───[ Holds open File Descriptor (FD) ]───> Inode #918237 (160GB)
                                                                       │
Administrator executes: rm /var/lib/docker/containers/.../app.log     │
                                                                       │
Result:
- Directory link removed -> du cannot see it! (du reports 18GB)
- Inode blocks NOT freed -> df reports 180GB allocated! (100% Full!)

Run `lsof +L1` to find open file descriptors on deleted files holding massive disk capacity.

# Find unlinked open files (> 1GB) holding disk space
sudo lsof +L1 /var/lib/docker | grep -E 'deleted|CONTAINER' | sort -nk 7 -r | head -n 10
# Sample output:
# COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK   NODE NAME
# java     4120 root    4w   REG  259,1 171798691840     0 918237 /var/lib/docker/.../app.log (deleted)

To immediately free the disk blocks without restarting or terminating the running container, truncate the file descriptor directly via `/proc//fd/`.

# Truncate unlinked file descriptor to 0 bytes
sudo truncate -s 0 /proc/4120/fd/4

# Verify disk space is instantly reclaimed in df
df -h /var/lib/docker
# Output: Available: 162GB (Discrepancy resolved instantly!)

Deploy an automated Prometheus alert tracking the delta between `node_filesystem_size_bytes - node_filesystem_free_bytes` and `du` scans. Add automated log truncation and logrotate policies.

# Prometheus Alert Rule
# expr: (node_filesystem_size_bytes{mountpoint="/var/lib/docker"} - node_filesystem_free_bytes{mountpoint="/var/lib/docker"}) / node_filesystem_size_bytes{mountpoint="/var/lib/docker"} > 0.85
# for: 5m
# labels:
#   severity: critical
# annotations:
#   summary: "Runaway Docker disk consumption detected! Check unlinked open files via lsof +L1."