Q: Linux Disk 100% Cleanup: How do you identify root cause, unlinked deleted files, and execute safe emergency remediation?
Production runbook for recovering from 100% disk utilization: inode exhaustion, du vs df discrepancies, the unlinked open file descriptor trap (lsof deleted), and safe log truncation.
Disk 100% Full After Deleting 40GB? The lsof Trap That Frees It | Linux Interview
Watch Naveed break down what happens when a Linux disk remains 100% full after deleting massive log files: identifying unlinked open file descriptors with lsof, safely truncating file descriptors via /proc/<pid>/fd, and preventing production outages in 60 seconds.
- Unlinked Open Files: Why rm /var/log/app.log fails to free disk space when a process holds the descriptor
- Triage Command: Use lsof +L1 or lsof | grep deleted to isolate active processes locking deleted files
- Zero Out Safely: Truncate open files directly via > /proc/<pid>/fd/<fd_num> without restarting services
- Prevent Outages: Configure logrotate copytruncate to avoid open file descriptor leaks
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Linux Disk 100% Cleanup Emergency Triage Runbook
Direct Answer: A Linux Disk 100% Cleanup emergency requires identifying whether disk consumption is driven by unlinked open file descriptors (deleted files held open by running processes) or runaway log files. Triage immediately using lsof +L1 to locate unlinked space-consuming processes and truncate active logs using > /var/log/file.log instead of deleting them.
| Triage Phase | Symptom & Observation | Root Cause | CLI Verification Command | Permanent Remediation |
|---|---|---|---|---|
| 1. Space vs Inode Audit | df -h shows 100% used, or No space left on device | Disk block saturation or inode exhaustion | df -h && df -i | Check both storage capacity and inode percentage |
| 2. Unlinked Open Files | Files deleted but disk space remains at 100% | Running process holds file descriptor open | lsof +L1 | Restart the holding process or truncate via /proc/<pid>/fd |
| 3. Runaway Directory Audit | Large log files consuming root filesystem | Application verbose debug logging | du -ahx / | sort -rh | head -20 | Truncate logs with true > file.log and configure logrotate |
Check Mounts and Inodes (df -h & df -i)
Identify which filesystem is full and whether it is bytes or inodes:
df -h: Identify which partition is at 100% (e.g./,/var, or/tmp).- The Inode Trap (
df -i): Ifdf -hshows 40% free space but applications report 'No space left on device', you have run out of Inodes! Millions of tiny 0-byte files (like session files in/var/spoolor PHP sessions) exhaust inodes even when gigabytes of disk remain.
Locate Largest Directories and Files (du)
Scan the filesystem safely without crossing mount boundaries:
du -ahx / | sort -rh | head -20: Lists top 20 space-consuming files/directories.- Why
-xflag is critical: The-xflag stays on one filesystem. Without it,duscans NFS mounts, Docker volumes, and virtual filesystems (/proc), crashing or taking hours. - Quick check on common culprits:
du -sh /var/log/* /var/lib/docker/* | sort -h.
The 'Deleted File Held Open' Trap (lsof +L1)
The #1 issue senior SREs look for:
- An engineer deletes a 40GB log file using
rm /var/log/app.log. Butdf -hSTILL shows 100% full! Why? - Linux Filesystem Mechanics: If a running process (NGINX, Java, database) has the file open, deleting the file removes the directory directory entry, but the OS cannot free the disk blocks until the file descriptor is closed!
- Command to identify:
lsof +L1orlsof | grep deleted. - How to fix without restarting service: Truncate the open file descriptor directly via
/proc:> /proc/<PID>/fd/<FD_NUMBER>(instantly frees the disk blocks without restarting the app!).
Safe Cleanup Best Practices
How to free space immediately without breaking services:
- Never rm an active log file: Always truncate:
truncate -s 0 /var/log/app.logor> /var/log/app.log. - Vacuum systemd journal logs:
journalctl --vacuum-time=2dorjournalctl --vacuum-size=500M. - Clean Docker leftovers:
docker system prune -af --volumes(verify unused volumes first). - Permanent fix: Configure
logrotatewith compression and retention limits.
- Check blocks & inodes: 'df -h' for disk space, 'df -i' for inode exhaustion (millions of tiny files).
- Find large files: 'du -ahx / | sort -rh | head -20' (use -x to avoid traversing other mounts/NFS).
- Check deleted files held open: 'lsof +L1' or 'lsof | grep deleted' for deleted files still locked by active processes.
- Fix open deleted files: Truncate via '> /proc/<PID>/fd/<FD>' to free disk space immediately without restarting process.
- Clean safely: Truncate logs (truncate -s 0 file.log) rather than 'rm'; run 'journalctl --vacuum-size=500M'.
- Prevent: Set up logrotate, CloudWatch disk space alarm at 80%, and automated cleanup jobs.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Mistaking high Linux load average caused by uninterruptible I/O wait for compute CPU saturation
Load average includes processes in uninterruptible disk sleep (
Dstate); high load with low CPU indicates disk I/O bottlenecks or NFS hangs rather than compute saturation. - Failing to check kernel ring buffer logs (dmesg -T) and journalctl during unexpected service crashes when addressing deleted Systemd service exits and kernel OOM reaper events are logged directly to the kernel ring buffer and systemd journal rather than application log files.
- Diagnosing infrastructure outages unsystematically without adhering to the USE or RED methodologies Senior Linux engineers systematically evaluate Utilization, Saturation, and Errors across compute, memory, and disk rather than making random configuration changes.
-
Applying kernel runtime sysctl parameters on the fly without persisting them to /etc/sysctl.d/
Applying runtime sysctl adjustments via
sysctl -wwithout configuration persistence causes critical tuning to vanish upon server reboot.