⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 2 of 135 in Linux & SRE
Senior DevOps / SRE Linux Storage & Filesystems Production Incident

Q: Linux Disk 100% Cleanup: How do you identify root cause, unlinked deleted files, and execute safe emergency remediation?

Production runbook for recovering from 100% disk utilization: inode exhaustion, du vs df discrepancies, the unlinked open file descriptor trap (lsof deleted), and safe log truncation.

#Linux #df #du #lsof #Deleted Files Trap #Inodes #Storage
🎙️ Candidate Opening & Architectural Context
"When a disk hits 100%, services begin dropping connections, crash, and cannot write logs or create PID files. The investigation requires checking both disk blocks and inodes, and avoiding the classic 'deleted file still held open' trap."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Linux Disk 100% Cleanup Emergency Triage Runbook

Direct Answer: A Linux Disk 100% Cleanup emergency requires identifying whether disk consumption is driven by unlinked open file descriptors (deleted files held open by running processes) or runaway log files. Triage immediately using lsof +L1 to locate unlinked space-consuming processes and truncate active logs using > /var/log/file.log instead of deleting them.

Triage Phase Symptom & Observation Root Cause CLI Verification Command Permanent Remediation
1. Space vs Inode Audit df -h shows 100% used, or No space left on device Disk block saturation or inode exhaustion df -h && df -i Check both storage capacity and inode percentage
2. Unlinked Open Files Files deleted but disk space remains at 100% Running process holds file descriptor open lsof +L1 Restart the holding process or truncate via /proc/<pid>/fd
3. Runaway Directory Audit Large log files consuming root filesystem Application verbose debug logging du -ahx / | sort -rh | head -20 Truncate logs with true > file.log and configure logrotate
1️⃣

Check Mounts and Inodes (df -h & df -i)

Identify which filesystem is full and whether it is bytes or inodes:

  • df -h: Identify which partition is at 100% (e.g. /, /var, or /tmp).
  • The Inode Trap (df -i): If df -h shows 40% free space but applications report 'No space left on device', you have run out of Inodes! Millions of tiny 0-byte files (like session files in /var/spool or PHP sessions) exhaust inodes even when gigabytes of disk remain.
2️⃣

Locate Largest Directories and Files (du)

Scan the filesystem safely without crossing mount boundaries:

  • du -ahx / | sort -rh | head -20: Lists top 20 space-consuming files/directories.
  • Why -x flag is critical: The -x flag stays on one filesystem. Without it, du scans NFS mounts, Docker volumes, and virtual filesystems (/proc), crashing or taking hours.
  • Quick check on common culprits: du -sh /var/log/* /var/lib/docker/* | sort -h.
Advertisement
3️⃣

The 'Deleted File Held Open' Trap (lsof +L1)

The #1 issue senior SREs look for:

  • An engineer deletes a 40GB log file using rm /var/log/app.log. But df -h STILL shows 100% full! Why?
  • Linux Filesystem Mechanics: If a running process (NGINX, Java, database) has the file open, deleting the file removes the directory directory entry, but the OS cannot free the disk blocks until the file descriptor is closed!
  • Command to identify: lsof +L1 or lsof | grep deleted.
  • How to fix without restarting service: Truncate the open file descriptor directly via /proc:
    > /proc/<PID>/fd/<FD_NUMBER> (instantly frees the disk blocks without restarting the app!).
4️⃣

Safe Cleanup Best Practices

How to free space immediately without breaking services:

  • Never rm an active log file: Always truncate: truncate -s 0 /var/log/app.log or > /var/log/app.log.
  • Vacuum systemd journal logs: journalctl --vacuum-time=2d or journalctl --vacuum-size=500M.
  • Clean Docker leftovers: docker system prune -af --volumes (verify unused volumes first).
  • Permanent fix: Configure logrotate with compression and retention limits.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Check 'df -i' for inode exhaustion. Use 'lsof +L1' to find deleted files held open by running processes, and truncate them via /proc/<PID>/fd/<FD> rather than rebooting."
⚡ 60-Second Elevator Pitch Talking Points
  • Check blocks & inodes: 'df -h' for disk space, 'df -i' for inode exhaustion (millions of tiny files).
  • Find large files: 'du -ahx / | sort -rh | head -20' (use -x to avoid traversing other mounts/NFS).
  • Check deleted files held open: 'lsof +L1' or 'lsof | grep deleted' for deleted files still locked by active processes.
  • Fix open deleted files: Truncate via '> /proc/<PID>/fd/<FD>' to free disk space immediately without restarting process.
  • Clean safely: Truncate logs (truncate -s 0 file.log) rather than 'rm'; run 'journalctl --vacuum-size=500M'.
  • Prevent: Set up logrotate, CloudWatch disk space alarm at 80%, and automated cleanup jobs.
🎬 60-Second Video Walkthrough
Watch the Animated Breakdown & Practice Live

Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.

🔔 Subscribe on YouTube (Free) ▶ Watch Short (60s) ↗
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Mistaking high Linux load average caused by uninterruptible I/O wait for compute CPU saturation Load average includes processes in uninterruptible disk sleep (D state); high load with low CPU indicates disk I/O bottlenecks or NFS hangs rather than compute saturation.
  2. Failing to check kernel ring buffer logs (dmesg -T) and journalctl during unexpected service crashes when addressing deleted Systemd service exits and kernel OOM reaper events are logged directly to the kernel ring buffer and systemd journal rather than application log files.
  3. Diagnosing infrastructure outages unsystematically without adhering to the USE or RED methodologies Senior Linux engineers systematically evaluate Utilization, Saturation, and Errors across compute, memory, and disk rather than making random configuration changes.
  4. Applying kernel runtime sysctl parameters on the fly without persisting them to /etc/sysctl.d/ Applying runtime sysctl adjustments via sysctl -w without configuration persistence causes critical tuning to vanish upon server reboot.
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →