Q: During an incident, you notice that a specific log file (`/var/log/app.log`) is growing at 5GB per minute, threatening to fill the disk. Deleting the file doesn't free up the disk space space. Why, and what do you do?
In Linux, when you rm a file, you delete the directory entry (the link to the inode). However, the disk blocks are not freed as long as a...
🛠️ Production Runbook & Step-by-Step Resolution
Initial Diagnostics & Root Cause Analysis
In Linux, when you rm a file, you delete the directory entry (the link to the inode). However, the disk blocks are not freed as long as any running process holds an open file descriptor to that file. The application is still writing to the deleted file's inode.
- Find PID using
lsof. - Find the FD number in
lsofoutput (e.g., FD 4). - Truncate it:
> /proc//fd/4
Remediation & Permanent Safeguards
Running lsof | grep deleted will show the application holding the file open. To actually free the space without restarting the application (which might cause downtime), I would truncate the file instead of deleting it. If the file is already deleted but held open, I would find it via the proc filesystem and truncate it: In the future, I'd configure logrotate to use copytruncate or send a SIGHUP to the app to force it to reopen logs gracefully.
- Find PID using lsof.
- Find the FD number in lsof output (e.g., FD 4).
- Truncate it: > /proc//fd/4