Q: An application deployed on a Linux machine suddenly starts failing with "No space left on device", but `df -h` shows the partition is only 50% full. What is happening?
This is a classic issue of running out of inodes. While the physical disk blocks might be available, the filesystem has exhausted its all...
🛠️ Production Runbook & Step-by-Step Resolution
Production Solution & Architecture
This is a classic issue of running out of inodes. While the physical disk blocks might be available, the filesystem has exhausted its allocation of inodes, which are required to track every file and directory. This typically happens when an application creates millions of tiny files (like session files, logs, or cache files). I would verify this by running df -i. If IUse% is at 100%, I need to find the directory with the massive number of files. To find the culprit, I can run: find / -type d -size +100K (Directories grow in size when they contain many entries) Or the slower but precise method: for i in /*; do echo $i; find $i |wc -l; done Once found (often /tmp or /var/lib/php/sessions), I would delete or archive the files, potentially using find to prevent arguments list too long errors.
- Immediate Triage: This is a classic issue of running out of inodes. While the physical disk blocks might be avail
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.