Q: A developer runs a containerized application that writes millions of tiny temporary cache files during processing. After a few hours, the container crashes with "No space left on device" even though `docker stats` shows plenty of disk available. What is happening?
The container is likely running out of inodes, not disk space. The overlay2 filesystem has a finite number of inodes, and millions of sma...
🛠️ Production Runbook & Step-by-Step Resolution
Production Solution & Architecture
The container is likely running out of inodes, not disk space. The overlay2 filesystem has a finite number of inodes, and millions of small files can exhaust the inode table before consuming significant disk bytes. df -i inside the container will confirm this. *Fix:* For temporary cache files that don't need to persist, use a tmpfs mount which stores data entirely in RAM and doesn't consume inodes from the overlay filesystem: docker run --tmpfs /app/cache:rw,size=512m myapp Alternatively, if the cache needs disk backing, mount a dedicated volume (-v cache-vol:/app/cache) that has its own independent inode table separate from the container's root filesystem.
- Immediate Triage: The container is likely running out of inodes, not disk space. The overlay2 filesystem has a fi
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.