Q: A developer runs a containerized application that writes millions of tiny temporary cache files during processing. After a few hours, the container crashes with "No space left on device" even though `docker stats` shows plenty of disk available. What is happening?
The container is likely running out of inodes, not disk space. The overlay2 filesystem has a finite number of inodes, and millions of small files can exhaust the inode table before consuming significant disk bytes.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In an interview, I explain how we diagnosed container runtime failures without guessing. The interviewer is testing: tmpfs mounts, inode exhaustion vs. disk space.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
The container is likely running out of inodes, not disk space. The overlay2 filesystem has a finite number of inodes, and millions of small files can exhaust the inode table before consuming significant disk bytes. df -i inside the container will confirm this. *Fix:* For temporary cache files that don't need to persist, use a tmpfs mount which stores data entirely in RAM and doesn't consume inodes from the overlay filesystem: docker run --tmpfs /app/cache:rw,size=512m myapp Alternatively, if the cache needs disk backing, mount a dedicated volume (-v cache-vol:/app/cache) that has its own independent inode table separate from the container's root filesystem.
- Immediate Triage: The container is likely running out of inodes, not disk space. The overlay2 filesystem has a fi
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring developer runs containerized
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring developer runs containerized
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring developer runs containerized Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring developer runs containerized Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.