Q: A busy CI/CD node running Docker suddenly rejects all container creation with the error `mkdir /var/lib/docker/overlay2/...: no space left on device`. Running `df -h` shows that `/var/lib/docker` has 240GB of free disk space (only 32% disk usage). However, `df -i` reveals 100% inode utilization (`IFree: 0`). You must locate the source of millions of orphaned inodes in the OverlayFS graph driver, understand how layer merges work, and recover the storage subsystem without data loss.
Triage and fix 'No space left on device' errors caused by OverlayFS inode exhaustion (100% IFree exhausted) despite ample disk gigabytes. Understand lowerdir, upperdir, workdir, and whiteout files.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Triage and fix 'No space left on device' errors caused by OverlayFS inode exhaustion (100% IFree exhausted) despite ample disk gigabytes. Understand lowerdir, upperdir, workdir, and whiteout files.
Deconstruct OverlayFS Architecture: Lower, Upper, Work, and Merged Dirs
Understand how OverlayFS mounts filesystem layers: Image layers are stacked as read-only lowerdir entries. The container's writable layer is the upperdir. The workdir is an internal scratchpad for atomic rename and copy-up operations. The container views the unified merged directory.
<!-- OverlayFS Layer Structure -->
/var/lib/docker/overlay2/<id>/
├── lowerdir/ (Read-only image layers, separated by ':')
├── upperdir/ (Writable container layer: modifications live here)
├── workdir/ (Kernel internal staging directory for atomic writes)
└── merged/ (Unified view presented inside the container)
Diagnose Inode Exhaustion using Linux Filesystem Inspection Tools
Verify inode exhaustion across mounted filesystems using df -i. Pinpoint the directory consuming millions of micro-files using find and xargs.
# Check inode usage across all mounts
df -i /var/lib/docker
# Output: Inodes: 10M, IUsed: 10M, IFree: 0 (100% Inode Exhaustion!)
# Find top 10 directories consuming the most inodes in Docker storage
find /var/lib/docker/overlay2 -maxdepth 2 -type d -exec sh -c 'echo "$(find "$1" | wc -l) $1"' _ {} \; | sort -rn | head -n 10
Identify Root Causes: Leaked Workdirs and High-Churn Temporary Files
Common culprits include: applications generating millions of tiny session/cache files inside container writable layers instead of mounted tmpfs or volumes; interrupted builds leaving orphaned diff layers; and whiteout files (c---) created when deleting files from lower read-only layers.
# Check for whiteout character devices (used to mask deleted lowerdir files)
find /var/lib/docker/overlay2 -type c -name '.*' | head -n 20
Safely Prune Inode Bloat and Implement Prevention Guardrails
Purge dangling build caches and stopped containers. Mount volatile application scratchpads on tmpfs to avoid consuming persistent disk inodes. Format the host backing filesystem (XFS) with increased inode density (ftype=1 and larger inopblock).
# Aggressively clean up unused build cache and dangling images
docker system prune -af --volumes
# Format backing disk with high inode ratio for container workloads
# mkfs.xfs -f -n ftype=1 -i maxpct=50 /dev/nvme0n1
upperdir, exhausting filesystem inode tables even when gigabytes of disk space remain free. Use tmpfs for temporary files and tune filesystem inode density."
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- d
- a
- s
- e
- v
- e
- r
- e
- '
- n
- o
- s
- p
- a
- c
- e
- l
- e
- f
- t
- o
- n
- d
- e
- v
- i
- c
- e
- '
- o
- u
- t
- a
- g
- e
- w
- h
- e
- r
- e
- d
- i
- s
- k
- s
- p
- a
- c
- e
- w
- a
- s
- o
- n
- l
- y
- 3
- 2
- %
- u
- t
- i
- l
- i
- z
- e
- d
- b
- y
- d
- i
- s
- c
- o
- v
- e
- r
- i
- n
- g
- 1
- 0
- 0
- %
- i
- n
- o
- d
- e
- e
- x
- h
- a
- u
- s
- t
- i
- o
- n
- i
- n
- `
- /
- v
- a
- r
- /
- l
- i
- b
- /
- d
- o
- c
- k
- e
- r
- /
- o
- v
- e
- r
- l
- a
- y
- 2
- `
- .
- A
- n
- a
- p
- p
- l
- i
- c
- a
- t
- i
- o
- n
- w
- a
- s
- g
- e
- n
- e
- r
- a
- t
- i
- n
- g
- m
- i
- l
- l
- i
- o
- n
- s
- o
- f
- e
- p
- h
- e
- m
- e
- r
- a
- l
- c
- a
- c
- h
- e
- f
- i
- l
- e
- s
- i
- n
- i
- t
- s
- w
- r
- i
- t
- a
- b
- l
- e
- u
- p
- p
- e
- r
- d
- i
- r
- .
- W
- e
- p
- u
- r
- g
- e
- d
- o
- r
- p
- h
- a
- n
- e
- d
- b
- u
- i
- l
- d
- l
- a
- y
- e
- r
- s
- w
- i
- t
- h
- `
- d
- o
- c
- k
- e
- r
- s
- y
- s
- t
- e
- m
- p
- r
- u
- n
- e
- `
- ,
- m
- i
- g
- r
- a
- t
- e
- d
- e
- p
- h
- e
- m
- e
- r
- a
- l
- c
- a
- c
- h
- e
- s
- t
- o
- i
- n
- -
- m
- e
- m
- o
- r
- y
- `
- t
- m
- p
- f
- s
- `
- m
- o
- u
- n
- t
- s
- ,
- a
- n
- d
- t
- u
- n
- e
- d
- t
- h
- e
- b
- a
- c
- k
- i
- n
- g
- X
- F
- S
- f
- i
- l
- e
- s
- y
- s
- t
- e
- m
- i
- n
- o
- d
- e
- c
- e
- i
- l
- i
- n
- g
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring OverlayFS Storage Driver
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring OverlayFS Storage Driver
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring OverlayFS Storage Driver Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring OverlayFS Storage Driver Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.