⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Docker & Containers Interview Questions Scenario 118 of 159 in Docker & Containers
Staff Infrastructure Architect Docker Container Runtime & Systems Engineering Production Scenario
🎯 Target Role / Context: Principal Docker Architect / Staff SRE Scenario

Q: A busy CI/CD node running Docker suddenly rejects all container creation with the error `mkdir /var/lib/docker/overlay2/...: no space left on device`. Running `df -h` shows that `/var/lib/docker` has 240GB of free disk space (only 32% disk usage). However, `df -i` reveals 100% inode utilization (`IFree: 0`). You must locate the source of millions of orphaned inodes in the OverlayFS graph driver, understand how layer merges work, and recover the storage subsystem without data loss.

Triage and fix 'No space left on device' errors caused by OverlayFS inode exhaustion (100% IFree exhausted) despite ample disk gigabytes. Understand lowerdir, upperdir, workdir, and whiteout files.

#Docker #Storage #Linux #Troubleshooting #SRE
🎙️ Candidate Opening & Architectural Context
"Triage and fix 'No space left on device' errors caused by OverlayFS inode exhaustion (100% IFree exhausted) despite ample disk gigabytes. Understand lowerdir, upperdir, workdir, and whiteout files."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Triage and fix 'No space left on device' errors caused by OverlayFS inode exhaustion (100% IFree exhausted) despite ample disk gigabytes. Understand lowerdir, upperdir, workdir, and whiteout files.

Step 1

Deconstruct OverlayFS Architecture: Lower, Upper, Work, and Merged Dirs

Understand how OverlayFS mounts filesystem layers: Image layers are stacked as read-only lowerdir entries. The container's writable layer is the upperdir. The workdir is an internal scratchpad for atomic rename and copy-up operations. The container views the unified merged directory.

<!-- OverlayFS Layer Structure -->
/var/lib/docker/overlay2/<id>/
  ├── lowerdir/  (Read-only image layers, separated by ':')
  ├── upperdir/  (Writable container layer: modifications live here)
  ├── workdir/   (Kernel internal staging directory for atomic writes)
  └── merged/    (Unified view presented inside the container)
Pro Tip: Deconstruct OverlayFS Architecture: Lower, Upper, Work, and Merged Dirs
Step 2

Diagnose Inode Exhaustion using Linux Filesystem Inspection Tools

Verify inode exhaustion across mounted filesystems using df -i. Pinpoint the directory consuming millions of micro-files using find and xargs.

# Check inode usage across all mounts
df -i /var/lib/docker
# Output: Inodes: 10M, IUsed: 10M, IFree: 0 (100% Inode Exhaustion!)

# Find top 10 directories consuming the most inodes in Docker storage
find /var/lib/docker/overlay2 -maxdepth 2 -type d -exec sh -c 'echo "$(find "$1" | wc -l) $1"' _ {} \; | sort -rn | head -n 10
Pro Tip: Diagnose Inode Exhaustion using Linux Filesystem Inspection Tools
Advertisement
Step 3

Identify Root Causes: Leaked Workdirs and High-Churn Temporary Files

Common culprits include: applications generating millions of tiny session/cache files inside container writable layers instead of mounted tmpfs or volumes; interrupted builds leaving orphaned diff layers; and whiteout files (c---) created when deleting files from lower read-only layers.

# Check for whiteout character devices (used to mask deleted lowerdir files)
find /var/lib/docker/overlay2 -type c -name '.*' | head -n 20
Pro Tip: Identify Root Causes: Leaked Workdirs and High-Churn Temporary Files
Step 4

Safely Prune Inode Bloat and Implement Prevention Guardrails

Purge dangling build caches and stopped containers. Mount volatile application scratchpads on tmpfs to avoid consuming persistent disk inodes. Format the host backing filesystem (XFS) with increased inode density (ftype=1 and larger inopblock).

# Aggressively clean up unused build cache and dangling images
docker system prune -af --volumes

# Format backing disk with high inode ratio for container workloads
# mkfs.xfs -f -n ftype=1 -i maxpct=50 /dev/nvme0n1
Pro Tip: Safely Prune Inode Bloat and Implement Prevention Guardrails
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"OverlayFS inode exhaustion occurs when containers generate millions of tiny files or whiteout deletion markers in their writable upperdir, exhausting filesystem inode tables even when gigabytes of disk space remain free. Use tmpfs for temporary files and tune filesystem inode density."
⚡ 60-Second Elevator Pitch Talking Points
  • W
  • e
  • d
  • i
  • a
  • g
  • n
  • o
  • s
  • e
  • d
  • a
  • s
  • e
  • v
  • e
  • r
  • e
  • '
  • n
  • o
  • s
  • p
  • a
  • c
  • e
  • l
  • e
  • f
  • t
  • o
  • n
  • d
  • e
  • v
  • i
  • c
  • e
  • '
  • o
  • u
  • t
  • a
  • g
  • e
  • w
  • h
  • e
  • r
  • e
  • d
  • i
  • s
  • k
  • s
  • p
  • a
  • c
  • e
  • w
  • a
  • s
  • o
  • n
  • l
  • y
  • 3
  • 2
  • %
  • u
  • t
  • i
  • l
  • i
  • z
  • e
  • d
  • b
  • y
  • d
  • i
  • s
  • c
  • o
  • v
  • e
  • r
  • i
  • n
  • g
  • 1
  • 0
  • 0
  • %
  • i
  • n
  • o
  • d
  • e
  • e
  • x
  • h
  • a
  • u
  • s
  • t
  • i
  • o
  • n
  • i
  • n
  • `
  • /
  • v
  • a
  • r
  • /
  • l
  • i
  • b
  • /
  • d
  • o
  • c
  • k
  • e
  • r
  • /
  • o
  • v
  • e
  • r
  • l
  • a
  • y
  • 2
  • `
  • .
  • A
  • n
  • a
  • p
  • p
  • l
  • i
  • c
  • a
  • t
  • i
  • o
  • n
  • w
  • a
  • s
  • g
  • e
  • n
  • e
  • r
  • a
  • t
  • i
  • n
  • g
  • m
  • i
  • l
  • l
  • i
  • o
  • n
  • s
  • o
  • f
  • e
  • p
  • h
  • e
  • m
  • e
  • r
  • a
  • l
  • c
  • a
  • c
  • h
  • e
  • f
  • i
  • l
  • e
  • s
  • i
  • n
  • i
  • t
  • s
  • w
  • r
  • i
  • t
  • a
  • b
  • l
  • e
  • u
  • p
  • p
  • e
  • r
  • d
  • i
  • r
  • .
  • W
  • e
  • p
  • u
  • r
  • g
  • e
  • d
  • o
  • r
  • p
  • h
  • a
  • n
  • e
  • d
  • b
  • u
  • i
  • l
  • d
  • l
  • a
  • y
  • e
  • r
  • s
  • w
  • i
  • t
  • h
  • `
  • d
  • o
  • c
  • k
  • e
  • r
  • s
  • y
  • s
  • t
  • e
  • m
  • p
  • r
  • u
  • n
  • e
  • `
  • ,
  • m
  • i
  • g
  • r
  • a
  • t
  • e
  • d
  • e
  • p
  • h
  • e
  • m
  • e
  • r
  • a
  • l
  • c
  • a
  • c
  • h
  • e
  • s
  • t
  • o
  • i
  • n
  • -
  • m
  • e
  • m
  • o
  • r
  • y
  • `
  • t
  • m
  • p
  • f
  • s
  • `
  • m
  • o
  • u
  • n
  • t
  • s
  • ,
  • a
  • n
  • d
  • t
  • u
  • n
  • e
  • d
  • t
  • h
  • e
  • b
  • a
  • c
  • k
  • i
  • n
  • g
  • X
  • F
  • S
  • f
  • i
  • l
  • e
  • s
  • y
  • s
  • t
  • e
  • m
  • i
  • n
  • o
  • d
  • e
  • c
  • e
  • i
  • l
  • i
  • n
  • g
  • .
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring OverlayFS Storage Driver Copying code before package manifests (package.json, go.mod, requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls.
  2. Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring OverlayFS Storage Driver If an application runs as PID 1 without an init system (like tini), it fails to reap zombie child processes and ignores SIGTERM signals, causing unclean 10s timeouts.
  3. Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring OverlayFS Storage Driver Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
  4. Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring OverlayFS Storage Driver Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.
Advertisement
Want more Docker & Containers scenarios?
Explore our complete collection of scenario-based Docker & Containers interview runbooks.
Browse All Docker & Containers Questions →