Q: A busy CI/CD node running Docker suddenly rejects all container creation with the error `mkdir /var/lib/docker/overlay2/...: no space left on device`. Running `df -h` shows that `/var/lib/docker` has 240GB of free disk space (only 32% disk usage). However, `df -i` reveals 100% inode utilization (`IFree: 0`). You must locate the source of millions of orphaned inodes in the OverlayFS graph driver, understand how layer merges work, and recover the storage subsystem without data loss.

Understand how OverlayFS mounts filesystem layers: Image layers are stacked as read-only `lowerdir` entries. The container's writable layer is the `upperdir`. The `workdir` is an internal scratchpad for atomic rename and copy-up operations. The container views the unified `merged` directory.

<!-- OverlayFS Layer Structure -->
/var/lib/docker/overlay2/<id>/
  ├── lowerdir/  (Read-only image layers, separated by ':')
  ├── upperdir/  (Writable container layer: modifications live here)
  ├── workdir/   (Kernel internal staging directory for atomic writes)
  └── merged/    (Unified view presented inside the container)

Verify inode exhaustion across mounted filesystems using `df -i`. Pinpoint the directory consuming millions of micro-files using `find` and `xargs`.

# Check inode usage across all mounts
df -i /var/lib/docker
# Output: Inodes: 10M, IUsed: 10M, IFree: 0 (100% Inode Exhaustion!)

# Find top 10 directories consuming the most inodes in Docker storage
find /var/lib/docker/overlay2 -maxdepth 2 -type d -exec sh -c 'echo "$(find "$1" | wc -l) $1"' _ {} \; | sort -rn | head -n 10

Common culprits include: applications generating millions of tiny session/cache files inside container writable layers instead of mounted `tmpfs` or volumes; interrupted builds leaving orphaned `diff` layers; and whiteout files (`c---`) created when deleting files from lower read-only layers.

# Check for whiteout character devices (used to mask deleted lowerdir files)
find /var/lib/docker/overlay2 -type c -name '.*' | head -n 20

Purge dangling build caches and stopped containers. Mount volatile application scratchpads on `tmpfs` to avoid consuming persistent disk inodes. Format the host backing filesystem (XFS) with increased inode density (`ftype=1` and larger `inopblock`).

# Aggressively clean up unused build cache and dangling images
docker system prune -af --volumes

# Format backing disk with high inode ratio for container workloads
# mkfs.xfs -f -n ftype=1 -i maxpct=50 /dev/nvme0n1