Q: After months of daily Docker builds, your production server's disk is full. You run `docker system prune -af` and reclaim some space, but the disk is still 90% full. `docker system df` shows minimal usage. Where is the hidden disk consumption?
docker system prune only removes objects tracked by the Docker daemon (images, containers, volumes, build cache). If the Docker daemon cr...
🛠️ Production Runbook & Step-by-Step Resolution
Production Solution & Architecture
docker system prune only removes objects tracked by the Docker daemon (images, containers, volumes, build cache). If the Docker daemon crashed or was force-killed during container operations, orphaned layer directories can accumulate in /var/lib/docker/overlay2/ that the daemon no longer tracks. *Diagnosis:* Run du -sh /var/lib/docker/overlay2/ and compare with docker system df. A large discrepancy confirms orphaned layers. *Fix:* The safest approach is to stop the Docker daemon (systemctl stop docker), back up any critical volumes, and reset the storage entirely (rm -rf /var/lib/docker). Restarting Docker recreates the directory structure. Re-pull needed images. For prevention, ensure the Docker daemon shuts down gracefully and monitor /var/lib/docker disk usage independently.
- Immediate Triage: docker system prune only removes objects tracked by the Docker daemon (images, containers, volu
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.