Q: In your Kubernetes production clusters running containerd, worker nodes frequently enter `DiskPressure` status. When disk usage reaches 85%, kubelet abruptly starts evicting running application pods, causing cascading service disruptions. SREs discover that unused container images take up 140GB of node disk space because image garbage collection thresholds were left at uncalibrated defaults. You must configure kubelet and containerd image garbage collection parameters to proactively purge stale images before pod eviction triggers.
Configure containerd and kubelet image garbage collection algorithms. Tune imageGCHighThresholdPercent, imageGCLowThresholdPercent, and imageMinimumGCAge to prevent node disk pressure evictions.
imageGCHighThresholdPercent, imageGCLowThresholdPercent, and imageMinimumGCAge to prevent node disk pressure evictions."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Configure containerd and kubelet image garbage collection algorithms. Tune imageGCHighThresholdPercent, imageGCLowThresholdPercent, and imageMinimumGCAge to prevent node disk pressure evictions.
Analyze Kubelet Image Garbage Collection Architecture
Kubelet manages image garbage collection via three primary parameters: imageGCHighThresholdPercent (disk percentage at which GC begins), imageGCLowThresholdPercent (target disk percentage to which GC frees space), and imageMinimumGCAge (minimum age an unused image must have before being eligible for deletion).
<!-- Kubelet Image GC Flow -->
Disk Usage climbs to 80% (imageGCHighThresholdPercent)
│
└── Kubelet initiates Image GC via CRI containerd
│ (Deletes unused images oldest first, respecting imageMinimumGCAge)
└── Stops deleting once disk drops to 65% (imageGCLowThresholdPercent)
Note: If usage hits 85% (evictionHard: nodefs.available<15%), Pod Eviction begins!
Calibrate Thresholds to Run Proactively Before Eviction Triggers
Configure kubelet configuration (/var/lib/kubelet/config.yaml) so image GC initiates well before hard eviction limits are reached.
# /var/lib/kubelet/config.yaml
imageGCHighThresholdPercent: 75
imageGCLowThresholdPercent: 60
imageMinimumGCAge: 20m
evictionHard:
nodefs.available: "10%"
imagefs.available: "15%"
Configure Dedicated ImageFS on Containerd Storage
To prevent container images and container writable layers from contending with host OS logs and root filesystem space, mount /var/lib/containerd on a dedicated partition (imagefs). Kubelet tracks imagefs independently of nodefs.
# Verify imagefs detection in crictl
crictl info | jq .status.storageStatus
# Confirms dedicated image filesystem partition
Test Image Garbage Collection Behavior
Simulate high image accumulation on a test node using crictl pull. Verify that upon crossing the 75% threshold, kubelet instructs containerd to prune unused images down to 60% without evicting any active pods.
# Monitor kubelet garbage collection logs
journalctl -u kubelet -f | grep -E 'ImageGC|garbage collecting'
imageGCHighThresholdPercent: 75 and imageGCLowThresholdPercent: 60 ensures containerd purges unreferenced images proactively, preventing node DiskPressure states."
- W
- e
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- n
- o
- d
- e
- D
- i
- s
- k
- P
- r
- e
- s
- s
- u
- r
- e
- e
- v
- i
- c
- t
- i
- o
- n
- s
- a
- c
- r
- o
- s
- s
- 3
- 0
- 0
- K
- u
- b
- e
- r
- n
- e
- t
- e
- s
- n
- o
- d
- e
- s
- b
- y
- r
- e
- -
- c
- a
- l
- i
- b
- r
- a
- t
- i
- n
- g
- k
- u
- b
- e
- l
- e
- t
- a
- n
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- i
- m
- a
- g
- e
- g
- a
- r
- b
- a
- g
- e
- c
- o
- l
- l
- e
- c
- t
- i
- o
- n
- .
- W
- e
- l
- o
- w
- e
- r
- e
- d
- t
- h
- e
- h
- i
- g
- h
- t
- h
- r
- e
- s
- h
- o
- l
- d
- t
- o
- 7
- 5
- %
- a
- n
- d
- t
- h
- e
- l
- o
- w
- t
- h
- r
- e
- s
- h
- o
- l
- d
- t
- o
- 6
- 0
- %
- ,
- a
- l
- l
- o
- w
- i
- n
- g
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- t
- o
- p
- r
- o
- a
- c
- t
- i
- v
- e
- l
- y
- p
- u
- r
- g
- e
- d
- e
- a
- d
- l
- a
- y
- e
- r
- s
- b
- e
- f
- o
- r
- e
- h
- a
- r
- d
- e
- v
- i
- c
- t
- i
- o
- n
- l
- i
- m
- i
- t
- s
- t
- r
- i
- g
- g
- e
- r
- ,
- s
- a
- v
- i
- n
- g
- r
- u
- n
- n
- i
- n
- g
- p
- o
- d
- s
- f
- r
- o
- m
- u
- n
- e
- x
- p
- e
- c
- t
- e
- d
- r
- e
- s
- c
- h
- e
- d
- u
- l
- i
- n
- g
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Tuning containerd Kubelet
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Tuning containerd Kubelet
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Tuning containerd Kubelet Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Tuning containerd Kubelet Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.