Q: In your Kubernetes production clusters running containerd, worker nodes frequently enter `DiskPressure` status. When disk usage reaches 85%, kubelet abruptly starts evicting running application pods, causing cascading service disruptions. SREs discover that unused container images take up 140GB of node disk space because image garbage collection thresholds were left at uncalibrated defaults. You must configure kubelet and containerd image garbage collection parameters to proactively purge stale images before pod eviction triggers.
Configure containerd and kubelet image garbage collection algorithms. Tune `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`, and `imageMinimumGCAge` to prevent node disk pressure evictions.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Analyze Kubelet Image Garbage Collection Architecture
Kubelet manages image garbage collection via three primary parameters: `imageGCHighThresholdPercent` (disk percentage at which GC begins), `imageGCLowThresholdPercent` (target disk percentage to which GC frees space), and `imageMinimumGCAge` (minimum age an unused image must have before being eligible for deletion).
<!-- Kubelet Image GC Flow -->
Disk Usage climbs to 80% (imageGCHighThresholdPercent)
│
└── Kubelet initiates Image GC via CRI containerd
│ (Deletes unused images oldest first, respecting imageMinimumGCAge)
└── Stops deleting once disk drops to 65% (imageGCLowThresholdPercent)
Note: If usage hits 85% (evictionHard: nodefs.available<15%), Pod Eviction begins!Calibrate Thresholds to Run Proactively Before Eviction Triggers
Configure kubelet configuration (`/var/lib/kubelet/config.yaml`) so image GC initiates well before hard eviction limits are reached.
# /var/lib/kubelet/config.yaml
imageGCHighThresholdPercent: 75
imageGCLowThresholdPercent: 60
imageMinimumGCAge: 20m
evictionHard:
nodefs.available: "10%"
imagefs.available: "15%"Configure Dedicated ImageFS on Containerd Storage
To prevent container images and container writable layers from contending with host OS logs and root filesystem space, mount `/var/lib/containerd` on a dedicated partition (`imagefs`). Kubelet tracks `imagefs` independently of `nodefs`.
# Verify imagefs detection in crictl
crictl info | jq .status.storageStatus
# Confirms dedicated image filesystem partitionTest Image Garbage Collection Behavior
Simulate high image accumulation on a test node using `crictl pull`. Verify that upon crossing the 75% threshold, kubelet instructs containerd to prune unused images down to 60% without evicting any active pods.
# Monitor kubelet garbage collection logs
journalctl -u kubelet -f | grep -E 'ImageGC|garbage collecting'- W
- e
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- n
- o
- d
- e
- D
- i
- s
- k
- P
- r
- e
- s
- s
- u
- r
- e
- e
- v
- i
- c
- t
- i
- o
- n
- s
- a
- c
- r
- o
- s
- s
- 3
- 0
- 0
- K
- u
- b
- e
- r
- n
- e
- t
- e
- s
- n
- o
- d
- e
- s
- b
- y
- r
- e
- -
- c
- a
- l
- i
- b
- r
- a
- t
- i
- n
- g
- k
- u
- b
- e
- l
- e
- t
- a
- n
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- i
- m
- a
- g
- e
- g
- a
- r
- b
- a
- g
- e
- c
- o
- l
- l
- e
- c
- t
- i
- o
- n
- .
- W
- e
- l
- o
- w
- e
- r
- e
- d
- t
- h
- e
- h
- i
- g
- h
- t
- h
- r
- e
- s
- h
- o
- l
- d
- t
- o
- 7
- 5
- %
- a
- n
- d
- t
- h
- e
- l
- o
- w
- t
- h
- r
- e
- s
- h
- o
- l
- d
- t
- o
- 6
- 0
- %
- ,
- a
- l
- l
- o
- w
- i
- n
- g
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- t
- o
- p
- r
- o
- a
- c
- t
- i
- v
- e
- l
- y
- p
- u
- r
- g
- e
- d
- e
- a
- d
- l
- a
- y
- e
- r
- s
- b
- e
- f
- o
- r
- e
- h
- a
- r
- d
- e
- v
- i
- c
- t
- i
- o
- n
- l
- i
- m
- i
- t
- s
- t
- r
- i
- g
- g
- e
- r
- ,
- s
- a
- v
- i
- n
- g
- r
- u
- n
- n
- i
- n
- g
- p
- o
- d
- s
- f
- r
- o
- m
- u
- n
- e
- x
- p
- e
- c
- t
- e
- d
- r
- e
- s
- c
- h
- e
- d
- u
- l
- i
- n
- g
- .