Q: A production node flags 'DiskPressure=True', and the Kubelet immediately begins evicting BestEffort and Burstable pods. The root disk is at 94% utilization. How do you triage whether the saturation is on nodefs or imagefs, safely free space immediately without causing outage, and configure permanent kubelet GC and log rotation guardrails?

Confirm the DiskPressure condition and determine which physical filesystem mount is exhausted:

• Check Node Condition: Run kubectl describe node <node-name> | grep -A 5 DiskPressure. Verify if Status: True and check the reason (KubeletHasDiskPressure).
• Inspect Host Mounts (nodefs vs imagefs): SSH into the worker node and run df -hT. If container runtime images reside on a dedicated partition (e.g. /var/lib/containerd), compare it against the root OS filesystem (/).
• Identify Eviction Storm: Run kubectl get pods -A --field-selector spec.nodeName=<node-name> to identify pods marked Evicted with reason The node was low on resource: ephemeral-storage.

If imagefs is saturated, manually trigger container image garbage collection using CRI tools:

• Audit Image Disk Consumption: Run crictl images and check total size consumed by unused images.
• Execute Prune Command: Run crictl rmi --prune. This safely removes all unreferenced container images that are not currently tied to running containers.
• Check Space Freed: Run df -h /var/lib/containerd. If space drops below the high threshold, Kubelet will clear the condition within the next sync cycle.

If nodefs root partition is full, runaway application logging in /var/log/pods is the most common culprit:

• Locate Top Consuming Pod Directories: Run du -sh /var/log/pods/* | sort -hr | head -n 10.
• Identify Massive Log Files: Find files larger than 500MB: find /var/log/pods -type f -name "*.log" -size +500M -exec ls -lh {} +.
• Safe Log Truncation: Never use rm because running containers maintain open file descriptors, keeping the inode allocated. Instead, truncate the file in-place: truncate -s 0 /var/log/pods/<path>/<container>/*.log.

Microservices writing temporary datasets to emptyDir volumes without resource limits can silently overwhelm nodefs:

• Scan Kubelet Pod Volume Mounts: Run du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~empty-dir/* | sort -hr | head -n 10.
• Match Directory to Pod UID: Use kubectl get pod <pod-name> -o jsonpath='{.metadata.uid}' to correlate the heavy folder back to the specific offending pod.
• Kill or Restart the Offending Workload: Delete the pod or scale down its deployment so Kubelet cleans up the temporary volume.

Configure Kubelet with proactive automated garbage collection thresholds so disk usage never reaches the emergency eviction limit:

• Configure Image GC Thresholds: In /var/lib/kubelet/config.yaml, configure:

  imageGCHighThresholdPercent: 80
  imageGCLowThresholdPercent: 65
  imageMinimumGCAge: 2m

• Configure Container Log Max Size: Set kubelet log rotation properties:

  containerLogMaxSize: 50Mi
  containerLogMaxFiles: 5

• Define Hard & Soft Eviction Thresholds:

  evictionHard:
    nodefs.available: 10%
    imagefs.available: 15%
  evictionSoft:
    nodefs.available: 15%
    imagefs.available: 20%
  evictionSoftGracePeriod:
    nodefs.available: 2m
    imagefs.available: 2m

Enforce pod-level quotas to guarantee that a rogue application is evicted individually before the whole node suffers DiskPressure:

• Set ephemeral-storage Limits: In deployment specs, define requests and limits:

  resources:
    requests:
      ephemeral-storage: "500Mi"
    limits:
      ephemeral-storage: "4Gi"

• Verify Kubelet Clears Condition: Confirm the node condition returns to DiskPressure: False in kubectl describe node <node-name>.
• Lab Runbook: Practice automated disk pressure triage and crictl GC in DevOps Lab: Node Disk Pressure Hands-On Runbook ↗.