Q: How would you troubleshoot high CPU or memory usage in Kubernetes?
End-to-end investigative procedure for identifying and resolving runaway CPU and memory saturation in Kubernetes clusters: isolating nodes vs pods, diagnosing memory leaks vs legitimate load, and application profiling.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a high CPU or memory alert triggers in Kubernetes, my workflow moves top-down: Cluster/Node level -> Pod level -> Container level -> Application runtime profiling (Go pprof / Java jstack).
Step 1: Isolate Affected Nodes and Pods
Identify where the saturation is concentrated:
kubectl top nodes: Pinpoints if a single node is saturated or if the entire cluster is running out of headroom.kubectl top pods -A --sort-by=cpu: Instantly lists the top CPU-consuming pods across all namespaces.kubectl top pods -A --sort-by=memory: Instantly lists the top memory-consuming pods.- Check if the pod is near its configured limit: compare
kubectl top pod <pod>withlimits.cpu/limits.memoryinkubectl describe pod.
Step 2: Differentiate CPU Spikes vs Memory Leaks
Analyze Grafana/Prometheus metrics to identify the pattern:
- CPU Saturation: Correlate with request rate (RPS) in ingress metrics. If RPS spiked 5x, it's legitimate traffic -> trigger HPA or scale replicas. If RPS is flat but CPU hit 100%, it's an infinite loop, thread lock, or regex catastrophic backtracking.
- Memory Saturation: Look at the memory graph over 24-48 hours. If memory steadily climbs in a sawtooth pattern and never drops after garbage collection, it is a Memory Leak.
Step 3: Capture Runtime Telemetry (Don't Restart Blindly)
Capture the root cause before restarting the container:
- Java / JVM: Exec into pod or use ephemeral container:
jcmd <pid> GC.heap_dump /tmp/dump.hprofandjstack <pid>to see locked threads. - Golang / Node.js: Query the pprof endpoint:
curl http://localhost:6060/debug/pprof/profile?seconds=30or heap profile. - Immediate Mitigation: Once profiling is captured, scale replicas horizontally (HPA) or do a rolling restart (
kubectl rollout restart deployment/<app>) to restore customer SLA.
- Identify culprit: 'kubectl top nodes' followed by 'kubectl top pods -A --sort-by=cpu / --sort-by=memory'.
- Correlate: Compare with Grafana request rate (RPS). Traffic spike = scale replicas. Flat traffic + 100% CPU = code deadlock/infinite loop.
- Memory triage: Check memory graph slope. Gradual steady climb without GC recovery = Memory Leak.
- Capture telemetry before restart: Run jstack/jcmd for Java, or pprof heap profiles for Go.
- Mitigate: Trigger HPA scale-out, or perform 'kubectl rollout restart' to buy time while developers fix the leak.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when configuring High Memory Usage Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes when configuring High Memory Usage
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending when configuring High Memory Usage
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods when configuring High Memory Usage Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.