Q: You see high CPU usage in one pod, but logs look clean. What next?
Root cause analysis workflow for diagnosing a Kubernetes pod pinning 100% CPU when application logs contain zero errors or anomalies.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a pod consumes 100% CPU while logging nothing, the application is stuck in a silent CPU-bound spin: an infinite loop, JVM garbage collection thrashing (Stop-The-World full GC), regex catastrophic backtracking, deadlock spinning on busy-wait locks, or an unhandled exception loop that silently swallows errors.
Identify Whether All Cores or Single Core Are Pinned
Exec into the pod or the underlying host node. Run top or htop and press 1 to see per-core CPU usage, and H to show thread-level CPU usage. Identify the exact thread ID (TID) consuming the CPU.
kubectl exec -it <pod-name> -- top -H
# Note the high CPU PID/TID: e.g. PID 42 using 99.8% CPU
Capture Thread Dumps or CPU Stack Traces
Convert the thread ID to hexadecimal and inspect thread stacks:
- Java / JVM: Use jstack and search for the hexadecimal TID (nid=0x2a).
- Go / Node / Python: Capture a pprof or async-profiler flamegraph.
# For JVM:
printf "%x\n" 42 # Output: 2a
jstack <pid> | grep -A 20 "nid=0x2a"
# For Go:
curl http://localhost:6060/debug/pprof/profile?seconds=30 > cpu.pprof
go tool pprof -http=:8080 cpu.pprof
Check Garbage Collection Thrashing & Memory Pressure
Inspect memory allocation. If the application heap is 98% full and JVM Garbage Collector threads (VM Thread, GC task thread#0) are running continuously attempting to reclaim unreachable objects, CPU will pin at 100% while application code is suspended.
jstat -gcutil <pid> 1000 5
# Inspect FGC (Full GC count) and FGCT (Full GC time)
Inspect Kernel Syscalls with strace / perf
If thread dumps cannot be gathered, attach perf or strace from the node to see what system calls the thread is executing (e.g. spinning on futex or non-blocking socket reads).
- Run top -H inside the container to identify the exact thread ID (TID) consuming CPU.
- Capture thread dumps (jstack) and correlate the hex TID to find the exact code line.
- Verify memory utilization to rule out continuous JVM Garbage Collection thrashing.
- Preserve thread dumps and heap telemetry for post-incident root cause analysis before pod restarts.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Restarting a crashing container without examining preceding termination logs via kubectl logs --previous when configuring High Usage Kubernetes Restarting the pod flushes ephemeral container runtime state, destroying vital panic stack traces and fatal initialization logs.
-
Misdiagnosing an application boot crash as a failure of aggressive liveness probes when configuring High Usage Kubernetes
Liveness probes with short timeouts or high failure thresholds kill slow-starting applications repeatedly, causing false
CrashLoopBackOffloops. -
Neglecting node taints, tolerations, and resource quotas when diagnosing pods stuck in Pending when configuring High Usage Kubernetes
Pods remain
Pendingwhen scheduler predicates fail;kubectl describe podreveals whether CPU/memory exhaustion or taints prevent placement. - Invoking kubectl delete pod --force --grace-period=0 indiscriminately on unready pods when configuring High Usage Kubernetes Forced deletion bypasses kubelet cleanup and can cause multi-attach errors on PersistentVolumes (PVs) or split-brain on stateful workloads.