Q: A critical service suddenly starts consuming excessive CPU and memory. What would your troubleshooting methodology look like? How would you distinguish between application-level problems, infrastructure issues, resource misconfiguration, and traffic anomalies?
Deep diagnostic framework for distinguishing application memory leaks, infinite loops, CFS throttling, external database deadlocks, and traffic anomalies under pressure.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a service suddenly spikes in both CPU and memory, engineers often assume traffic surge or code bug. A rigorous SRE methodology classifies the anomaly across 4 distinct buckets before taking action.
Step 1: Classify the Anomaly Across 4 Buckets
Telemetry triage in the first 2 minutes:
- Bucket 1: Traffic Anomaly (Legit vs DDoS): Check Ingress RPS and active connections. If RPS jumped 10x proportionally with CPU, it's traffic load → Scale out horizontally (HPA / ASG).
- Bucket 2: Application-Level Bug: Traffic is flat, but CPU is pegged at 100% and memory climbs steadily without dropping → Infinite loop, regex catastrophic backtracking (ReDoS), or memory leak.
- Bucket 3: Downstream Bottleneck (Thread Pileup): Database or third-party payment gateway is slow/locked. Requests pile up waiting for I/O, exhausting thread pools and accumulating in-memory buffers.
- Bucket 4: Resource Misconfiguration: Low CPU limits causing Linux CFS throttling; or JVM max heap (
-Xmx) exceeding container memory limit, causing kernel OOMKill.
Step 2: CPU Deep-Dive (Thread & Syscall Profiling)
Isolate user code vs kernel syscalls:
top -H -p <PID>: Identify the specific Thread ID (TID) consuming CPU.strace -p <PID> -c: Check time spent in syscalls. If high infutex, service is suffering from thread lock contention.- Capture profiling artifacts before killing: Run
perf top, capture Javajstack <PID>, or profile Go runtime withpprof.
Step 3: Memory Deep-Dive (Leak vs Cache)
Distinguish healthy cache from dangerous memory leaks:
- Garbage Collection Metrics: Check GC frequency and duration in Prometheus. If Major/Full GC runs constantly and memory never recovers, it is an active Memory Leak (unbounded static collections, unclosed streams, or thread-local leaks).
- Capture Heap Dump:
jmap -dump:live,format=b,file=heap.bin <PID>or Gopprof/heap. - Analyze heap dump in Eclipse MAT (Memory Analyzer Tool) to find the largest memory-retaining objects.
Step 4: Mitigate Safely and Protect Customers
Immediate stabilization and safeguards:
- If thread pileup from downstream failure: Enable Circuit Breakers (Resilience4j / Envoy) and aggressive connection timeouts to fail fast.
- If memory leak: Temporarily increase memory limits or restart pods in a rolling fashion while developers analyze the heap dump.
- Tune cgroups: Remove artificial CPU limits that trigger kernel throttling if average node utilization is healthy.
- Classify into 4 buckets: Traffic surge, application bug (leak/loop), downstream bottleneck (thread pileup), or misconfiguration.
- CPU drilldown: 'top -H -p <PID>' to find rogue thread; capture thread dump (jstack/pprof) or strace before killing.
- Memory drilldown: Monitor GC pauses in Prometheus. If full GC doesn't reclaim memory, capture heap dump (jmap/heap profile).
- Downstream check: Verify if slow database queries are causing worker thread accumulation and buffer exhaustion.
- Mitigate: Trip circuit breakers, scale out horizontally, or roll restart pods with captured diagnostic dumps.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when configuring Critical Service Excessive SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
-
Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when configuring Critical Service Excessive
Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (
df -h) shows ample free space. - Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections when configuring Critical Service Excessive Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
-
Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping when configuring Critical Service Excessive
Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (
pid_max).