Q: EC2 CPU suddenly reaches 100% — how would you troubleshoot?
Triage runbook for handling sudden 100% CPU utilization on production EC2 instances: telemetry triage, identifying culprit processes, handling legit vs malicious load, and permanent safeguards.
EC2 CPU at 100%? Do THIS Like a Senior SRE (Interview Runbook)
Watch Naveed break down the step-by-step triage sequence when a production AWS EC2 instance spikes to 100% CPU—from CloudWatch telemetry correlation and thread-dump forensics to graceful mitigation and permanent ASG safeguards.
- Correlate CloudWatch CPU metrics with NetworkIn and ALB request rates first
- Inspect system telemetry via SSM Session Manager: top -c, vmstat, and ps aux
- Capture thread dumps (jstack/pprof) or strace before sending any kill signals
- Implement ASG auto-scaling policies, CloudWatch 75% alarms, and cgroup limits
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a production EC2 instance hits 100% CPU, my priority is two-fold: stop customer impact immediately (mitigation) while capturing telemetry to pinpoint the exact culprit process (root cause analysis).
Triage From Outside (CloudWatch & Telemetry)
Before logging in, look at multi-dimensional metrics to classify the nature of the spike:
- Correlate Metrics: In CloudWatch, compare
CPUUtilizationagainstNetworkIn,NetworkOut, andDiskReadOps/DiskWriteBytes. - Traffic Surge vs Rogue Process: If high CPU coincides with a 10x spike in NetworkIn and ALB requests, it is likely legitimate or DDoS traffic. If NetworkIn is flat but CPU spiked abruptly, it is likely an internal process, runaway job, or compromised instance.
- Check Auto Scaling Group (ASG): Check if ASG target tracking is scaling out instances or if the ASG has hit its
max_sizelimit.
SSH / SSM Session & Process Inspection
Log into the instance (using AWS Systems Manager Session Manager or SSH) and inspect active processes:
uptime: Check load average against total core count (nproc). If 4 cores and load is 40, system is severely saturated.top -corhtop: PressPto sort by CPU consumption. Note the PID, USER, and COMMAND.ps aux --sort=-%cpu | head -10: Print the top 10 CPU-consuming processes with full command-line arguments.vmstat 1 5: Check if CPU is spending time in user space (us), system/kernel (sy), or I/O wait (wa). High I/O wait indicates a storage bottleneck rather than pure computation.
Diagnose the Culprit Process
Examine the identified process to determine what it is doing:
- Thread Inspection: Run
top -H -p <PID>to inspect individual threads inside the process. - Syscall Tracing: Run
strace -p <PID> -cfor 10 seconds to identify spinning loops or repetitive failing syscalls. - Application Thread Dump: For Java/Node/Go runtimes, capture a thread dump (e.g.
jstack <PID>or Go pprof) before terminating. - Check Recent Crons: Check
/var/log/syslogor/var/log/cronfor heavy scheduled backup, antivirus, or indexing tasks.
Mitigate and Restore Service
Take decisive, safe actions based on the diagnosis:
- If Rogue Application Process: Issue
kill -15 <PID>(SIGTERM) for graceful shutdown. Only usekill -9if unresponsive. - If Legitimate Traffic Surge: Manually increase ASG desired capacity, or provision a read replica / cache if backend DB queries are choking the app.
- If Compromised / Crypto-Miner: Immediately isolate instance: change Security Group to quarantine SG (no outbound/inbound except security audit), capture EBS snapshot for forensics, and terminate.
- Check CloudWatch: correlate CPU with NetworkIn and ALB traffic to differentiate traffic surge from rogue process.
- Access instance via SSM Session Manager / SSH; run 'top -c', 'uptime', and 'ps aux --sort=-%cpu | head -10'.
- Check 'vmstat 1 5' for 'us' (app code) vs 'wa' (I/O wait bottleneck) vs 'sy' (kernel context switching).
- Inspect threads with 'top -H -p <PID>' and capture thread dump (jstack/pprof) or strace before killing.
- Mitigate: graceful SIGTERM, scale out ASG, or quarantine if compromised.
- Prevent: CloudWatch Alarm at 75% CPU, ASG auto-scaling, CPU limits (cgroups/Docker), and query optimization.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Jumping straight to SSH/SSM without checking CloudWatch metrics first Interviewers expect triage from outside the instance first to correlate CPU with NetworkIn, NetworkOut, and ALB requests. Tunnel-visioning straight to terminal access signals reactive server administration rather than cloud architecture and holistic observability.
- Recommending an immediate instance reboot or termination Rebooting destroys the smoking gun: active processes, kernel thread states, CPU profiles, and memory telemetry are permanently lost. Senior engineers isolate the node from the load balancer, capture diagnostics, and attempt graceful mitigation before considering a restart.
- Ignoring the vmstat breakdown (us vs sy vs wa) Stating "CPU is 100%" without identifying whether CPU time is spent in user space (application code), kernel/system (syscall churn or lock contention), or I/O wait (EBS volume IOPS/throughput exhaustion) demonstrates surface-level Linux troubleshooting.
- Immediately reaching for kill -9 (SIGKILL) instead of kill -15 (SIGTERM) Unconditional kill -9 corrupts in-flight database transactions, leaves locks dangling, and bypasses application cleanup handlers. Always send SIGTERM first and allow reasonable drain time before escalating to SIGKILL.
- Failing to evaluate security compromise or cryptomining malware A sudden 100% CPU spike with zero corresponding network traffic spike is a classic footprint of a compromised host running an unauthorized crypto-miner. Omitting security containment (security group quarantine, EBS snapshot forensics) is a significant gap in SRE interviews.
- Omitting proactive architectural safeguards and auto-remediation Ending your answer at killing the process shows a reactive break-fix mindset. Senior engineers articulate root-cause prevention: CloudWatch Alarms at 75% CPU, Auto Scaling target tracking policies, container cgroup CPU limits, and database query optimization.