Q: EC2 CPU suddenly reaches 100% — how would you troubleshoot?

Before logging in, look at multi-dimensional metrics to classify the nature of the spike:

• Correlate Metrics: In CloudWatch, compare CPUUtilization against NetworkIn, NetworkOut, and DiskReadOps/DiskWriteBytes.
• Traffic Surge vs Rogue Process: If high CPU coincides with a 10x spike in NetworkIn and ALB requests, it is likely legitimate or DDoS traffic. If NetworkIn is flat but CPU spiked abruptly, it is likely an internal process, runaway job, or compromised instance.
• Check Auto Scaling Group (ASG): Check if ASG target tracking is scaling out instances or if the ASG has hit its max_size limit.

Log into the instance (using AWS Systems Manager Session Manager or SSH) and inspect active processes:

• uptime: Check load average against total core count (nproc). If 4 cores and load is 40, system is severely saturated.
• top -c or htop: Press P to sort by CPU consumption. Note the PID, USER, and COMMAND.
• ps aux --sort=-%cpu | head -10: Print the top 10 CPU-consuming processes with full command-line arguments.
• vmstat 1 5: Check if CPU is spending time in user space (us), system/kernel (sy), or I/O wait (wa). High I/O wait indicates a storage bottleneck rather than pure computation.

Examine the identified process to determine what it is doing:

• Thread Inspection: Run top -H -p <PID> to inspect individual threads inside the process.
• Syscall Tracing: Run strace -p <PID> -c for 10 seconds to identify spinning loops or repetitive failing syscalls.
• Application Thread Dump: For Java/Node/Go runtimes, capture a thread dump (e.g. jstack <PID> or Go pprof) before terminating.
• Check Recent Crons: Check /var/log/syslog or /var/log/cron for heavy scheduled backup, antivirus, or indexing tasks.

Take decisive, safe actions based on the diagnosis:

• If Rogue Application Process: Issue kill -15 <PID> (SIGTERM) for graceful shutdown. Only use kill -9 if unresponsive.
• If Legitimate Traffic Surge: Manually increase ASG desired capacity, or provision a read replica / cache if backend DB queries are choking the app.
• If Compromised / Crypto-Miner: Immediately isolate instance: change Security Group to quarantine SG (no outbound/inbound except security audit), capture EBS snapshot for forensics, and terminate.