Q: Process is consuming high CPU — which commands would you use?
Full command toolkit for diagnosing high CPU processes: thread-level drilldown (top -H), syscall tracing (strace), kernel profiling (perf), and graceful mitigation.
Process Consuming High CPU — Linux Deep Inspection in 60s
Watch Naveed break down how to investigate a process consuming high CPU—drilling down from load averages and top -c to thread inspection (top -H), strace syscall latency, and graceful mitigation in 60 seconds.
- Step 1: Isolate the PID with top -c and verify CPU saturation vs load average with uptime & vmstat
- Step 2: Drill down into individual thread IDs (TID) consuming CPU using top -H -p <PID>
- Step 3: Profile syscall latency and lock contention using strace -p <PID> -c and perf top
- Step 4: Gracefully deprioritize with renice +10 or send SIGTERM (kill -15) before SIGKILL
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a process consumes high CPU, my goal is to drill down from the system process level down to the exact thread, system call, or application function causing the burn.
Identify Process & System Context
Find the PID and understand its resource footprint:
top -c: PressPto sort by CPU. Shows full command-line paths.ps aux --sort=-%cpu | head -10: Fast scriptable capture of top 10 CPU consumers.pidstat 1 5 -u: Granular per-process CPU statistics over 5 seconds.
Thread-Level Inspection (top -H)
Multi-threaded runtimes (Java, Go, C++, Node worker threads) distribute load across threads:
top -H -p <PID>: Primary diagnostic command — displays individual threads inside the process as if they were processes. Identify the specific Thread ID (TID) pinned at 100%.ps -T -p <PID>: List all threads and their CPU consumption.- For Java: Convert TID to hexadecimal (
printf '%x\n' <TID>) and grep inside ajstack <PID>thread dump to find the exact line of Java code that is looping!
Trace Syscalls & Profile CPU (strace / perf)
Determine if CPU is burned in user code or kernel syscalls:
strace -p <PID> -c: Run for 10 seconds. Summarizes system calls, error rates, and time spent. If 90% time is infutex, process is suffering thread lock contention.strace -p <PID> -t -e trace=all: View live system calls in real-time.- Kernel Profiling (
perf): Runperf top -p <PID>orperf record -F 99 -p <PID> -g -- sleep 10 && perf reportto generate a call graph / flamegraph showing exact function hotspots.
Mitigate and Control Priority
Managing the process safely:
- Lower process priority:
renice +10 -p <PID>gives other critical system processes CPU priority. - Graceful shutdown:
kill -15 <PID>(SIGTERM). Only usekill -9as a last resort. - Restrict with cgroups: Apply CPU bandwidth limits via systemd slice or Docker
--cpus=1.5.
- Find process: 'top -c' (sort by P) or 'ps aux --sort=-%cpu | head -10' or 'pidstat 1 5 -u'.
- Inspect threads: 'top -H -p <PID>' to find the exact thread ID (TID) burning CPU.
- Trace syscalls: 'strace -p <PID> -c' to see time spent in kernel syscalls vs user code.
- Profile CPU: 'perf top -p <PID>' or capture thread dumps (jstack/pprof) before killing.
- Control: 'renice +10 <PID>' to deprioritize, or 'kill -15 <PID>' for graceful termination.
- Prevent: Set cgroup / systemd CPUQuota limits and container resource limits.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Mistaking high Linux load average caused by uninterruptible I/O wait for compute CPU saturation
Load average includes processes in uninterruptible disk sleep (
Dstate); high load with low CPU indicates disk I/O bottlenecks or NFS hangs rather than compute saturation. - Failing to check kernel ring buffer logs (dmesg -T) and journalctl during unexpected service crashes when addressing consuming Systemd service exits and kernel OOM reaper events are logged directly to the kernel ring buffer and systemd journal rather than application log files.
- Diagnosing infrastructure outages unsystematically without adhering to the USE or RED methodologies Senior Linux engineers systematically evaluate Utilization, Saturation, and Errors across compute, memory, and disk rather than making random configuration changes.
-
Applying kernel runtime sysctl parameters on the fly without persisting them to /etc/sysctl.d/
Applying runtime sysctl adjustments via
sysctl -wwithout configuration persistence causes critical tuning to vanish upon server reboot.