Q: Production server becomes slow — how do you identify CPU/memory/disk/network issues?
Mastering the USE Method (Utilization, Saturation, Errors) to isolate system bottlenecks across CPU, Memory, Disk I/O, and Network in under 60 seconds.
Production Server Becomes Slow? The USE Method Finds It in 60s
Watch Naveed break down what to do when a production server becomes slow using Brendan Gregg's USE Method to isolate CPU, memory paging, disk I/O, and network saturation in 60 seconds.
- Step 1: Check uptime and compare 1/5/15m load average against nproc CPU core count
- Step 2: Inspect vmstat 1 5 run queue (r) for CPU saturation vs blocked (b) and iowait (wa)
- Step 3: Check free -h available memory and vmstat si/so swap thrashing activity
- Step 4: Isolate disk bottlenecks with iostat -xz (%util and await) and network drops with sar -n DEV
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a production server becomes slow, I apply Brendan Gregg's USE Method (Utilization, Saturation, Errors) using a systematic 60-second command checklist to isolate which subsystem is bottlenecking.
First 15 Seconds: System Overview
Assess overall system pressure:
uptime: Look at load average (1m, 5m, 15m). Compare against core count (nproc). If 4 cores and load average is 28, system is heavily saturated.dmesg -T | tail -30: Check kernel ring buffer for hardware errors, TCP drop warnings, or OOM killer events. Compare with our Pod CrashLoopBackOff & OOMKilled exit code 137 runbook.
CPU vs Disk I/O: vmstat 1 5
The most informative single command in Linux:
- Run:
vmstat 1 5and read the columns: - r (run queue): Processes waiting for CPU. If r > total cores, CPU is saturated.
- b (blocked): Processes blocked waiting for I/O. If b > 0, system is waiting on disk/NFS.
- us (user) vs sy (system): High
us= application code. Highsy= kernel context switching or excessive syscalls. - wa (iowait): High
wa(>20%) = CPU is idle because it is waiting for disk/storage to respond! The bottleneck is DISK, not CPU!
Memory Check: free -h & vmstat swap
Inspect real available memory and paging:
free -h: Look at available (not free). Available accounts for reclaimable page cache and buffers.- In
vmstat 1 5, look at si (swap in) and so (swap out). Ifso > 0, the system is actively thrashing memory to disk, causing severe latency. dmesg -T | grep -i 'killed process': Check if kernel OOM killer has executed. For containerized workloads under memory pressure, see our Kubernetes Pod Pending & Memory Pressure runbook.
Disk I/O & Network Deep Dive
Isolate storage or packet bottlenecks:
- Disk I/O (
iostat -xz 1 5): Check%utilandawait. If%util > 90%orawait > 15ms, physical EBS/disk storage is saturated. - Network (
sar -n DEV 1 5): Check rxpck/s, txpck/s, and rxdrop/txdrop for dropped packets. - Socket Pressure (
ss -s): Check TCP socket counts. Iftimewaitorestabis in tens of thousands, connection pool exhaustion is occurring. For multi-tier cloud routing issues, cross-reference our Outside-In VPC Network Tracing runbook.
- Step 1: 'uptime' -> compare 1/5/15m load average against 'nproc' core count.
- Step 2: 'vmstat 1 5' -> check 'r' (CPU saturation), 'b' (I/O blocked), and 'wa' (if iowait is high, bottleneck is disk, not CPU).
- Step 3: 'free -h' -> check 'available' memory; check 'si/so' in vmstat for swap thrashing.
- Step 4: 'iostat -xz 1 5' -> check '%util' and 'await' response times for storage bottlenecks.
- Step 5: 'sar -n DEV 1 5' and 'ss -s' -> check network throughput, dropped packets, and socket saturation.
- Step 6: 'dmesg -T | tail -30' for OOM kills and hardware errors.
Mastered this procedure? Subscribe to @naveedkumbhar01 on YouTube for daily 60-second SRE incident runbooks, Kubernetes triage shortcuts, and high-impact interview elevator pitches.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Mistaking high Linux load average caused by uninterruptible I/O wait for compute CPU saturation
Load average includes processes in uninterruptible disk sleep (
Dstate); high load with low CPU indicates disk I/O bottlenecks or NFS hangs rather than compute saturation. - Failing to check kernel ring buffer logs (dmesg -T) and journalctl during unexpected service crashes when addressing identify Systemd service exits and kernel OOM reaper events are logged directly to the kernel ring buffer and systemd journal rather than application log files.
- Diagnosing infrastructure outages unsystematically without adhering to the USE or RED methodologies Senior Linux engineers systematically evaluate Utilization, Saturation, and Errors across compute, memory, and disk rather than making random configuration changes.
-
Applying kernel runtime sysctl parameters on the fly without persisting them to /etc/sysctl.d/
Applying runtime sysctl adjustments via
sysctl -wwithout configuration persistence causes critical tuning to vanish upon server reboot.