Q: A deployment completed successfully, but response time increased from 200 ms to 3 seconds. Walk me through your troubleshooting approach.
Step-by-step diagnostic workflow for triaging a 15x response time regression (200ms to 3s) immediately following a production deployment without catastrophic error spikes.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a deployment completes with green health checks but latency explodes from 200ms to 3 seconds, my immediate priority is blast radius mitigation followed by differential tracing between the previous and current release versions.
Immediate Blast Radius Containment
Check if a canary, blue-green, or rolling deployment is in progress. If traffic is still on a subset of pods, halt rollout immediately or divert traffic back to the known good baseline while preserving 1-2 regressed pods for live forensic debugging.
Distributed Tracing Differential Comparison
Open Datadog, Jaeger, or AWS X-Ray and compare latency flame graphs of the new release versus the previous tag. Pinpoint whether the extra 2.8 seconds is spent on external HTTP calls, database query lock waiting, or CPU execution.
# Inspect container CPU throttling and garbage collection pauses
kubectl top pod -l app=payment-service
cat /sys/fs/cgroup/cpu/cpu.stat | grep nr_throttled
Database Query & Connection Pool Analysis
Check database slow query logs and active lock trees. Frequently, a new feature introduces an unindexed query (causing full table scans) or an N+1 query pattern where a loop issues 50 sequential queries instead of a batch fetch.
-- PostgreSQL lock inspection query
SELECT pid, age(clock_timestamp(), query_start), usename, query, state
FROM pg_stat_activity
WHERE state != 'idle' AND age(clock_timestamp(), query_start) > interval '2 seconds';
Cgroup CPU Throttling & Memory Pressure
Verify whether the new image introduced heavier memory consumption causing frequent JVM/Go GC cycles or exceeded Kubernetes CPU limits resulting in CFS throttling.
- Halt rollout immediately or route traffic back to the prior stable version while retaining one canaried instance for debugging.
- Compare APM distributed traces (flame graphs) between v1 and v2 to isolate the exact function or query responsible.
- Check database slow query logs for missing indexes or sequential N+1 query patterns.
- Examine container cgroup metrics for CFS CPU throttling (nr_throttled) and GC execution duration.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when addressing post-deployment SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
-
Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when addressing step-by-step
Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (
df -h) shows ample free space. - Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
-
Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping
Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (
pid_max).