⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 126 of 135 in Linux & SRE
Senior DevOps / SRE Linux SRE & Performance Engineering Barclays Classic

Q: A deployment completed successfully, but response time increased from 200 ms to 3 seconds. Walk me through your troubleshooting approach.

Step-by-step diagnostic workflow for triaging a 15x response time regression (200ms to 3s) immediately following a production deployment without catastrophic error spikes.

#Performance #Latency #APM #Troubleshooting #Barclays #SRE #Linux
🎙️ Candidate Opening & Architectural Context
"When a deployment completes with green health checks but latency explodes from 200ms to 3 seconds, my immediate priority is blast radius mitigation followed by differential tracing between the previous and current release versions."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When a deployment completes with green health checks but latency explodes from 200ms to 3 seconds, my immediate priority is blast radius mitigation followed by differential tracing between the previous and current release versions.

1

Immediate Blast Radius Containment

Check if a canary, blue-green, or rolling deployment is in progress. If traffic is still on a subset of pods, halt rollout immediately or divert traffic back to the known good baseline while preserving 1-2 regressed pods for live forensic debugging.

2

Distributed Tracing Differential Comparison

Open Datadog, Jaeger, or AWS X-Ray and compare latency flame graphs of the new release versus the previous tag. Pinpoint whether the extra 2.8 seconds is spent on external HTTP calls, database query lock waiting, or CPU execution.

# Inspect container CPU throttling and garbage collection pauses
kubectl top pod -l app=payment-service
cat /sys/fs/cgroup/cpu/cpu.stat | grep nr_throttled
Advertisement
3

Database Query & Connection Pool Analysis

Check database slow query logs and active lock trees. Frequently, a new feature introduces an unindexed query (causing full table scans) or an N+1 query pattern where a loop issues 50 sequential queries instead of a batch fetch.

-- PostgreSQL lock inspection query
SELECT pid, age(clock_timestamp(), query_start), usename, query, state 
FROM pg_stat_activity 
WHERE state != 'idle' AND age(clock_timestamp(), query_start) > interval '2 seconds';
4

Cgroup CPU Throttling & Memory Pressure

Verify whether the new image introduced heavier memory consumption causing frequent JVM/Go GC cycles or exceeded Kubernetes CPU limits resulting in CFS throttling.

Pro Tip: CFS throttling under strict CPU limits can stretch a 10ms CPU-bound calculation across 2,000ms of CFS quotas without maxing out host CPU.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Halt rollout or rollback first to preserve SLA. Use APM flame-graph diffing to instantly see whether the 2.8s jump is database N+1 queries, external API blocking, or cgroup CFS throttling."
⚡ 60-Second Elevator Pitch Talking Points
  • Halt rollout immediately or route traffic back to the prior stable version while retaining one canaried instance for debugging.
  • Compare APM distributed traces (flame graphs) between v1 and v2 to isolate the exact function or query responsible.
  • Check database slow query logs for missing indexes or sequential N+1 query patterns.
  • Examine container cgroup metrics for CFS CPU throttling (nr_throttled) and GC execution duration.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Issuing immediate kill -9 (SIGKILL) commands rather than allowing graceful termination via kill -15 (SIGTERM) when addressing post-deployment SIGKILL immediately halts execution without allowing processes to flush memory buffers, commit database transactions, or remove stale PID locks.
  2. Overlooking file system inode exhaustion (df -i) when diagnosing 'No space left on device' errors when addressing step-by-step Millions of small files or unrotated session tokens can exhaust available file system inodes even when standard disk storage (df -h) shows ample free space.
  3. Exhausting open file descriptor allocations (ulimit -n) under intensive concurrent socket connections Reaching file descriptor exhaustion throws 'Too many open files' errors, preventing services from accepting new TCP client socket connections.
  4. Permitting defunct zombie processes to accumulate without fixing parent process SIGCHLD waitpid reaping Zombie processes consume process table entries; accumulating zombies can exhaust the system maximum PID limit (pid_max).
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →