⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Linux & SRE Interview Questions Scenario 90 of 135 in Linux & SRE
Staff SRE / Principal Architect [L3] Linux Linux / SRE — Scenario-Based Interview Questions Staff SRE Scenario [L3]
🎯 Target Role / Context: Linux Scenario #49 (L3) · Linux / SRE — Scenario-Based Interview Questions

Q: Your application maintains persistent TCP connections to a backend service. The backend silently crashes and restarts behind a load balancer, but the application continues to hold stale connections and times out. How do you configure TCP keepalive at the kernel level to detect dead connections faster?

TCP keepalive is a kernel-level mechanism that sends periodic probe packets on idle connections to verify the remote end is still alive.

#Linux #L3 #SRE #Systems #Troubleshooting
🎙️ Candidate Opening & Architectural Context
""During an on-call shift, our alerts triggered when a critical Linux production server exhibited this behavior. The interviewer is testing: TCP keepalive tuning, sysctl networking, connection health.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "During an on-call shift, our alerts triggered when a critical Linux production server exhibited this behavior. The interviewer is testing: TCP keepalive tuning, sysctl networking, connection health.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."

1️⃣

Initial Diagnostics & Root Cause Analysis

TCP keepalive is a kernel-level mechanism that sends periodic probe packets on idle connections to verify the remote end is still alive.

  • net.ipv4.tcp_keepalive_time = 7200 (Wait 2 hours before sending the first probe)
  • net.ipv4.tcp_keepalive_intvl = 75 (75 seconds between probes)
  • net.ipv4.tcp_keepalive_probes = 9 (Send 9 probes before declaring dead)
2️⃣

Remediation & Permanent Safeguards

The default Linux settings are extremely conservative: With defaults, it takes 2 hours + 9×75s = ~2h11m to detect a dead connection. For production, I would aggressively tune: Now dead connections are detected in 60 + 6×10 = 120 seconds. These settings are persisted in /etc/sysctl.conf. Note: the application must also enable SO_KEEPALIVE on its sockets for these kernel settings to take effect.

sysctl -w net.ipv4.tcp_keepalive_time=60
sysctl -w net.ipv4.tcp_keepalive_intvl=10
sysctl -w net.ipv4.tcp_keepalive_probes=6
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: net.ipv4.tcp_keepalive_time = 7200 (Wait 2 hours before sending the first probe)."
⚡ 60-Second Elevator Pitch Talking Points
  • net.ipv4.tcp_keepalive_time = 7200 (Wait 2 hours before sending the first probe)
  • net.ipv4.tcp_keepalive_intvl = 75 (75 seconds between probes)
  • net.ipv4.tcp_keepalive_probes = 9 (Send 9 probes before declaring dead)
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring application maintains persistent 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
  2. Setting target deregistration delay to zero instead of allowing connection draining when configuring application maintains persistent Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
  3. Failing to query CloudWatch ELB metrics (HTTPCode_ELB_5XX vs HTTPCode_Target_5XX) during load balancer outages when configuring application maintains persistent Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime.
  4. Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring application maintains persistent Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.
Advertisement
Want more Linux & SRE scenarios?
Explore our complete collection of scenario-based Linux & SRE interview runbooks.
Browse All Linux & SRE Questions →