Q: Your application maintains persistent TCP connections to a backend service. The backend silently crashes and restarts behind a load balancer, but the application continues to hold stale connections and times out. How do you configure TCP keepalive at the kernel level to detect dead connections faster?
TCP keepalive is a kernel-level mechanism that sends periodic probe packets on idle connections to verify the remote end is still alive.
Want to master this scenario in a live sandbox? The Linux Foundation's Certified System Administrator (LFCS) & SRE Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "During an on-call shift, our alerts triggered when a critical Linux production server exhibited this behavior. The interviewer is testing: TCP keepalive tuning, sysctl networking, connection health.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Initial Diagnostics & Root Cause Analysis
TCP keepalive is a kernel-level mechanism that sends periodic probe packets on idle connections to verify the remote end is still alive.
net.ipv4.tcp_keepalive_time = 7200(Wait 2 hours before sending the first probe)net.ipv4.tcp_keepalive_intvl = 75(75 seconds between probes)net.ipv4.tcp_keepalive_probes = 9(Send 9 probes before declaring dead)
Remediation & Permanent Safeguards
The default Linux settings are extremely conservative: With defaults, it takes 2 hours + 9×75s = ~2h11m to detect a dead connection. For production, I would aggressively tune: Now dead connections are detected in 60 + 6×10 = 120 seconds. These settings are persisted in /etc/sysctl.conf. Note: the application must also enable SO_KEEPALIVE on its sockets for these kernel settings to take effect.
sysctl -w net.ipv4.tcp_keepalive_time=60
sysctl -w net.ipv4.tcp_keepalive_intvl=10
sysctl -w net.ipv4.tcp_keepalive_probes=6
- net.ipv4.tcp_keepalive_time = 7200 (Wait 2 hours before sending the first probe)
- net.ipv4.tcp_keepalive_intvl = 75 (75 seconds between probes)
- net.ipv4.tcp_keepalive_probes = 9 (Send 9 probes before declaring dead)
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring application maintains persistent 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring application maintains persistent Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring application maintains persistent Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring application maintains persistent Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.