Q: 504 errors on the playback service. Cloud LB shows healthy, mesh sidecars pass, but users can’t stream. Triage.
Master production incident triage for a distributed streaming failure where Cloud Load Balancers report healthy target groups, mesh sidecars show green, yet end users experience 504 Gateway Timeouts.
Want to master this scenario in a live sandbox? The Linux Foundation's Prometheus Certified Associate (PCA) & Monitoring Labs covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: During the premiere of a major title, users flood customer support: video streams freeze and return HTTP 504 Gateway Timeouts. The Cloud Load Balancer (ALB) dashboard reports 100% healthy backend target instances, and internal Kubernetes Istio/Envoy sidecars report normal HTTP 200 health check responses. Yet, video playback requests fail. We must trace the end-to-end request flow to expose why health checks deceive the load balancer.
The Root Illusion: Health Check vs Video Stream Decoupling
Understand why health check metrics diverge completely from user traffic:
- ALB target health checks query a lightweight endpoint (e.g.
GET /healthz) which returns in 2ms from memory. - Actual playback requests execute high-overhead operations: manifest generation (
.mpd / .m3u8), DRM license token verification, and CDN origin segment fetches. - The container has sufficient thread capacity to answer the 2ms health check, while all streaming worker threads are blocked waiting on slow downstream dependencies.
Inspect ALB and Envoy Idle Timeout Race Conditions
A 504 Gateway Timeout means a proxy in the path gave up waiting for an upstream response:
# 1. Inspect ALB Access Logs for upstream latency and target processing time
# Fields: target_processing_time, request_processing_time, response_processing_time
cat alb-access.log | awk '{print $9, $10, $11, $13}' | grep "504"
# 2. Check Envoy upstream request timeout metrics
kubectl exec -it <playback-pod> -c istio-proxy -- \
curl -s localhost:15000/stats | grep "upstream_rq_timeout\|upstream_cx_destroy_local"
- If ALB idle timeout is 60 seconds, and backend playback service manifest generation takes 62 seconds due to DRM database locks, the ALB closes the connection with 504.
- Conversely, if Envoy's route timeout (default 15s) is shorter than the client request expectation, Envoy returns 504 with response flag
UT(Upstream Timeout).
Trace the Blocking Dependency via Distributed Tracing (OpenTelemetry / Jaeger)
Inspect distributed traces for the playback session span:
- DRM License Token Service: Is the external Widevine/FairPlay key exchange service hitting rate limits?
- Object Storage / S3 Egress: Are S3 GET requests for video manifest chunks experiencing 503 SlowDown or NAT Gateway bandwidth saturation?
- Egress Connection Pool Starvation: Did the playback service exhaust outbound HTTP client connection pool sockets connecting to the metadata store?
Immediate Mitigation Actions
Execute tactical containment during live incident response:
- Graceful Degradation: Shed non-critical calls (e.g. disable real-time viewing history and personalized bitrate recommendations) to free up playback thread pools.
- Static Fallback Manifests: Serve pre-computed static video manifests from edge CDN cache rather than dynamic computation.
- Tune Keep-Alive & Timeouts: Ensure backend idle timeouts exceed load balancer idle timeouts to eliminate silent connection drops.
- A 504 Gateway Timeout while health checks pass means the health endpoint is decoupled from real work: it returns 200 OK while worker threads are saturated on streaming dependencies.
- First, I inspect the ALB access log fields: if 'target_processing_time' exceeds 60s, the ALB timed out waiting on the backend. If it's 15s, an intermediate Envoy proxy timeout triggered the 504.
- Second, I pull OpenTelemetry traces for the failing playback endpoint to pinpoint the bottleneck—typically DRM license verification, S3 API throttling, or egress DB pool exhaustion.
- To mitigate immediately, we activate graceful degradation: shed personalization and analytics dependencies to reclaim worker threads, and serve pre-generated static manifests from CDN edge cache.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Playback Stream Timeouts 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Playback Stream Timeouts Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Playback Stream Timeouts Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Playback Stream Timeouts Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.