Q: You see a gradual increase in latency across an NGINX ingress controller, but CPU and memory are completely stable. What’s your next move?
Deep dive into NGINX Ingress controller latency regressions occurring under stable CPU and memory utilization due to socket backlog saturation and upstream connection churn.
somaxconn), or DNS resolution delays on upstream service endpoints."
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When NGINX Ingress latency climbs steadily while CPU and memory remain flat, NGINX is not bottlenecking on computation. It is bottlenecking on socket connection management: upstream TCP connection thrashing (lack of upstream keepalives), worker connection pool exhaustion, Linux kernel socket listen backlog overflow (somaxconn), or DNS resolution delays on upstream service endpoints.
Check Upstream Keepalive Configuration (Connection Churn)
By default, NGINX Ingress establishes a brand new TCP connection (and TLS handshake) for EVERY request sent to backend pods unless upstream-keepalive-connections is explicitly enabled. Under high RPS, this triggers massive ephemeral port exhaustion and TCP handshake latency spikes while CPU remains low.
# Enable upstream keepalive in Ingress ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
upstream-keepalive-connections: "100"
upstream-keepalive-timeout: "60"
upstream-keepalive-requests: "10000"
Inspect Socket Listen Backlog & Drops (netstat -s / ss -lnt)
Check if the operating system is dropping incoming TCP SYN packets because the socket listen backlog is saturated.
kubectl exec -it <nginx-ingress-pod> -- netstat -s | grep -i "listen"
# Look for: "times the listen queue of a socket overflowed" or "SYNs to LISTEN sockets dropped"
Inspect NGINX Worker Connections & Event Loop Limits
Check whether NGINX is hitting worker-connections (default 16384). If saturated, new client requests queue up in OS buffers, causing perceived latency to soar.
kubectl logs <nginx-ingress-pod> -n ingress-nginx | grep -i "worker_connections"
Verify Dynamic Upstream Service Endpoint Churn
If backend application pods are frequently scaling or restarting, NGINX Ingress constantly reloads or queries the kube-apiserver endpoints controller. Enable Lua-based dynamic upstream routing to eliminate configuration reload latency.
- Inspect upstream keepalives: NGINX defaults to opening a fresh TCP connection per request, choking on handshakes.
- Check kernel socket listen queue overflows via netstat -s in the ingress pod.
- Verify NGINX worker-connections limits and tune net.core.somaxconn.
- Ensure dynamic endpoint routing is enabled to avoid frequent configuration reloads during autoscaling.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when configuring Gradual Latency Spike Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
- Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when configuring Gradual Latency Spike Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
- Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes when configuring Gradual Latency Spike Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
- Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes when configuring Gradual Latency Spike Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.