Q: Users resolve `api.myapp.com`. Half of them connect successfully to the new server, and half keep hitting the old deprecated server even though you changed the Route53 DNS record an hour ago. Why?
This is classic DNS caching behavior related to TTL (Time To Live).
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Isolating network failures requires proving whether packets are dropped by route tables, security groups, or stateless NACLs. The interviewer is testing: DNS propagation, TTL (Time To Live).. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
This is classic DNS caching behavior related to TTL (Time To Live). When the original DNS record was created, it had a TTL (e.g., 24 hours). When Local ISPs, recursive resolvers (like 8.8.8.8), and user browsers resolve the domain, they cache the IP address for that duration. Even though you updated the authoritative server in Route53, the downstream internet caches will not query Route53 again until their local TTL expires. To prevent this in the future, you must lower the TTL on the old record to something short (e.g., 60 seconds) at least 24 hours *before* the migration, do the migration, and then raise the TTL back up on the new IP.
- Immediate Triage: This is classic DNS caching behavior related to TTL (Time To Live).
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when configuring Networking Users resolve Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
- Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when configuring Networking Users resolve Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
- Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes when configuring Networking Users resolve Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
- Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes when configuring Networking Users resolve Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.