Q: Users resolve `api.myapp.com`. Half of them connect successfully to the new server, and half keep hitting the old deprecated server even though you changed the Route53 DNS record an hour ago. Why?
This is classic DNS caching behavior related to TTL (Time To Live).
#Networking #Networking #L2 #VPC #DNS #Security
🎙️ Candidate Opening & Architectural Context
""Isolating network failures requires proving whether packets are dropped by route tables, security groups, or stateless NACLs. The interviewer is testing: DNS propagation, TTL (Time To Live).. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third.""
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Production Solution & Architecture
This is classic DNS caching behavior related to TTL (Time To Live). When the original DNS record was created, it had a TTL (e.g., 24 hours). When Local ISPs, recursive resolvers (like 8.8.8.8), and user browsers resolve the domain, they cache the IP address for that duration. Even though you updated the authoritative server in Route53, the downstream internet caches will not query Route53 again until their local TTL expires. To prevent this in the future, you must lower the TTL on the old record to something short (e.g., 60 seconds) at least 24 hours *before* the migration, do the migration, and then raise the TTL back up on the new IP.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: This is classic DNS caching behavior related to TTL (Time To Live).."
⚡ 60-Second Elevator Pitch Talking Points
- Immediate Triage: This is classic DNS caching behavior related to TTL (Time To Live).
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Advertisement