Q: During an Azure deployment, you receive intermittent DNS resolution issues. What can be the causes and how do you resolve them?
Root cause analysis and resolution runbook for intermittent DNS resolution timeouts during Azure application deployments.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Intermittent DNS failures in Azure usually stem from three architectural limits: hitting the Azure VM DNS resolution limit of 1024 packets per second (pps), CoreDNS connection throttling inside AKS, missing or improperly linked Azure Private DNS zones across VNets, or single-point-of-failure custom DNS virtual appliances.
Diagnose the 1024 PPS Azure WireServer DNS Limit
Every Azure VM interface has a hard limit of 1,024 UDP/TCP packets per second to Azure WireServer DNS (168.63.129.16). High-concurrency container clusters without local caching easily exceed this limit, causing Azure to drop subsequent DNS queries silently resulting in 5-second timeout retries.
# Check dropped DNS queries via Azure Network Watcher or VM metrics
# PromQL in AKS to check CoreDNS forward errors:
sum(rate(coredns_dns_request_duration_seconds_count[5m])) by (server)
Deploy NodeLocal DNSCache in AKS
Implement NodeLocal DNSCache as a DaemonSet. It runs a local DNS caching agent on every Kubernetes node listening on 169.254.20.10, intercepting DNS requests before they leave the node. This eliminates UDP conntrack races and cuts WireServer requests by over 90%.
# Enable NodeLocal DNSCache in AKS cluster
az aks update -g rg-banking -n aks-prod --enable-node-local-dns
Verify Azure Private DNS Zone VNet Link Configurations
If resolving internal private endpoints (e.g. privatelink.database.windows.net), check whether the VNet containing the deploying VMs has an active Virtual Network Link attached to the Private DNS Zone. If auto-registration is overloaded or links are missing across peered VNets, resolution intermittently falls back to public IPs.
az network private-dns link vnet list \
--zone-name "privatelink.database.windows.net" \
--resource-group rg-dns
Tune ndots and Single-Request-Options in resolv.conf
By default, Linux resolv.conf has ndots:5, forcing the resolver to search multiple cluster search domains before resolving external hostnames. Optimize pod DNS configurations with single-request-reopen and reduced ndots.
- Check if the workload exceeds Azure's 1,024 packets/second VM DNS WireServer ceiling.
- Deploy NodeLocal DNSCache in AKS to handle DNS caching locally on every worker node.
- Validate Azure Private DNS Zone VNet links across hub-and-spoke virtual networks.
- Optimize pod resolv.conf options with single-request-reopen and lower ndots values.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when addressing intermittent Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
- Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when addressing deployment Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
- Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
- Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.