Q: During an Azure deployment, you receive intermittent DNS resolution issues. What can be the causes and how do you resolve them?
Root cause analysis and resolution runbook for intermittent DNS resolution timeouts during Azure application deployments.
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Diagnose the 1024 PPS Azure WireServer DNS Limit
Every Azure VM interface has a hard limit of 1,024 UDP/TCP packets per second to Azure WireServer DNS (`168.63.129.16`). High-concurrency container clusters without local caching easily exceed this limit, causing Azure to drop subsequent DNS queries silently resulting in 5-second timeout retries.
# Check dropped DNS queries via Azure Network Watcher or VM metrics
# PromQL in AKS to check CoreDNS forward errors:
sum(rate(coredns_dns_request_duration_seconds_count[5m])) by (server)
Deploy NodeLocal DNSCache in AKS
Implement NodeLocal DNSCache as a DaemonSet. It runs a local DNS caching agent on every Kubernetes node listening on `169.254.20.10`, intercepting DNS requests before they leave the node. This eliminates UDP conntrack races and cuts WireServer requests by over 90%.
# Enable NodeLocal DNSCache in AKS cluster
az aks update -g rg-banking -n aks-prod --enable-node-local-dns
Verify Azure Private DNS Zone VNet Link Configurations
If resolving internal private endpoints (e.g. `privatelink.database.windows.net`), check whether the VNet containing the deploying VMs has an active Virtual Network Link attached to the Private DNS Zone. If auto-registration is overloaded or links are missing across peered VNets, resolution intermittently falls back to public IPs.
az network private-dns link vnet list \
--zone-name "privatelink.database.windows.net" \
--resource-group rg-dns
Tune ndots and Single-Request-Options in resolv.conf
By default, Linux `resolv.conf` has `ndots:5`, forcing the resolver to search multiple cluster search domains before resolving external hostnames. Optimize pod DNS configurations with `single-request-reopen` and reduced `ndots`.
- Check if the workload exceeds Azure's 1,024 packets/second VM DNS WireServer ceiling.
- Deploy NodeLocal DNSCache in AKS to handle DNS caching locally on every worker node.
- Validate Azure Private DNS Zone VNet links across hub-and-spoke virtual networks.
- Optimize pod resolv.conf options with single-request-reopen and lower ndots values.