Q: Your microservices on a custom Docker bridge network experience intermittent 5,000ms latency spikes when making external API calls to AWS S3 and payment gateways. Application logs show sporadic `getaddrinfo EAI_AGAIN` or `dns lookup timed out` errors. Direct `curl` commands to external IP addresses respond in 15ms. You must trace the container's DNS resolution path through the embedded Docker DNS server at `127.0.0.11`, identify UDP packet drops and search domain amplification, and implement a resilient DNS configuration.
Diagnose intermittent 5-second DNS resolution timeouts inside Docker containers. Understand Docker's internal 127.0.0.11 resolver, resolv.conf ndots:5 search path amplification, and UDP socket drops.
resolv.conf ndots:5 search path amplification, and UDP socket drops."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Diagnose intermittent 5-second DNS resolution timeouts inside Docker containers. Understand Docker's internal 127.0.0.11 resolver, resolv.conf ndots:5 search path amplification, and UDP socket drops.
Understand Docker Embedded DNS Resolver Mechanics
On user-defined bridge networks, Docker injects an internal DNS server listening at 127.0.0.11:53. This server intercepts DNS queries: container names are resolved via internal Docker daemon records, while external queries are forwarded to upstream DNS servers listed in the host's /etc/resolv.conf.
<!-- Docker DNS Path -->
Container App (getaddrinfo)
└── Queries: 127.0.0.11:53 (Docker Embedded DNS)
├── Matches service name? -> Returns container IP (172.18.0.4)
└── External domain? -> Forwards UDP packet to Host DNS (e.g., 8.8.8.8)
Diagnose the 5-Second Timeout and ndots Search Amplification
Standard Linux glibc resolvers append every domain listed in the search directive of /etc/resolv.conf if the query contains fewer dots than ndots (default is often 5 in container environments). Querying api.stripe.com generates redundant queries like api.stripe.com.default.svc.cluster.local, causing upstream rate limiting or dropped UDP packets that trigger a 5-second timeout.
# Inspect container resolv.conf
docker exec -it my-app cat /etc/resolv.conf
# Sample output:
# nameserver 127.0.0.11
# options ndots:5
# Trace DNS resolution using dig and tcpdump
docker exec -it my-app dig +trace api.stripe.com
Capture DNS Packets and Detect Conntrack Race Conditions
Use tcpdump inside the container or on the bridge interface. High-concurrency DNS lookups often trigger a known Linux kernel conntrack race condition where simultaneous A and AAAA DNS queries collide on the same UDP socket.
# Monitor DNS traffic on Docker bridge interface
sudo tcpdump -i br-1a2b3c4d -n "udp port 53" -vv
Configure Explicit Upstream DNS and Single-Request Options
Fix the issue by specifying reliable upstream DNS servers and passing single-request-reopen via --dns-opt to force sequential UDP lookups, or lowering ndots.
# Docker run with hardened DNS options
docker run -d \
--name resilient-api \
--dns 1.1.1.1 \
--dns 8.8.8.8 \
--dns-opt ndots:2 \
--dns-opt single-request-reopen \
--dns-opt timeout:2 \
my-api:v1.0
# Or globally in /etc/docker/daemon.json:
# {
# "dns": ["1.1.1.1", "8.8.8.8"],
# "dns-opts": ["single-request-reopen", "timeout:2"]
# }
127.0.0.11. High-concurrency lookups suffer 5-second timeouts due to ndots search list amplification and concurrent A/AAAA UDP socket collisions in kernel conntrack. Setting single-request-reopen and configuring explicit DNS servers resolves the latency."
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- d
- i
- n
- t
- e
- r
- m
- i
- t
- t
- e
- n
- t
- 5
- -
- s
- e
- c
- o
- n
- d
- l
- a
- t
- e
- n
- c
- y
- s
- p
- i
- k
- e
- s
- i
- n
- e
- x
- t
- e
- r
- n
- a
- l
- A
- P
- I
- c
- a
- l
- l
- s
- b
- y
- c
- a
- p
- t
- u
- r
- i
- n
- g
- D
- N
- S
- q
- u
- e
- r
- i
- e
- s
- o
- n
- D
- o
- c
- k
- e
- r
- b
- r
- i
- d
- g
- e
- n
- e
- t
- w
- o
- r
- k
- s
- .
- T
- h
- e
- i
- s
- s
- u
- e
- s
- t
- e
- m
- m
- e
- d
- f
- r
- o
- m
- g
- l
- i
- b
- c
- a
- p
- p
- e
- n
- d
- i
- n
- g
- s
- e
- a
- r
- c
- h
- d
- o
- m
- a
- i
- n
- s
- f
- o
- r
- e
- x
- t
- e
- r
- n
- a
- l
- F
- Q
- D
- N
- s
- a
- n
- d
- c
- o
- n
- c
- u
- r
- r
- e
- n
- t
- A
- /
- A
- A
- A
- A
- s
- o
- c
- k
- e
- t
- c
- o
- l
- l
- i
- s
- i
- o
- n
- s
- i
- n
- L
- i
- n
- u
- x
- c
- o
- n
- n
- t
- r
- a
- c
- k
- .
- C
- o
- n
- f
- i
- g
- u
- r
- i
- n
- g
- `
- s
- i
- n
- g
- l
- e
- -
- r
- e
- q
- u
- e
- s
- t
- -
- r
- e
- o
- p
- e
- n
- `
- a
- n
- d
- t
- u
- n
- i
- n
- g
- `
- n
- d
- o
- t
- s
- :
- 2
- `
- i
- n
- `
- d
- a
- e
- m
- o
- n
- .
- j
- s
- o
- n
- `
- c
- o
- m
- p
- l
- e
- t
- e
- l
- y
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- t
- h
- e
- D
- N
- S
- t
- i
- m
- e
- o
- u
- t
- s
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Relying on legacy bucket ACLs instead of modern S3 Block Public Access and IAM bucket policies AWS recommends disabling ACLs (Bucket Owner Enforced) in favor of centralized declarative bucket policies and account-level Block Public Access controls.
-
Confusing SSE-S3 encryption with KMS customer-managed key (SSE-KMS) authorization policies
Accessing SSE-KMS encrypted objects requires both
s3:GetObjectandkms:Decryptpermissions; neglecting KMS key policies causes silent 403 Forbidden errors. - Using sequential timestamps or predictable prefixes for high-throughput S3 read and write keys Sequential key prefixes limit S3 horizontal partitioning; randomized hash prefixes distribute workloads across partitions to achieve 3,500 PUT and 5,500 GET requests/sec.
- Failing to audit S3 object delete markers when troubleshooting missing versioned objects in docker-embedded-dns-127-0-0-11-timeouts Issuing a DELETE on a versioned bucket only adds a delete marker; engineers must retrieve previous versions or purge markers to restore data.