Q: Your microservices on a custom Docker bridge network experience intermittent 5,000ms latency spikes when making external API calls to AWS S3 and payment gateways. Application logs show sporadic `getaddrinfo EAI_AGAIN` or `dns lookup timed out` errors. Direct `curl` commands to external IP addresses respond in 15ms. You must trace the container's DNS resolution path through the embedded Docker DNS server at `127.0.0.11`, identify UDP packet drops and search domain amplification, and implement a resilient DNS configuration.
Diagnose intermittent 5-second DNS resolution timeouts inside Docker containers. Understand Docker's internal 127.0.0.11 resolver, `resolv.conf` `ndots:5` search path amplification, and UDP socket drops.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Understand Docker Embedded DNS Resolver Mechanics
On user-defined bridge networks, Docker injects an internal DNS server listening at `127.0.0.11:53`. This server intercepts DNS queries: container names are resolved via internal Docker daemon records, while external queries are forwarded to upstream DNS servers listed in the host's `/etc/resolv.conf`.
<!-- Docker DNS Path -->
Container App (getaddrinfo)
└── Queries: 127.0.0.11:53 (Docker Embedded DNS)
├── Matches service name? -> Returns container IP (172.18.0.4)
└── External domain? -> Forwards UDP packet to Host DNS (e.g., 8.8.8.8)
Diagnose the 5-Second Timeout and ndots Search Amplification
Standard Linux glibc resolvers append every domain listed in the `search` directive of `/etc/resolv.conf` if the query contains fewer dots than `ndots` (default is often 5 in container environments). Querying `api.stripe.com` generates redundant queries like `api.stripe.com.default.svc.cluster.local`, causing upstream rate limiting or dropped UDP packets that trigger a 5-second timeout.
# Inspect container resolv.conf
docker exec -it my-app cat /etc/resolv.conf
# Sample output:
# nameserver 127.0.0.11
# options ndots:5
# Trace DNS resolution using dig and tcpdump
docker exec -it my-app dig +trace api.stripe.com
Capture DNS Packets and Detect Conntrack Race Conditions
Use `tcpdump` inside the container or on the bridge interface. High-concurrency DNS lookups often trigger a known Linux kernel conntrack race condition where simultaneous A and AAAA DNS queries collide on the same UDP socket.
# Monitor DNS traffic on Docker bridge interface
sudo tcpdump -i br-1a2b3c4d -n "udp port 53" -vv
Configure Explicit Upstream DNS and Single-Request Options
Fix the issue by specifying reliable upstream DNS servers and passing `single-request-reopen` via `--dns-opt` to force sequential UDP lookups, or lowering `ndots`.
# Docker run with hardened DNS options
docker run -d \
--name resilient-api \
--dns 1.1.1.1 \
--dns 8.8.8.8 \
--dns-opt ndots:2 \
--dns-opt single-request-reopen \
--dns-opt timeout:2 \
my-api:v1.0
# Or globally in /etc/docker/daemon.json:
# {
# "dns": ["1.1.1.1", "8.8.8.8"],
# "dns-opts": ["single-request-reopen", "timeout:2"]
# }
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- d
- i
- n
- t
- e
- r
- m
- i
- t
- t
- e
- n
- t
- 5
- -
- s
- e
- c
- o
- n
- d
- l
- a
- t
- e
- n
- c
- y
- s
- p
- i
- k
- e
- s
- i
- n
- e
- x
- t
- e
- r
- n
- a
- l
- A
- P
- I
- c
- a
- l
- l
- s
- b
- y
- c
- a
- p
- t
- u
- r
- i
- n
- g
- D
- N
- S
- q
- u
- e
- r
- i
- e
- s
- o
- n
- D
- o
- c
- k
- e
- r
- b
- r
- i
- d
- g
- e
- n
- e
- t
- w
- o
- r
- k
- s
- .
- T
- h
- e
- i
- s
- s
- u
- e
- s
- t
- e
- m
- m
- e
- d
- f
- r
- o
- m
- g
- l
- i
- b
- c
- a
- p
- p
- e
- n
- d
- i
- n
- g
- s
- e
- a
- r
- c
- h
- d
- o
- m
- a
- i
- n
- s
- f
- o
- r
- e
- x
- t
- e
- r
- n
- a
- l
- F
- Q
- D
- N
- s
- a
- n
- d
- c
- o
- n
- c
- u
- r
- r
- e
- n
- t
- A
- /
- A
- A
- A
- A
- s
- o
- c
- k
- e
- t
- c
- o
- l
- l
- i
- s
- i
- o
- n
- s
- i
- n
- L
- i
- n
- u
- x
- c
- o
- n
- n
- t
- r
- a
- c
- k
- .
- C
- o
- n
- f
- i
- g
- u
- r
- i
- n
- g
- `
- s
- i
- n
- g
- l
- e
- -
- r
- e
- q
- u
- e
- s
- t
- -
- r
- e
- o
- p
- e
- n
- `
- a
- n
- d
- t
- u
- n
- i
- n
- g
- `
- n
- d
- o
- t
- s
- :
- 2
- `
- i
- n
- `
- d
- a
- e
- m
- o
- n
- .
- j
- s
- o
- n
- `
- c
- o
- m
- p
- l
- e
- t
- e
- l
- y
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- t
- h
- e
- D
- N
- S
- t
- i
- m
- e
- o
- u
- t
- s
- .