Q: Your team runs a 12-node Docker Swarm cluster hosting an API gateway. Clients report sporadic 504 Gateway Timeouts during peak traffic. Investigation reveals that the Swarm Ingress Routing Mesh uses IPVS to distribute incoming connections across all nodes via the `ingress` overlay network. This SNATs (source NATs) all client traffic to an internal overlay IP (e.g., `10.255.0.x`), masking client real IPs from security firewalls and causing severe IPVS connection table drops during high connection churn. You must analyze the routing mesh mechanics, restore real client IPs, and configure host-mode port publishing.
Understand Docker Swarm's Ingress Routing Mesh, IPVS load balancing, source IP preservation, and how to bypass the routing mesh using host-mode port publishing to eliminate hairpin NAT bottlenecks.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Analyze Swarm Ingress Routing Mesh and IPVS Mechanics
Deconstruct how Docker Swarm routes ingress traffic: Any published port (`-p 80:80`) binds on all Swarm nodes. Incoming packets on node A are routed through the Linux kernel IPVS (IP Virtual Server) load balancer to a task running on node B via VXLAN overlay encapsulation, altering the source IP to an internal ingress IP.
<!-- Swarm Routing Mesh Path -->
Client (203.0.113.5) ---> Node A (:80)
│ (IPVS SNAT: Client IP converted to 10.255.0.2)
└─── VXLAN Overlay (Port 4789 UDP) ───> Node B Task Container
(Container sees 10.255.0.2!)
Diagnose IPVS Connection Drops and Conntrack Table Exhaustion
Inspect IPVS connection states and kernel conntrack utilization inside the Swarm ingress sandbox network namespace.
# Inspect IPVS connections in ingress sandbox
ip netns exec $(ls /var/run/docker/netns | grep ingress) ipvsadm -ln
# Check conntrack table saturation on host
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
Bypass Routing Mesh Using Host-Mode Port Publishing
To preserve client real IPs and eliminate IPVS routing overhead, publish ports in `host` mode. In host mode, ports bind strictly to nodes where the container task is actively running, completely bypassing the Swarm ingress overlay network.
# docker-compose.yml (Swarm Stack)
version: '3.8'
services:
api-gateway:
image: nginx:alpine
ports:
- target: 80
published: 80
protocol: tcp
mode: host # Bypasses routing mesh, binds directly to host interface
- target: 443
published: 443
protocol: tcp
mode: host
deploy:
mode: global # Runs one instance on every Swarm node
Configure Upstream Load Balancer Health Checking
Pair `mode: host` with an external AWS ALB or HAProxy load balancer configured to check health on each individual node's published port. When a task dies or moves, the upstream load balancer stops sending traffic to that node immediately.
- W
- e
- r
- e
- s
- o
- l
- v
- e
- d
- s
- p
- o
- r
- a
- d
- i
- c
- t
- i
- m
- e
- o
- u
- t
- s
- a
- n
- d
- c
- l
- i
- e
- n
- t
- I
- P
- m
- a
- s
- k
- i
- n
- g
- i
- n
- D
- o
- c
- k
- e
- r
- S
- w
- a
- r
- m
- b
- y
- s
- h
- i
- f
- t
- i
- n
- g
- o
- u
- r
- e
- d
- g
- e
- g
- a
- t
- e
- w
- a
- y
- s
- f
- r
- o
- m
- t
- h
- e
- i
- n
- g
- r
- e
- s
- s
- r
- o
- u
- t
- i
- n
- g
- m
- e
- s
- h
- t
- o
- `
- m
- o
- d
- e
- :
- h
- o
- s
- t
- `
- p
- o
- r
- t
- p
- u
- b
- l
- i
- s
- h
- i
- n
- g
- .
- B
- y
- p
- a
- i
- r
- i
- n
- g
- g
- l
- o
- b
- a
- l
- s
- e
- r
- v
- i
- c
- e
- d
- e
- p
- l
- o
- y
- m
- e
- n
- t
- s
- w
- i
- t
- h
- e
- x
- t
- e
- r
- n
- a
- l
- l
- o
- a
- d
- b
- a
- l
- a
- n
- c
- i
- n
- g
- ,
- w
- e
- b
- y
- p
- a
- s
- s
- e
- d
- I
- P
- V
- S
- S
- N
- A
- T
- e
- n
- t
- i
- r
- e
- l
- y
- ,
- s
- l
- a
- s
- h
- i
- n
- g
- n
- e
- t
- w
- o
- r
- k
- l
- a
- t
- e
- n
- c
- y
- a
- n
- d
- r
- e
- s
- t
- o
- r
- i
- n
- g
- t
- r
- u
- e
- c
- l
- i
- e
- n
- t
- I
- P
- v
- i
- s
- i
- b
- i
- l
- i
- t
- y
- a
- c
- r
- o
- s
- s
- o
- u
- r
- l
- o
- g
- g
- i
- n
- g
- a
- n
- d
- r
- a
- t
- e
- -
- l
- i
- m
- i
- t
- i
- n
- g
- t
- i
- e
- r
- s
- .