⚡ ~/naveed Interview Prep
⚡ Portfolio Home ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Docker & Containers Interview Questions Scenario 121 of 158 in Docker & Containers
Senior DevOps Engineer Docker Container Runtime & Systems Engineering Production Scenario

Q: Your team runs a 12-node Docker Swarm cluster hosting an API gateway. Clients report sporadic 504 Gateway Timeouts during peak traffic. Investigation reveals that the Swarm Ingress Routing Mesh uses IPVS to distribute incoming connections across all nodes via the `ingress` overlay network. This SNATs (source NATs) all client traffic to an internal overlay IP (e.g., `10.255.0.x`), masking client real IPs from security firewalls and causing severe IPVS connection table drops during high connection churn. You must analyze the routing mesh mechanics, restore real client IPs, and configure host-mode port publishing.

Understand Docker Swarm's Ingress Routing Mesh, IPVS load balancing, source IP preservation, and how to bypass the routing mesh using host-mode port publishing to eliminate hairpin NAT bottlenecks.

#Docker #Docker Swarm #Networking #IPVS #Troubleshooting
🎙️ Candidate Opening & Architectural Context
"Understand Docker Swarm's Ingress Routing Mesh, IPVS load balancing, source IP preservation, and how to bypass the routing mesh using host-mode port publishing to eliminate hairpin NAT bottlenecks."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Step 1

Analyze Swarm Ingress Routing Mesh and IPVS Mechanics

Deconstruct how Docker Swarm routes ingress traffic: Any published port (`-p 80:80`) binds on all Swarm nodes. Incoming packets on node A are routed through the Linux kernel IPVS (IP Virtual Server) load balancer to a task running on node B via VXLAN overlay encapsulation, altering the source IP to an internal ingress IP.

<!-- Swarm Routing Mesh Path -->
Client (203.0.113.5) ---> Node A (:80)
                             │ (IPVS SNAT: Client IP converted to 10.255.0.2)
                             └─── VXLAN Overlay (Port 4789 UDP) ───> Node B Task Container
                                                                      (Container sees 10.255.0.2!)
Pro Tip: Analyze Swarm Ingress Routing Mesh and IPVS Mechanics
Step 2

Diagnose IPVS Connection Drops and Conntrack Table Exhaustion

Inspect IPVS connection states and kernel conntrack utilization inside the Swarm ingress sandbox network namespace.

# Inspect IPVS connections in ingress sandbox
ip netns exec $(ls /var/run/docker/netns | grep ingress) ipvsadm -ln

# Check conntrack table saturation on host
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
Pro Tip: Diagnose IPVS Connection Drops and Conntrack Table Exhaustion
Advertisement
Step 3

Bypass Routing Mesh Using Host-Mode Port Publishing

To preserve client real IPs and eliminate IPVS routing overhead, publish ports in `host` mode. In host mode, ports bind strictly to nodes where the container task is actively running, completely bypassing the Swarm ingress overlay network.

# docker-compose.yml (Swarm Stack)
version: '3.8'
services:
  api-gateway:
    image: nginx:alpine
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host  # Bypasses routing mesh, binds directly to host interface
      - target: 443
        published: 443
        protocol: tcp
        mode: host
    deploy:
      mode: global  # Runs one instance on every Swarm node
Pro Tip: Bypass Routing Mesh Using Host-Mode Port Publishing
Step 4

Configure Upstream Load Balancer Health Checking

Pair `mode: host` with an external AWS ALB or HAProxy load balancer configured to check health on each individual node's published port. When a task dies or moves, the upstream load balancer stops sending traffic to that node immediately.

Pro Tip: Configure Upstream Load Balancer Health Checking
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Docker Swarm's ingress routing mesh uses IPVS and VXLAN SNAT, which conceals client source IPs and adds conntrack overhead. Publishing ports with `mode: host` directly on the host interface preserves client IPs and delivers maximum throughput for edge gateways."
⚡ 60-Second Elevator Pitch Talking Points
  • W
  • e
  • r
  • e
  • s
  • o
  • l
  • v
  • e
  • d
  • s
  • p
  • o
  • r
  • a
  • d
  • i
  • c
  • t
  • i
  • m
  • e
  • o
  • u
  • t
  • s
  • a
  • n
  • d
  • c
  • l
  • i
  • e
  • n
  • t
  • I
  • P
  • m
  • a
  • s
  • k
  • i
  • n
  • g
  • i
  • n
  • D
  • o
  • c
  • k
  • e
  • r
  • S
  • w
  • a
  • r
  • m
  • b
  • y
  • s
  • h
  • i
  • f
  • t
  • i
  • n
  • g
  • o
  • u
  • r
  • e
  • d
  • g
  • e
  • g
  • a
  • t
  • e
  • w
  • a
  • y
  • s
  • f
  • r
  • o
  • m
  • t
  • h
  • e
  • i
  • n
  • g
  • r
  • e
  • s
  • s
  • r
  • o
  • u
  • t
  • i
  • n
  • g
  • m
  • e
  • s
  • h
  • t
  • o
  • `
  • m
  • o
  • d
  • e
  • :
  • h
  • o
  • s
  • t
  • `
  • p
  • o
  • r
  • t
  • p
  • u
  • b
  • l
  • i
  • s
  • h
  • i
  • n
  • g
  • .
  • B
  • y
  • p
  • a
  • i
  • r
  • i
  • n
  • g
  • g
  • l
  • o
  • b
  • a
  • l
  • s
  • e
  • r
  • v
  • i
  • c
  • e
  • d
  • e
  • p
  • l
  • o
  • y
  • m
  • e
  • n
  • t
  • s
  • w
  • i
  • t
  • h
  • e
  • x
  • t
  • e
  • r
  • n
  • a
  • l
  • l
  • o
  • a
  • d
  • b
  • a
  • l
  • a
  • n
  • c
  • i
  • n
  • g
  • ,
  • w
  • e
  • b
  • y
  • p
  • a
  • s
  • s
  • e
  • d
  • I
  • P
  • V
  • S
  • S
  • N
  • A
  • T
  • e
  • n
  • t
  • i
  • r
  • e
  • l
  • y
  • ,
  • s
  • l
  • a
  • s
  • h
  • i
  • n
  • g
  • n
  • e
  • t
  • w
  • o
  • r
  • k
  • l
  • a
  • t
  • e
  • n
  • c
  • y
  • a
  • n
  • d
  • r
  • e
  • s
  • t
  • o
  • r
  • i
  • n
  • g
  • t
  • r
  • u
  • e
  • c
  • l
  • i
  • e
  • n
  • t
  • I
  • P
  • v
  • i
  • s
  • i
  • b
  • i
  • l
  • i
  • t
  • y
  • a
  • c
  • r
  • o
  • s
  • s
  • o
  • u
  • r
  • l
  • o
  • g
  • g
  • i
  • n
  • g
  • a
  • n
  • d
  • r
  • a
  • t
  • e
  • -
  • l
  • i
  • m
  • i
  • t
  • i
  • n
  • g
  • t
  • i
  • e
  • r
  • s
  • .
Advertisement
Want more Docker & Containers scenarios?
Explore our complete collection of scenario-based Docker & Containers interview runbooks.
Browse All Docker & Containers Questions →