⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 100 of 207 in Kubernetes
Staff SRE / Principal Architect [L3] Kubernetes Advanced Scenarios Staff SRE Scenario [L3]
🎯 Target Role / Context: Kubernetes Scenario #90 (L3) · Advanced Scenarios

Q: Explain how kube-proxy implements Services using iptables.

kube-proxy implements Services using iptables by watching Services and EndpointSlices and programming NAT rules that rewrite a Service's virtual ClusterIP to a randomly chosen pod IP (DNAT).

#Kubernetes #Advanced Scenarios #L3 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>kube-proxy implements Services using iptables by watching Services and EndpointSlices and programming NAT rules that rewrite a Service's virtual ClusterIP to a randomly chosen pod IP (DNAT).</strong> There is no listening process on the ClusterIP: packets are rewritten in the kernel by netfilter before they leave the client's node. Interviewers ask this to check you understand that a Service is a set of rules, not a proxy, and that you can debug it by reading chains. A strong answer covers the chain layout, probabilistic load balancing, conntrack behaviour, masquerading, and the limits that led to IPVS and nftables modes."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>kube-proxy implements Services using iptables by watching Services and EndpointSlices and programming NAT rules that rewrite a Service's virtual ClusterIP to a randomly chosen pod IP (DNAT).</strong> There is no listening process on the ClusterIP: packets are rewritten in the kernel by netfilter before they leave the client's node. Interviewers ask this to check you understand that a Service is a set of rules, not a proxy, and that you can debug it by reading chains. A strong answer covers the chain layout, probabilistic load balancing, conntrack behaviour, masquerading, and the limits that led to IPVS and nftables modes.

1️⃣

Mental model: watch, translate, program

kube-proxy runs on every node as a controller with no data path of its own in iptables mode.

  • Inputs: it watches Services and EndpointSlices through the API server, and only ready endpoints are included in rules.
  • Output: it writes the nat table rules, typically using iptables-restore to apply changes in batches, and re-syncs periodically and on changes.
  • Data path: traffic from pods or the host hits the PREROUTING or OUTPUT chain, jumps into KUBE-SERVICES, matches the ClusterIP and port, then DNATs to a pod IP.
  • Replacement: some CNIs such as Cilium can replace kube-proxy entirely with eBPF, so first check which implementation your cluster uses.
  • Why no listener: because rewriting happens in netfilter, ss -lntp will not show anything bound to the ClusterIP. That is why you cannot curl-debug the ClusterIP from a machine outside the cluster network.
2️⃣

Hands-on: how kube-proxy implements Services using iptables, read the chains

Dump the NAT table on a node and follow one Service through its chains.

# On a node (root). Find the Service's rules by ClusterIP
kubectl get svc my-app -o jsonpath='{.spec.clusterIP}'
sudo iptables -t nat -S KUBE-SERVICES | grep 10.96.12.34

# Follow the chain: service -> endpoints -> DNAT
sudo iptables -t nat -S KUBE-SVC-XXXXXXXXXXXXXXXX
#  -A KUBE-SVC-X ... -m statistic --mode random --probability 0.33333 -j KUBE-SEP-AAA
#  -A KUBE-SVC-X ... -m statistic --mode random --probability 0.50000 -j KUBE-SEP-BBB
#  -A KUBE-SVC-X ... -j KUBE-SEP-CCC
sudo iptables -t nat -S KUBE-SEP-AAA
#  -A KUBE-SEP-AAA -p tcp -m tcp -j DNAT --to-destination 10.244.1.7:8080
  • KUBE-SERVICES: the entry chain with one rule per Service port, matching destination ClusterIP, protocol and port.
  • KUBE-SVC-<hash>: one chain per Service. It holds one rule per endpoint, using the statistic --mode random --probability match so each pod gets an approximately equal share.
  • KUBE-SEP-<hash>: one chain per endpoint, containing the DNAT rule to podIP:targetPort.
  • Other chains: KUBE-NODEPORTS handles NodePort traffic, and KUBE-MARK-MASQ marks packets that need source NAT.
  • Probability maths: with three endpoints the first rule matches one third, the second matches one half of the remainder, and the last matches everything left, which yields equal shares. Ready-state changes rewrite these probabilities.
Advertisement
3️⃣

Return traffic, masquerading and conntrack

The first packet of a connection selects the endpoint. Everything after that relies on connection tracking.

# Inspect tracked connections to the Service (conntrack-tools)
sudo conntrack -L -d 10.96.12.34

# Which mode is kube-proxy in?
kubectl -n kube-system get cm kube-proxy -o yaml | grep -E 'mode:|syncPeriod'
kubectl -n kube-system logs ds/kube-proxy | head
  • Conntrack: the kernel stores the DNAT mapping, so later packets and replies are translated back automatically and stick to the same pod for the life of the connection.
  • Load balancing granularity: balancing is per connection, not per request. Long-lived HTTP/2 or gRPC connections can therefore pin to one pod.
  • Masquerade: KUBE-MARK-MASQ and KUBE-POSTROUTING perform SNAT for traffic that leaves the node or for hairpin cases, so replies return through the same node.
  • externalTrafficPolicy: Local preserves the client source IP and only routes to local endpoints, while Cluster may SNAT and balance across nodes.
  • Session affinity: with sessionAffinity: ClientIP kube-proxy uses the recent iptables match to keep a client on the same endpoint for the configured timeout, overriding random selection.
4️⃣

Debugging and failure modes

Most Service problems are visible in endpoints and rules, not in kube-proxy itself.

  • No endpoints: kubectl get endpointslices -l kubernetes.io/service-name=my-app. An empty list means a selector mismatch or failing readiness, so there are no DNAT rules and the connection is rejected.
  • Stale rules: if kube-proxy is crashing or lagging, rules can stay out of date. Check its logs and the rule sync metrics.
  • Scale cost: iptables rules are evaluated linearly and rebuilt as a whole, so very large numbers of Services increase update latency and CPU. That is the motivation for IPVS and nftables modes.
  • Conntrack exhaustion: a full table drops new connections, so monitor conntrack usage on busy nodes and tune limits accordingly.
  • Terminating endpoints: EndpointSlices mark endpoints ready, serving and terminating, which lets newer kube-proxy versions drain connections gracefully. Combine this with a preStop hook and readiness probes to avoid dropped requests during rollouts.
5️⃣

Alternatives and production practice

Know the other proxy modes and how to operate kube-proxy safely.

  • IPVS mode: uses kernel IPVS hash tables with multiple scheduling algorithms, scaling better for large clusters. Newer Kubernetes releases add an nftables mode, so check release notes for the status in your version.
  • Monitoring: scrape kube-proxy metrics such as sync duration and rule counts, and alert on sync errors.
  • Upgrades: keep kube-proxy version aligned with the control plane skew policy.
  • Troubleshooting order: Service, EndpointSlices, pod readiness, node rules, conntrack, NetworkPolicy, then CNI.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Remember that a ClusterIP is only rules in the kernel, so debug a broken Service by walking KUBE-SERVICES, KUBE-SVC and KUBE-SEP, and check EndpointSlices first."
⚡ 60-Second Elevator Pitch Talking Points
  • kube-proxy watches Services and EndpointSlices and programs iptables NAT rules so the kernel rewrites a ClusterIP to a pod IP; no proxy process sits in the data path.
  • Chains go KUBE-SERVICES to KUBE-SVC to KUBE-SEP, with random probability rules for balancing and conntrack keeping each connection on one pod.
  • I verify with iptables -t nat -S and EndpointSlices, and know IPVS, nftables or eBPF replacements for large clusters.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Thinking kube-proxy forwards the traffic itself In iptables mode packets are rewritten by netfilter in the kernel; kube-proxy only programs rules.
  2. Expecting per-request load balancing Selection happens per connection, so long-lived gRPC or keep-alive connections stick to one pod.
  3. Debugging rules before checking endpoints A Service with no ready endpoints has no DNAT target, and the cause is selector or readiness.
  4. Ignoring conntrack limits A full conntrack table drops new connections even though iptables rules are correct.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →