Q: Explain how kube-proxy implements Services using iptables.
kube-proxy implements Services using iptables by watching Services and EndpointSlices and programming NAT rules that rewrite a Service's virtual ClusterIP to a randomly chosen pod IP (DNAT).
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>kube-proxy implements Services using iptables by watching Services and EndpointSlices and programming NAT rules that rewrite a Service's virtual ClusterIP to a randomly chosen pod IP (DNAT).</strong> There is no listening process on the ClusterIP: packets are rewritten in the kernel by netfilter before they leave the client's node. Interviewers ask this to check you understand that a Service is a set of rules, not a proxy, and that you can debug it by reading chains. A strong answer covers the chain layout, probabilistic load balancing, conntrack behaviour, masquerading, and the limits that led to IPVS and nftables modes.
Mental model: watch, translate, program
kube-proxy runs on every node as a controller with no data path of its own in iptables mode.
- Inputs: it watches Services and EndpointSlices through the API server, and only ready endpoints are included in rules.
- Output: it writes the
nattable rules, typically usingiptables-restoreto apply changes in batches, and re-syncs periodically and on changes. - Data path: traffic from pods or the host hits the
PREROUTINGorOUTPUTchain, jumps intoKUBE-SERVICES, matches the ClusterIP and port, then DNATs to a pod IP. - Replacement: some CNIs such as Cilium can replace kube-proxy entirely with eBPF, so first check which implementation your cluster uses.
- Why no listener: because rewriting happens in netfilter,
ss -lntpwill not show anything bound to the ClusterIP. That is why you cannot curl-debug the ClusterIP from a machine outside the cluster network.
Hands-on: how kube-proxy implements Services using iptables, read the chains
Dump the NAT table on a node and follow one Service through its chains.
# On a node (root). Find the Service's rules by ClusterIP
kubectl get svc my-app -o jsonpath='{.spec.clusterIP}'
sudo iptables -t nat -S KUBE-SERVICES | grep 10.96.12.34
# Follow the chain: service -> endpoints -> DNAT
sudo iptables -t nat -S KUBE-SVC-XXXXXXXXXXXXXXXX
# -A KUBE-SVC-X ... -m statistic --mode random --probability 0.33333 -j KUBE-SEP-AAA
# -A KUBE-SVC-X ... -m statistic --mode random --probability 0.50000 -j KUBE-SEP-BBB
# -A KUBE-SVC-X ... -j KUBE-SEP-CCC
sudo iptables -t nat -S KUBE-SEP-AAA
# -A KUBE-SEP-AAA -p tcp -m tcp -j DNAT --to-destination 10.244.1.7:8080
- KUBE-SERVICES: the entry chain with one rule per Service port, matching destination ClusterIP, protocol and port.
- KUBE-SVC-<hash>: one chain per Service. It holds one rule per endpoint, using the
statistic --mode random --probabilitymatch so each pod gets an approximately equal share. - KUBE-SEP-<hash>: one chain per endpoint, containing the
DNATrule topodIP:targetPort. - Other chains:
KUBE-NODEPORTShandles NodePort traffic, andKUBE-MARK-MASQmarks packets that need source NAT. - Probability maths: with three endpoints the first rule matches one third, the second matches one half of the remainder, and the last matches everything left, which yields equal shares. Ready-state changes rewrite these probabilities.
Return traffic, masquerading and conntrack
The first packet of a connection selects the endpoint. Everything after that relies on connection tracking.
# Inspect tracked connections to the Service (conntrack-tools)
sudo conntrack -L -d 10.96.12.34
# Which mode is kube-proxy in?
kubectl -n kube-system get cm kube-proxy -o yaml | grep -E 'mode:|syncPeriod'
kubectl -n kube-system logs ds/kube-proxy | head
- Conntrack: the kernel stores the DNAT mapping, so later packets and replies are translated back automatically and stick to the same pod for the life of the connection.
- Load balancing granularity: balancing is per connection, not per request. Long-lived HTTP/2 or gRPC connections can therefore pin to one pod.
- Masquerade:
KUBE-MARK-MASQandKUBE-POSTROUTINGperform SNAT for traffic that leaves the node or for hairpin cases, so replies return through the same node. - externalTrafficPolicy:
Localpreserves the client source IP and only routes to local endpoints, whileClustermay SNAT and balance across nodes. - Session affinity: with
sessionAffinity: ClientIPkube-proxy uses therecentiptables match to keep a client on the same endpoint for the configured timeout, overriding random selection.
Debugging and failure modes
Most Service problems are visible in endpoints and rules, not in kube-proxy itself.
- No endpoints:
kubectl get endpointslices -l kubernetes.io/service-name=my-app. An empty list means a selector mismatch or failing readiness, so there are no DNAT rules and the connection is rejected. - Stale rules: if kube-proxy is crashing or lagging, rules can stay out of date. Check its logs and the rule sync metrics.
- Scale cost: iptables rules are evaluated linearly and rebuilt as a whole, so very large numbers of Services increase update latency and CPU. That is the motivation for IPVS and nftables modes.
- Conntrack exhaustion: a full table drops new connections, so monitor conntrack usage on busy nodes and tune limits accordingly.
- Terminating endpoints: EndpointSlices mark endpoints ready, serving and terminating, which lets newer kube-proxy versions drain connections gracefully. Combine this with a
preStophook and readiness probes to avoid dropped requests during rollouts.
Alternatives and production practice
Know the other proxy modes and how to operate kube-proxy safely.
- IPVS mode: uses kernel IPVS hash tables with multiple scheduling algorithms, scaling better for large clusters. Newer Kubernetes releases add an nftables mode, so check release notes for the status in your version.
- Monitoring: scrape kube-proxy metrics such as sync duration and rule counts, and alert on sync errors.
- Upgrades: keep kube-proxy version aligned with the control plane skew policy.
- Troubleshooting order: Service, EndpointSlices, pod readiness, node rules, conntrack, NetworkPolicy, then CNI.
- kube-proxy watches Services and EndpointSlices and programs iptables NAT rules so the kernel rewrites a ClusterIP to a pod IP; no proxy process sits in the data path.
- Chains go KUBE-SERVICES to KUBE-SVC to KUBE-SEP, with random probability rules for balancing and conntrack keeping each connection on one pod.
- I verify with iptables -t nat -S and EndpointSlices, and know IPVS, nftables or eBPF replacements for large clusters.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Thinking kube-proxy forwards the traffic itself In iptables mode packets are rewritten by netfilter in the kernel; kube-proxy only programs rules.
- Expecting per-request load balancing Selection happens per connection, so long-lived gRPC or keep-alive connections stick to one pod.
- Debugging rules before checking endpoints A Service with no ready endpoints has no DNAT target, and the cause is selector or readiness.
- Ignoring conntrack limits A full conntrack table drops new connections even though iptables rules are correct.