Q: How does Kubernetes service discovery work internally?
Master Kubernetes service discovery internals: examine how CoreDNS resolves cluster.local queries, EndpointSlice controllers track pod health, and kube-proxy configures kernel netfilter rules for seamless intra-cluster traffic.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: In Kubernetes, pod IP addresses are completely ephemeral. When a pod crashes or reschedules, its IP changes. In an interview, I explain how Kubernetes service discovery decouples client services from ephemeral pod IPs using a two-tier discovery architecture: DNS-based name resolution via CoreDNS and kernel-level virtual IP packet routing via kube-proxy.
The Ephemeral Pod Problem & ClusterIP Virtual IP
Kubernetes introduces the Service resource with a stable virtual IP (ClusterIP) allocated from service-cluster-ip-range. Clients connect to the Service name or ClusterIP, completely insulated from pod recreation or auto-scaling.
# Inspect Service ClusterIP and port mappings
kubectl get svc payment-service -n production -o wide- Virtual Abstraction: ClusterIP is not bound to any physical interface; it exists solely as virtual IP rules managed in kernel memory.
- Decoupled Architecture: Frontend services connect to
payment-servicewithout needing to track underlying pod IP churn.
How Does CoreDNS Handle Kubernetes Service Discovery?
Every pod is provisioned with /etc/resolv.conf pointing its nameserver to CoreDNS (typically 10.96.0.10). CoreDNS dynamically resolves queries matching
# Inspecting /etc/resolv.conf inside a pod
nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local
options ndots:5- FQDN Resolution: Resolves short service names (
payment-service) by appending the search paths sequentially. - CoreDNS Architecture: CoreDNS watches the Kubernetes API for Service and Endpoint events, caching DNS records in memory for sub-millisecond lookup latency.
EndpointSlice Controller: Tracking Pod Readiness
The kube-controller-manager runs the EndpointSlice controller. It continuously reconciles Service label selectors against active Pod labels, maintaining EndpointSlice objects that contain the IP addresses of all pods passing readiness probes.
# Inspect EndpointSlices tracking active ready pod IPs
kubectl get endpointslices -l kubernetes.io/service-name=payment-service- Scalable Endpoint Slices: Replaced monolithic Endpoints objects to support clusters with thousands of pods without saturating apiserver bandwidth.
- Readiness Gating: Pods failing readiness checks are immediately removed from EndpointSlices, cutting off traffic before downtime occurs.
kube-proxy & Kernel Netfilter (iptables / IPVS / eBPF)
kube-proxy runs on every worker node as a DaemonSet. It watches Service and EndpointSlice changes and programs kernel packet routing rules to intercept connections to the ClusterIP.
# View iptables KUBE-SVC chains managing random pod balancing
sudo iptables -t nat -L KUBE-SVC-XYZ -n -v- iptables Mode: Uses statistical random probability rules to distribute connections across endpoints.
- IPVS Mode: Uses Linux IP Virtual Server hash tables with O(1) lookup complexity, ideal for clusters with >10,000 services.
- eBPF (Cilium): Bypasses iptables entirely, translating socket addresses directly at the socket connect syscall layer.
Kubernetes Service Discovery Routing Modes Compared
Architectural comparison of packet routing implementations powering Kubernetes service discovery:
Headless Services & Direct Peer-to-Peer Discovery
When clusterIP: None is specified, kube-proxy skips virtual IP programming entirely. CoreDNS returns the list of all individual pod A-records directly, enabling peer discovery for distributed systems like Kafka, Cassandra, and MongoDB.
# CoreDNS dig response for a Headless Service
dig +short redis-headless.production.svc.cluster.local
# Returns:
# 10.244.1.10
# 10.244.2.14
# 10.244.3.22- Direct Addressing: Clients receive direct pod IPs and establish custom client-side connection pooling and load balancing.
- Decoupled Addressing: Services assign a static virtual ClusterIP so clients are insulated from ephemeral pod IP recycling.
- DNS Name Resolution: CoreDNS resolves service names (e.g. payment.production.svc.cluster.local) via /etc/resolv.conf in the pod.
- Endpoint Health Tracking: The EndpointSlice controller automatically adds or removes pod IPs based on real-time readiness probe health.
- Kernel DNAT: kube-proxy programs iptables or IPVS on worker nodes to intercept ClusterIP traffic and load balance across pod IPs.
- Headless Services: Specifying clusterIP: None bypasses kube-proxy, returning direct pod IP A-records for distributed database peer discovery.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Attempting to ICMP ping a ClusterIP to verify service connectivity ClusterIPs are virtual routing rules in kernel memory; iptables only translates TCP/UDP ports, so ICMP ping packets are dropped.
- Ignoring CoreDNS query amplification caused by ndots:5 configurations ndots:5 forces external domain lookups (e.g. api.stripe.com) to search cluster.local paths first, creating 4-5 failed queries for every external call.
- Allowing pods without readiness probes into Service selectors Without readiness probes, pods starting up or initializing databases receive traffic immediately and return HTTP 502/503 errors.
- Using iptables mode on massive clusters with tens of thousands of services iptables has O(n) linear evaluation overhead; switching to IPVS or Cilium eBPF provides O(1) performance and slashes CPU usage.