Q: You're designing a microservices architecture with 50 services. Each service is dynamically deployed by Kubernetes, with IPs changing hourly. How do you enable service discovery so one service can reliably reach another without hardcoding IPs or DNS names?
There are three main service discovery approaches, each with tradeoffs:
Want to master this scenario in a live sandbox? KodeKloud's Istio Service Mesh & Advanced Kubernetes Networking Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "Isolating network failures requires proving whether packets are dropped by route tables, security groups, or stateless NACLs. The interviewer is testing: Service discovery patterns, DNS vs API-based discovery, microservices networking.. I structure my answer around systematic triage first, root cause analysis second, and permanent remediation third."
Production Solution & Architecture
There are three main service discovery approaches, each with tradeoffs: 1. DNS-Based Discovery (Traditional): Kubernetes automatically registers each service in DNS: my-service.default.svc.cluster.local resolves to the service's cluster IP (virtual, stable). Pros: Simple, works with existing apps. Cons: DNS caching issues (Java caches DNS indefinitely), DNS TTL can cause stale endpoints, no realtime updates if a pod crashes mid-request. 2. API-Based Discovery (Service Mesh): Istio/Linkerd intercepts all outbound traffic via sidecar proxies. The proxy dynamically queries the service registry (etcd) for live endpoint lists, updating in realtime as pods scale up/down. Pros: Realtime, handles pod failures gracefully, circuit breaking, retries, mTLS. Cons: Complexity, 5-10% CPU overhead per pod, steep learning curve. 3. Hybrid (DNS + API): Use DNS for initial discovery, but rely on service mesh sidecars for active health checking and load balancing. Recommendation for 50 microservices: Start with Kubernetes native DNS (simplest, lowest overhead): Services call each other: curl http://payment-service:8080/charge. Kubernetes DNS resolves and load balances automatically. If you hit scaling issues (DNS TTL, pod crash recovery), adopt Istio for realtime discovery and traffic management.
Client → '`curl http://my-service:8080/api`' → Kubelet's DNS (CoreDNS) → Service ClusterIP → Round-robin load balance to Pod IPs
- Immediate Triage: There are three main service discovery approaches, each with tradeoffs:
- Run targeted verification commands before modifying configuration.
- Automate permanent guardrails (CI check, alerts, IaC policy) to prevent recurrence.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Treating stateless Network ACLs as stateful firewalls without specifying outbound ephemeral return ports when configuring Networking Youre designing Security Groups track connection state automatically, whereas stateless Network ACLs require explicit ephemeral port rules (1024-65535) for return traffic.
- Opening short-lived non-pooled TCP client connections, exhausting ephemeral ports in TIME_WAIT state when configuring Networking Youre designing Rapidly opening and closing short-lived TCP sockets without connection pooling exhausts ephemeral ports, causing connection refusal errors.
- Directing load balancer health checks to heavy computational endpoints rather than dedicated health routes when configuring Networking Youre designing Deep health check paths amplify database load during traffic spikes and risk cascading eviction of healthy application nodes.
- Triggering BGP route dampening by rapidly announcing and withdrawing multihomed network prefixes when configuring Networking Youre designing Rapidly flapping BGP route announcements triggers upstream ISP route dampening, effectively blackholing network traffic for hours.