Q: How do you implement multi-cluster service discovery so Service A in cluster 1 can discover and call Service B in cluster 2 with low latency, security, and DNS resolution?
Architecture and networking patterns for Kubernetes multi-cluster service discovery: evaluating the Kubernetes MCS API (ServiceExport/ServiceImport), Cilium ClusterMesh with eBPF, Submariner Lighthouse, and Istio multi-primary meshes.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Kubernetes SIG Multi-Cluster Services API (MCS API - KEP-1645)
The standardized Kubernetes cloud-native approach introduces two core Custom Resource Definitions:
- ServiceExport: Created in Cluster 2 on Service B (
apiVersion: multicluster.x-k8s.io/v1alpha1) to declare that the service should be visible to other clusters in the cluster set. - ServiceImport: Automatically synchronized into Cluster 1 by the multi-cluster controller. It assigns a ClusterSetIP.
- DNS Resolution: Pods in Cluster 1 discover Service B using the domain
service-b.namespace.svc.clusterset.local, resolved automatically by CoreDNS.
Cilium ClusterMesh (High-Performance eBPF Routing)
If clusters share routable VPC subnets with non-overlapping pod CIDRs, Cilium ClusterMesh provides the highest throughput:
- Direct Pod-to-Pod Routing: ClusterMesh synchronizes etcd/kvstore state between clusters, allowing eBPF datapath programs to route packets directly across VPC peering or AWS Transit Gateway.
- Global Services: Simply annotate Service B with
io.cilium/global-service: 'true'. Any client in either cluster queryingservice-b.default.svc.cluster.localautomatically load balances traffic across healthy endpoints in both clusters.
Submariner for Encrypted Hybrid/Multi-Cloud Networks
When clusters are hosted across different cloud providers or on-premises networks with non-routable private IPs:
- WireGuard / IPSec Gateways: Submariner deploys gateway nodes that automatically negotiate encrypted tunnels between disparate VPCs.
- Lighthouse CoreDNS Plugin: Provides MCS-compliant DNS resolution under
clusterset.local, forwarding requests through the gateway tunnels directly.
Istio Multi-Primary Mesh with East-West Gateways
For zero-trust mTLS encryption, fine-grained traffic shifting, and canary cutovers across clusters:
- Each cluster runs its own Istio control plane, sharing a common root CA.
- Cross-cluster traffic passes through dedicated East-West Ingress Gateways using Envoy Server Name Indication (SNI) proxying without exposing pods directly to external networks.
- Leveraged Kubernetes MCS API standardizing on ServiceExport and ServiceImport CRDs under clusterset.local.
- Deployed Cilium ClusterMesh for high-throughput, low-latency eBPF pod-to-pod routing across VPC peered clusters.
- Used global-service annotations to distribute failover traffic across active-active clusters.
- Secured cross-cluster egress with mTLS via Istio East-West gateways for zero-trust compliance.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Configuring Service selector labels that mismatch underlying Pod manifest metadata when configuring Kubernetes Multi Cluster
A typo in
spec.selectorcauses the Service endpoints list (kubectl get endpoints) to remain empty, resulting in503 Service Unavailableor timeouts. -
Omitting explicit ingressClassName specifications when running multiple ingress controllers when configuring Kubernetes Multi Cluster
Without an explicit
ingressClassName, ingress resources are either ignored or picked up unpredictably by competing controllers. -
Depleting worker node subnet IP address allocations under VPC CNI or Azure CNI when configuring Kubernetes Multi Cluster
Direct VPC pod networking allocates real subnet IPs; failing to monitor IP pools causes new pods to stay stuck indefinitely in
ContainerCreating. - Attempting network triage without inspecting kubectl describe service and EndpointSlice resources when configuring Kubernetes Multi Cluster Verifying EndpointSlice population immediately determines whether a connectivity failure is caused by pod labeling, readiness probes, or routing.