⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 129 of 207 in Kubernetes
Staff SRE / Principal Architect [L3] Kubernetes Additional Kubernetes Scenarios (Q101-Q200) Staff SRE Scenario [L3]
🎯 Target Role / Context: Kubernetes Scenario #119 (L3) · Additional Kubernetes Scenarios (Q101-Q200)

Q: How do you implement multi-cluster service discovery so Service A in cluster 1 can discover and call Service B in cluster 2 with low latency, security, and DNS resolution?

Architecture and networking patterns for Kubernetes multi-cluster service discovery: evaluating the Kubernetes MCS API (ServiceExport/ServiceImport), Cilium ClusterMesh with eBPF, Submariner Lighthouse, and Istio multi-primary meshes.

#kubernetes multi cluster service discovery #multi cluster kubernetes #Cilium ClusterMesh #Submariner #ServiceExport ServiceImport #MCS API #Istio East-West Gateway #Kubernetes #Networking #SRE
🎙️ Candidate Opening & Architectural Context
""In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "In one of our high-traffic production clusters, our SRE team handled this incident using a standardized runbook. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."

1️⃣

Kubernetes SIG Multi-Cluster Services API (MCS API - KEP-1645)

The standardized Kubernetes cloud-native approach introduces two core Custom Resource Definitions:

  • ServiceExport: Created in Cluster 2 on Service B (apiVersion: multicluster.x-k8s.io/v1alpha1) to declare that the service should be visible to other clusters in the cluster set.
  • ServiceImport: Automatically synchronized into Cluster 1 by the multi-cluster controller. It assigns a ClusterSetIP.
  • DNS Resolution: Pods in Cluster 1 discover Service B using the domain service-b.namespace.svc.clusterset.local, resolved automatically by CoreDNS.
2️⃣

Cilium ClusterMesh (High-Performance eBPF Routing)

If clusters share routable VPC subnets with non-overlapping pod CIDRs, Cilium ClusterMesh provides the highest throughput:

  • Direct Pod-to-Pod Routing: ClusterMesh synchronizes etcd/kvstore state between clusters, allowing eBPF datapath programs to route packets directly across VPC peering or AWS Transit Gateway.
  • Global Services: Simply annotate Service B with io.cilium/global-service: 'true'. Any client in either cluster querying service-b.default.svc.cluster.local automatically load balances traffic across healthy endpoints in both clusters.
Advertisement
3️⃣

Submariner for Encrypted Hybrid/Multi-Cloud Networks

When clusters are hosted across different cloud providers or on-premises networks with non-routable private IPs:

  • WireGuard / IPSec Gateways: Submariner deploys gateway nodes that automatically negotiate encrypted tunnels between disparate VPCs.
  • Lighthouse CoreDNS Plugin: Provides MCS-compliant DNS resolution under clusterset.local, forwarding requests through the gateway tunnels directly.
4️⃣

Istio Multi-Primary Mesh with East-West Gateways

For zero-trust mTLS encryption, fine-grained traffic shifting, and canary cutovers across clusters:

  • Each cluster runs its own Istio control plane, sharing a common root CA.
  • Cross-cluster traffic passes through dedicated East-West Ingress Gateways using Envoy Server Name Indication (SNI) proxying without exposing pods directly to external networks.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"For same-cloud or VPC-peered clusters, Cilium ClusterMesh delivers unbeatable zero-overhead eBPF routing. For heterogeneous multi-cloud environments, Kubernetes MCS API combined with Submariner or Istio East-West gateways provides standardized clusterset.local discovery."
⚡ 60-Second Elevator Pitch Talking Points
  • Leveraged Kubernetes MCS API standardizing on ServiceExport and ServiceImport CRDs under clusterset.local.
  • Deployed Cilium ClusterMesh for high-throughput, low-latency eBPF pod-to-pod routing across VPC peered clusters.
  • Used global-service annotations to distribute failover traffic across active-active clusters.
  • Secured cross-cluster egress with mTLS via Istio East-West gateways for zero-trust compliance.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Configuring Service selector labels that mismatch underlying Pod manifest metadata when configuring Kubernetes Multi Cluster A typo in spec.selector causes the Service endpoints list (kubectl get endpoints) to remain empty, resulting in 503 Service Unavailable or timeouts.
  2. Omitting explicit ingressClassName specifications when running multiple ingress controllers when configuring Kubernetes Multi Cluster Without an explicit ingressClassName, ingress resources are either ignored or picked up unpredictably by competing controllers.
  3. Depleting worker node subnet IP address allocations under VPC CNI or Azure CNI when configuring Kubernetes Multi Cluster Direct VPC pod networking allocates real subnet IPs; failing to monitor IP pools causes new pods to stay stuck indefinitely in ContainerCreating.
  4. Attempting network triage without inspecting kubectl describe service and EndpointSlice resources when configuring Kubernetes Multi Cluster Verifying EndpointSlice population immediately determines whether a connectivity failure is caused by pod labeling, readiness probes, or routing.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →