⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Databases & Storage Interview Questions Scenario 29 of 54 in Databases & Storage
Staff Cloud Architect Database Distributed Data & Event Streaming Distributed Streaming
🎯 Target Role / Context: Staff Distributed Systems / Streaming Platform Engineer Scenario

Q: Your enterprise runs transactional order services in AWS and analytics engines in GCP. You need real-time, bi-directional event streaming between AWS Kafka (MSK) and GCP Kafka. How do you design MirrorMaker 2 to replicate topics across clouds without causing infinite replication loops, and how do you handle consumer group offset migration?

Engineering a resilient active-active cross-cloud event streaming platform with Apache Kafka MirrorMaker 2 (MM2) running between AWS MSK and GCP Strimzi Kafka with cycle detection and offset translation.

#Multi-Cloud #Kafka #MirrorMaker 2 #AWS #GCP #Active-Active #Event-Driven
🎙️ Candidate Opening & Architectural Context
"Uncoordinated bi-directional replication between clouds can easily enter an infinite feedback loop where topic messages mirror back and forth indefinitely. We implemented Apache Kafka MirrorMaker 2 with active namespace prefixing and automated offset translation."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Uncoordinated bi-directional replication between clouds can easily enter an infinite feedback loop where topic messages mirror back and forth indefinitely. We implemented Apache Kafka MirrorMaker 2 with active namespace prefixing and automated offset translation.

1️⃣

Establish Kafka Topologies & Private Cross-Cloud Network

Connect AWS MSK and GCP Kafka clusters via encrypted private peering:

  • Clusters: Configured aws-cluster (AWS MSK in us-east-1) and gcp-cluster (Strimzi Kafka on GKE in us-central1).
  • Network Transport: Routed traffic across private interconnect with mutual TLS (mTLS) authentication and TLS 1.3 encryption.
Pro Tip: Replicating Kafka traffic over private interconnect ensures low-latency byte transfer without incurring public cloud IP transit vulnerabilities.
2️⃣

Configure MirrorMaker 2 Connectors with Automated Topic Renaming

Deploy MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector:

  • Source Connector: Replicates topics from aws-cluster into GCP with prefixed topic names: aws-cluster.orders.
  • Cycle Prevention: MM2 automatically detects source prefixes and refuses to mirror a topic back to its originating cluster, preventing infinite replication loops.
Pro Tip: Topic prefixing (e.g. source-cluster.topic-name) is the foundational architectural mechanism that guarantees loop prevention in active-active topologies.
Advertisement
3️⃣

Configure MirrorCheckpointConnector for Consumer Offset Synchronization

Translate consumer group offsets between disparate Kafka cluster partitions:

  • Checkpoint Connector: Emits consumer offset checkpoints every 60 seconds into the internal checkpoints.internal topic.
  • Offset Translation: Because partition offsets differ between clusters, MM2 maps consumer group progress so consumers failing over to GCP can resume from the exact logical record.
Pro Tip: Without offset translation, failing over a consumer group across clouds results in either massive message replay or data loss.
4️⃣

Deploy Heartbeat Probes & End-to-End Replication Lag Telemetry

Track cross-cloud replication latency and connector health:

  • Heartbeat Connector: Publishes round-trip timestamps to heartbeats topic every 5 seconds.
  • Prometheus Telemetry: Monitored kafka_consumergroup_lag and replication-latency-ms; alerted in Grafana if cross-cloud replication lag exceeded 1,200ms.
Pro Tip: Heartbeat metrics measure actual end-to-end delivery latency, alerting on network congestion before consumer lag accumulates.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Apache Kafka MirrorMaker 2 enables active-active cross-cloud event streaming by using namespace prefixes to prevent replication cycles and checkpoint connectors to translate consumer offsets."
⚡ 60-Second Elevator Pitch Talking Points
  • Connect AWS and GCP Kafka clusters via encrypted private interconnect with mTLS.
  • Deploy MirrorMaker 2 with MirrorSourceConnector using cluster namespace prefixing.
  • Use MirrorCheckpointConnector to synchronize and translate consumer group offsets.
  • Monitor cross-cloud replication lag and heartbeats in Prometheus to ensure sub-second sync.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Applying exclusive-lock DDL table migrations on live production databases during peak traffic windows when configuring Cross Event Streaming DDL migrations that acquire an exclusive table lock block all concurrent queries, queuing connections until the database connection pool exhausts.
  2. Allowing high-frequency queries to trigger full table scans without appropriate composite indexing when configuring Cross Event Streaming Sequential table scans saturate database CPU and disk I/O, evicting frequently queried data from the buffer cache and causing cascading timeouts.
  3. Disabling referential integrity constraints during bulk ETL imports without post-import consistency validation when configuring Cross Event Streaming Skipping relational consistency checks risks introducing orphaned records and referential integrity violations into production tables.
  4. Relying on database snapshot backups that have never been verified through automated restoration drills when configuring Cross Event Streaming Backups that have never been tested with automated restore verification frequently fail during real disaster recovery scenarios.
Advertisement
Want more Databases & Storage scenarios?
Explore our complete collection of scenario-based Databases & Storage interview runbooks.
Browse All Databases & Storage Questions →