Q: Your enterprise runs transactional order services in AWS and analytics engines in GCP. You need real-time, bi-directional event streaming between AWS Kafka (MSK) and GCP Kafka. How do you design MirrorMaker 2 to replicate topics across clouds without causing infinite replication loops, and how do you handle consumer group offset migration?
Engineering a resilient active-active cross-cloud event streaming platform with Apache Kafka MirrorMaker 2 (MM2) running between AWS MSK and GCP Strimzi Kafka with cycle detection and offset translation.
#Multi-Cloud #Kafka #MirrorMaker 2 #AWS #GCP #Active-Active #Event-Driven
🎙️ Candidate Opening & Architectural Context
"Uncoordinated bi-directional replication between clouds can easily enter an infinite feedback loop where topic messages mirror back and forth indefinitely. We implemented Apache Kafka MirrorMaker 2 with active namespace prefixing and automated offset translation."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Establish Kafka Topologies & Private Cross-Cloud Network
Connect AWS MSK and GCP Kafka clusters via encrypted private peering:
- Clusters: Configured
aws-cluster(AWS MSK in us-east-1) andgcp-cluster(Strimzi Kafka on GKE in us-central1). - Network Transport: Routed traffic across private interconnect with mutual TLS (mTLS) authentication and TLS 1.3 encryption.
Pro Tip: Replicating Kafka traffic over private interconnect ensures low-latency byte transfer without incurring public cloud IP transit vulnerabilities.
2️⃣
Configure MirrorMaker 2 Connectors with Automated Topic Renaming
Deploy MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector:
- Source Connector: Replicates topics from
aws-clusterinto GCP with prefixed topic names:aws-cluster.orders. - Cycle Prevention: MM2 automatically detects source prefixes and refuses to mirror a topic back to its originating cluster, preventing infinite replication loops.
Pro Tip: Topic prefixing (e.g. source-cluster.topic-name) is the foundational architectural mechanism that guarantees loop prevention in active-active topologies.
3️⃣
Configure MirrorCheckpointConnector for Consumer Offset Synchronization
Translate consumer group offsets between disparate Kafka cluster partitions:
- Checkpoint Connector: Emits consumer offset checkpoints every 60 seconds into the internal
checkpoints.internaltopic. - Offset Translation: Because partition offsets differ between clusters, MM2 maps consumer group progress so consumers failing over to GCP can resume from the exact logical record.
Pro Tip: Without offset translation, failing over a consumer group across clouds results in either massive message replay or data loss.
4️⃣
Deploy Heartbeat Probes & End-to-End Replication Lag Telemetry
Track cross-cloud replication latency and connector health:
- Heartbeat Connector: Publishes round-trip timestamps to
heartbeatstopic every 5 seconds. - Prometheus Telemetry: Monitored
kafka_consumergroup_lagandreplication-latency-ms; alerted in Grafana if cross-cloud replication lag exceeded 1,200ms.
Pro Tip: Heartbeat metrics measure actual end-to-end delivery latency, alerting on network congestion before consumer lag accumulates.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Apache Kafka MirrorMaker 2 enables active-active cross-cloud event streaming by using namespace prefixes to prevent replication cycles and checkpoint connectors to translate consumer offsets."
⚡ 60-Second Elevator Pitch Talking Points
- Connect AWS and GCP Kafka clusters via encrypted private interconnect with mTLS.
- Deploy MirrorMaker 2 with MirrorSourceConnector using cluster namespace prefixing.
- Use MirrorCheckpointConnector to synchronize and translate consumer group offsets.
- Monitor cross-cloud replication lag and heartbeats in Prometheus to ensure sub-second sync.
Advertisement