Q: Your enterprise runs transactional order services in AWS and analytics engines in GCP. You need real-time, bi-directional event streaming between AWS Kafka (MSK) and GCP Kafka. How do you design MirrorMaker 2 to replicate topics across clouds without causing infinite replication loops, and how do you handle consumer group offset migration?
Engineering a resilient active-active cross-cloud event streaming platform with Apache Kafka MirrorMaker 2 (MM2) running between AWS MSK and GCP Strimzi Kafka with cycle detection and offset translation.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Uncoordinated bi-directional replication between clouds can easily enter an infinite feedback loop where topic messages mirror back and forth indefinitely. We implemented Apache Kafka MirrorMaker 2 with active namespace prefixing and automated offset translation.
Establish Kafka Topologies & Private Cross-Cloud Network
Connect AWS MSK and GCP Kafka clusters via encrypted private peering:
- Clusters: Configured
aws-cluster(AWS MSK in us-east-1) andgcp-cluster(Strimzi Kafka on GKE in us-central1). - Network Transport: Routed traffic across private interconnect with mutual TLS (mTLS) authentication and TLS 1.3 encryption.
Configure MirrorMaker 2 Connectors with Automated Topic Renaming
Deploy MirrorSourceConnector, MirrorCheckpointConnector, and MirrorHeartbeatConnector:
- Source Connector: Replicates topics from
aws-clusterinto GCP with prefixed topic names:aws-cluster.orders. - Cycle Prevention: MM2 automatically detects source prefixes and refuses to mirror a topic back to its originating cluster, preventing infinite replication loops.
Configure MirrorCheckpointConnector for Consumer Offset Synchronization
Translate consumer group offsets between disparate Kafka cluster partitions:
- Checkpoint Connector: Emits consumer offset checkpoints every 60 seconds into the internal
checkpoints.internaltopic. - Offset Translation: Because partition offsets differ between clusters, MM2 maps consumer group progress so consumers failing over to GCP can resume from the exact logical record.
Deploy Heartbeat Probes & End-to-End Replication Lag Telemetry
Track cross-cloud replication latency and connector health:
- Heartbeat Connector: Publishes round-trip timestamps to
heartbeatstopic every 5 seconds. - Prometheus Telemetry: Monitored
kafka_consumergroup_lagandreplication-latency-ms; alerted in Grafana if cross-cloud replication lag exceeded 1,200ms.
- Connect AWS and GCP Kafka clusters via encrypted private interconnect with mTLS.
- Deploy MirrorMaker 2 with MirrorSourceConnector using cluster namespace prefixing.
- Use MirrorCheckpointConnector to synchronize and translate consumer group offsets.
- Monitor cross-cloud replication lag and heartbeats in Prometheus to ensure sub-second sync.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Applying exclusive-lock DDL table migrations on live production databases during peak traffic windows when configuring Cross Event Streaming DDL migrations that acquire an exclusive table lock block all concurrent queries, queuing connections until the database connection pool exhausts.
- Allowing high-frequency queries to trigger full table scans without appropriate composite indexing when configuring Cross Event Streaming Sequential table scans saturate database CPU and disk I/O, evicting frequently queried data from the buffer cache and causing cascading timeouts.
- Disabling referential integrity constraints during bulk ETL imports without post-import consistency validation when configuring Cross Event Streaming Skipping relational consistency checks risks introducing orphaned records and referential integrity violations into production tables.
- Relying on database snapshot backups that have never been verified through automated restoration drills when configuring Cross Event Streaming Backups that have never been tested with automated restore verification frequently fail during real disaster recovery scenarios.