⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 138 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE GCP & Cloud Serverless & Event-Driven Architecture Serverless Architecture
🎯 Target Role / Context: Senior Distributed Systems & Cloud SRE Scenario

Q: In an event-driven microservices architecture on GCP, Cloud Pub/Sub delivers millions of financial payment events per hour with at-least-once semantics. How do you design Cloud Functions Gen 2 and Eventarc to process events idempotently, prevent duplicate transactions during retries, and isolate poison-pill events?

Design pattern for ensuring strictly idempotent, at-least-once message processing in high-throughput GCP Cloud Functions Gen 2 triggered by Eventarc and Cloud Pub/Sub with dead-letter queue recovery.

#GCP #Cloud Functions #Eventarc #Pub/Sub #Idempotency #Memorystore
🎙️ Candidate Opening & Architectural Context
"When our payment webhook processor experienced transient downstream gateway timeouts, Cloud Pub/Sub retried duplicate delivery of thousands of checkout events. Because the consumer was not strictly idempotent, multiple customer cards were double-charged."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: When our payment webhook processor experienced transient downstream gateway timeouts, Cloud Pub/Sub retried duplicate delivery of thousands of checkout events. Because the consumer was not strictly idempotent, multiple customer cards were double-charged.

1️⃣

Configure Cloud Functions Gen 2 with Eventarc Pub/Sub Triggers

Deploy second-generation Cloud Functions powered by Cloud Run infrastructure for enhanced concurrency and request lifecycles:

  • Deployment CLI: Deployed function with gcloud functions deploy payment-processor --gen2 --runtime=nodejs20 --trigger-topic=payment-events --concurrency=80 --min-instances=5 --max-instances=100 --ingress-settings=internal-only.
  • Eventarc Event Filter: Configured CloudEvents standard envelope parsing to inspect unique event ID metadata (ce-id).
Pro Tip: Cloud Functions Gen 2 supports up to 1000 concurrent requests per instance, reducing cold-start latency compared to Gen 1 single-concurrency execution.
2️⃣

Implement Distributed Deduplication with Memorystore Redis

Enforce atomic key locking and lease expiration using Google Cloud Memorystore for Redis:

  • Atomic SETNX Lock: Evaluated incoming event.id or payload transaction_id using Redis atomic command SET payment:idempotency:{id} 'PROCESSING' EX 300 NX.
  • Deduplication Flow: If key already exists with status 'COMPLETED', immediately acknowledge (ACK) Pub/Sub event and return HTTP 200 without re-executing credit card charges.
  • Commit State: Upon successful transaction completion, update status to COMPLETED with 24-hour TTL.
Pro Tip: Using Redis NX (Not Exists) locking ensures that concurrent duplicate deliveries across different container instances are safely intercepted.
Advertisement
3️⃣

Configure Pub/Sub Dead-Letter Queues (DLQ) & Exponential Backoff

Isolate unparseable payloads and persistent downstream failures without stalling the primary ingestion pipeline:

  • DLQ Policy: Configured subscription with gcloud pubsub subscriptions update payment-sub --dead-letter-topic=payment-dlq --max-delivery-attempts=5.
  • Exponential Backoff: Configured minimum backoff of 10 seconds and maximum backoff of 600 seconds to prevent thundering herd during downstream API brownouts.
Pro Tip: Without a DLQ, a single malformed 'poison-pill' JSON payload will cause infinite retries, consuming CPU quota and skewing consumer metrics.
4️⃣

Deploy DLQ Replay Pipeline & Cloud Monitoring Alerting

Establish operational recovery tooling to inspect, fix, and re-inject failed events back into the primary pipeline:

  • Metric Alert: Created alert on pubsub.googleapis.com/subscription/dead_letter_message_count > 0 routing immediately to PagerDuty.
  • Replay Cloud Run Worker: Built an authenticated operational script using Pub/Sub Pull API to inspect failed payloads and republish to primary topic after root-cause resolution.
Pro Tip: Every message routed to a DLQ must include original delivery failure headers and exception traces for rapid triage.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pub/Sub guarantees at-least-once delivery, making duplicate events inevitable. Robust serverless pipelines require atomic Redis idempotency keys, CloudEvents ID validation, and strict Dead-Letter Queues with exponential backoff."
⚡ 60-Second Elevator Pitch Talking Points
  • Deploy Cloud Functions Gen 2 on Cloud Run with Eventarc CloudEvents triggers for high concurrency.
  • Enforce strict idempotency using Memorystore Redis atomic SETNX keys with 24h expiration.
  • Isolate unprocessable payloads using Pub/Sub Dead-Letter Queues with max 5 delivery attempts.
  • Build automated DLQ monitoring and administrative replay tooling for safe disaster recovery.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Engineering bespoke in-house developer tools instead of adopting standardized platform Golden Paths when configuring Architecting Idempotent Event Bespoke internal tools create long-term maintenance debt and fragmented workflows, whereas standardized Golden Paths accelerate delivery.
  2. Overlooking infrastructure cost visibility and FinOps allocation tagging across multi-tenant environments when configuring Architecting Idempotent Event Without granular resource tagging, engineering organizations cannot attribute cloud infrastructure costs to specific teams or business units.
  3. Failing to define explicit Service Level Objectives (SLOs) and Error Budgets to balance feature velocity with reliability when configuring Architecting Idempotent Event Without agreed SLOs, engineering and product teams lack an objective data-driven framework to prioritize technical debt over new features.
  4. Neglecting automated continuous drift detection between declarative configuration and deployed cloud state when configuring Architecting Idempotent Event Unmanaged drift accumulates over time, causing future automated deployments to fail or overwrite emergency manual adjustments.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →