⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 75 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE [L2] AWS Cost & Architecture Production Scenario [L2]
🎯 Target Role / Context: AWS Scenario #75 (L2) · Cost & Architecture

Q: Your SQS consumer occasionally processes the same message twice. How do you handle this?

If an SQS consumer processes the same message twice, treat it as expected behavior to design for, not a bug to eliminate: standard queues guarantee at-least-once delivery, and a message can reappear when the visibility timeout.

#AWS #Cost & Architecture #L2 #Cloud #Infrastructure
🎙️ Candidate Opening & Architectural Context
"If an <strong>SQS consumer processes the same message twice</strong>, treat it as expected behavior to design for, not a bug to eliminate: standard queues guarantee at-least-once delivery, and a message can reappear when the visibility timeout expires before deletion or when the delete call fails. The senior answer is to make the consumer idempotent, tune the visibility timeout and batch handling so duplicates are rare, and use FIFO deduplication only where its limits fit. Interviewers want to hear the causes, a concrete idempotency pattern with a conditional write, and the monitoring that exposes duplicate rates."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: If an <strong>SQS consumer processes the same message twice</strong>, treat it as expected behavior to design for, not a bug to eliminate: standard queues guarantee at-least-once delivery, and a message can reappear when the visibility timeout expires before deletion or when the delete call fails. The senior answer is to make the consumer idempotent, tune the visibility timeout and batch handling so duplicates are rare, and use FIFO deduplication only where its limits fit. Interviewers want to hear the causes, a concrete idempotency pattern with a conditional write, and the monitoring that exposes duplicate rates.

1️⃣

Why duplicates happen

Know the three common sources before choosing a fix.

  • At-least-once delivery: standard queues are distributed; in rare cases a message is delivered more than once even if you deleted it correctly.
  • Visibility timeout too short: when processing takes longer than the timeout, the message becomes visible again and another worker (or the same one) receives it while the first is still working.
  • Delete failure or crash: the worker finishes the side effect but crashes or times out before DeleteMessage, so the message returns.
  • Lambda partial batch failure: if one record in a batch fails and you throw, the whole batch is retried unless you return batch item failures.
  • Interview framing: exactly-once processing is built from at-least-once delivery plus idempotent handling; no queue gives it for free end to end.
2️⃣

Hands-on: SQS consumer processes the same message twice, find out why

Use message attributes and queue metrics to confirm the cause.

aws sqs get-queue-attributes --queue-url "$Q_URL" \
  --attribute-names VisibilityTimeout ReceiveMessageWaitTimeSeconds RedrivePolicy

aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 \
  --attribute-names ApproximateReceiveCount --message-attribute-names All

# Raise the timeout (must exceed worst-case processing time; for Lambda, use at least
# 6x the function timeout as AWS recommends)
aws sqs set-queue-attributes --queue-url "$Q_URL" --attributes VisibilityTimeout=120
  • Receive count: request ApproximateReceiveCount; values above 1 mean redelivery.
  • Compare timeout and duration: if p99 processing time exceeds VisibilityTimeout, that is your cause.
  • Metrics: watch ApproximateAgeOfOldestMessage, NumberOfMessagesDeleted versus NumberOfMessagesReceived; a persistent gap implies messages are received but not deleted.
  • Log the MessageId and your business key at consume time to quantify duplicates.
  • Heartbeat check: if a long job extends visibility, confirm the extension actually runs; a stalled heartbeat thread is a classic hidden cause.
  • Rule out duplicate producers: two different MessageIds with the same payload mean the producer published twice, which a consumer-side timeout fix will not cure.
Advertisement
3️⃣

Fix: make the consumer idempotent

Record a unique key atomically so a second delivery becomes a no-op.

import boto3, time
from botocore.exceptions import ClientError
table = boto3.resource("dynamodb").Table("processed-messages")

def handle(record):
    key = record["order_id"]
    try:
        table.put_item(
            Item={"pk": key, "status": "IN_PROGRESS", "ttl": int(time.time()) + 7*86400},
            ConditionExpression="attribute_not_exists(pk)")
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return  # duplicate, safe to delete the message
        raise
    do_work(record)
    table.update_item(Key={"pk": key}, UpdateExpression="SET #s=:d",
                      ExpressionAttributeNames={"#s": "status"},
                      ExpressionAttributeValues={":d": "DONE"})
  • Choose the key: a business identifier such as order ID or an idempotency key set by the producer, not the SQS MessageId (a producer retry creates a new MessageId).
  • Atomic claim: a DynamoDB conditional PutItem with attribute_not_exists succeeds for exactly one caller.
  • TTL: expire keys after a retention window longer than the maximum redelivery period.
  • Process then mark, or mark then process: mark first with a status of IN_PROGRESS and update to DONE, so crashed attempts can be retried safely. Prefer naturally idempotent operations such as upserts.
  • Powertools option: in Lambda, the AWS Lambda Powertools idempotency utility implements this pattern with DynamoDB, including payload hashing and expiry, so you need not hand-roll it.
4️⃣

Fix: tune queue and Lambda settings

Reduce how often redelivery occurs.

  • Visibility timeout: set above worst-case processing time; for long jobs, extend dynamically with ChangeMessageVisibility heartbeats.
  • Partial batch response: in the Lambda event source mapping set FunctionResponseTypes=[ReportBatchItemFailures] and return only failed itemIdentifiers so successful records are not retried.
  • FIFO queues: provide MessageDeduplicationId (or enable content-based deduplication); duplicates sent within the 5-minute deduplication interval are dropped. This stops producer-side duplicates but not consumer-side redelivery, and has lower throughput limits.
  • DLQ: add a redrive policy with maxReceiveCount so poison messages move aside instead of cycling.
  • Long polling: set ReceiveMessageWaitTimeSeconds to 20 to reduce empty receives; it does not change duplicates but lowers cost and noise when you analyze logs.
5️⃣

Verify and monitor

Prove duplicates are harmless and measurable.

  • Replay test: send the same message twice to staging and assert the side effect happens once.
  • Metrics: emit a custom metric on each duplicate skipped, and alarm on a sudden rise, which signals a timeout regression.
  • Downstream calls: pass your idempotency key to payment or email APIs that support it so external effects are deduplicated too.
  • Outbox note: for database-plus-queue consistency, use the transactional outbox pattern rather than hoping a single publish is exactly once.
  • Capacity of the dedup table: on-demand DynamoDB billing avoids throttling the consumer when traffic spikes.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Treat duplicates as normal, key your idempotency check on a business identifier with a DynamoDB conditional write, and set the visibility timeout above worst-case processing time."
⚡ 60-Second Elevator Pitch Talking Points
  • Standard SQS is at-least-once, and a message also reappears if processing outlasts the visibility timeout or the delete fails.
  • I check ApproximateReceiveCount and compare processing duration with the timeout to find the cause.
  • I make handlers idempotent with a conditional write keyed by business ID, report partial batch failures, add a DLQ, and use FIFO dedup only where needed.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Using the SQS MessageId as the idempotency key A producer retry creates a new MessageId for the same business event, so duplicates slip through.
  2. Relying on FIFO deduplication to protect consumers It only drops duplicate sends within a 5-minute window; consumer redelivery after a timeout still occurs.
  3. Throwing on one bad record in a Lambda batch The entire batch is retried, re-running successful records unless partial batch responses are enabled.
  4. Check-then-act without an atomic write Two workers can both pass the check; use a conditional write or unique constraint.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →