Q: Your SQS consumer occasionally processes the same message twice. How do you handle this?
If an SQS consumer processes the same message twice, treat it as expected behavior to design for, not a bug to eliminate: standard queues guarantee at-least-once delivery, and a message can reappear when the visibility timeout.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: If an <strong>SQS consumer processes the same message twice</strong>, treat it as expected behavior to design for, not a bug to eliminate: standard queues guarantee at-least-once delivery, and a message can reappear when the visibility timeout expires before deletion or when the delete call fails. The senior answer is to make the consumer idempotent, tune the visibility timeout and batch handling so duplicates are rare, and use FIFO deduplication only where its limits fit. Interviewers want to hear the causes, a concrete idempotency pattern with a conditional write, and the monitoring that exposes duplicate rates.
Why duplicates happen
Know the three common sources before choosing a fix.
- At-least-once delivery: standard queues are distributed; in rare cases a message is delivered more than once even if you deleted it correctly.
- Visibility timeout too short: when processing takes longer than the timeout, the message becomes visible again and another worker (or the same one) receives it while the first is still working.
- Delete failure or crash: the worker finishes the side effect but crashes or times out before
DeleteMessage, so the message returns. - Lambda partial batch failure: if one record in a batch fails and you throw, the whole batch is retried unless you return batch item failures.
- Interview framing: exactly-once processing is built from at-least-once delivery plus idempotent handling; no queue gives it for free end to end.
Hands-on: SQS consumer processes the same message twice, find out why
Use message attributes and queue metrics to confirm the cause.
aws sqs get-queue-attributes --queue-url "$Q_URL" \
--attribute-names VisibilityTimeout ReceiveMessageWaitTimeSeconds RedrivePolicy
aws sqs receive-message --queue-url "$Q_URL" --max-number-of-messages 1 \
--attribute-names ApproximateReceiveCount --message-attribute-names All
# Raise the timeout (must exceed worst-case processing time; for Lambda, use at least
# 6x the function timeout as AWS recommends)
aws sqs set-queue-attributes --queue-url "$Q_URL" --attributes VisibilityTimeout=120
- Receive count: request
ApproximateReceiveCount; values above 1 mean redelivery. - Compare timeout and duration: if p99 processing time exceeds
VisibilityTimeout, that is your cause. - Metrics: watch
ApproximateAgeOfOldestMessage,NumberOfMessagesDeletedversusNumberOfMessagesReceived; a persistent gap implies messages are received but not deleted. - Log the MessageId and your business key at consume time to quantify duplicates.
- Heartbeat check: if a long job extends visibility, confirm the extension actually runs; a stalled heartbeat thread is a classic hidden cause.
- Rule out duplicate producers: two different MessageIds with the same payload mean the producer published twice, which a consumer-side timeout fix will not cure.
Fix: make the consumer idempotent
Record a unique key atomically so a second delivery becomes a no-op.
import boto3, time
from botocore.exceptions import ClientError
table = boto3.resource("dynamodb").Table("processed-messages")
def handle(record):
key = record["order_id"]
try:
table.put_item(
Item={"pk": key, "status": "IN_PROGRESS", "ttl": int(time.time()) + 7*86400},
ConditionExpression="attribute_not_exists(pk)")
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
return # duplicate, safe to delete the message
raise
do_work(record)
table.update_item(Key={"pk": key}, UpdateExpression="SET #s=:d",
ExpressionAttributeNames={"#s": "status"},
ExpressionAttributeValues={":d": "DONE"})
- Choose the key: a business identifier such as order ID or an idempotency key set by the producer, not the SQS MessageId (a producer retry creates a new MessageId).
- Atomic claim: a DynamoDB conditional
PutItemwithattribute_not_existssucceeds for exactly one caller. - TTL: expire keys after a retention window longer than the maximum redelivery period.
- Process then mark, or mark then process: mark first with a status of IN_PROGRESS and update to DONE, so crashed attempts can be retried safely. Prefer naturally idempotent operations such as upserts.
- Powertools option: in Lambda, the AWS Lambda Powertools idempotency utility implements this pattern with DynamoDB, including payload hashing and expiry, so you need not hand-roll it.
Fix: tune queue and Lambda settings
Reduce how often redelivery occurs.
- Visibility timeout: set above worst-case processing time; for long jobs, extend dynamically with
ChangeMessageVisibilityheartbeats. - Partial batch response: in the Lambda event source mapping set
FunctionResponseTypes=[ReportBatchItemFailures]and return only faileditemIdentifiers so successful records are not retried. - FIFO queues: provide
MessageDeduplicationId(or enable content-based deduplication); duplicates sent within the 5-minute deduplication interval are dropped. This stops producer-side duplicates but not consumer-side redelivery, and has lower throughput limits. - DLQ: add a redrive policy with
maxReceiveCountso poison messages move aside instead of cycling. - Long polling: set
ReceiveMessageWaitTimeSecondsto 20 to reduce empty receives; it does not change duplicates but lowers cost and noise when you analyze logs.
Verify and monitor
Prove duplicates are harmless and measurable.
- Replay test: send the same message twice to staging and assert the side effect happens once.
- Metrics: emit a custom metric on each duplicate skipped, and alarm on a sudden rise, which signals a timeout regression.
- Downstream calls: pass your idempotency key to payment or email APIs that support it so external effects are deduplicated too.
- Outbox note: for database-plus-queue consistency, use the transactional outbox pattern rather than hoping a single publish is exactly once.
- Capacity of the dedup table: on-demand DynamoDB billing avoids throttling the consumer when traffic spikes.
- Standard SQS is at-least-once, and a message also reappears if processing outlasts the visibility timeout or the delete fails.
- I check ApproximateReceiveCount and compare processing duration with the timeout to find the cause.
- I make handlers idempotent with a conditional write keyed by business ID, report partial batch failures, add a DLQ, and use FIFO dedup only where needed.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Using the SQS MessageId as the idempotency key A producer retry creates a new MessageId for the same business event, so duplicates slip through.
- Relying on FIFO deduplication to protect consumers It only drops duplicate sends within a 5-minute window; consumer redelivery after a timeout still occurs.
- Throwing on one bad record in a Lambda batch The entire batch is retried, re-running successful records unless partial batch responses are enabled.
- Check-then-act without an atomic write Two workers can both pass the check; use a conditional write or unique constraint.