Q: A serverless microservice using DynamoDB is throttled with 'ProvisionedThroughputExceededException', even though total consumed capacity is only 25% of the provisioned table limits. Investigation reveals 95% of traffic queries a single partition key ('Status = PENDING'). Furthermore, an asynchronous Global Secondary Index (GSI) has reached write capacity limits and is backpressuring the main table. How do you re-architect the partition key scheme and mitigate GSI write throttling?
Solve DynamoDB ProvisionedThroughputExceededException throttling caused by asymmetric partition key distribution and asynchronous Global Secondary Index write backpressure.
Want to master this scenario in a live sandbox? KodeKloud's PostgreSQL Database Administration & High Availability Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Amazon DynamoDB distributes data across storage partitions based on the hash of the Partition Key (PK). Each individual physical partition is capped at 1,000 Write Capacity Units (WCUs) and 3,000 Read Capacity Units (RCUs). When workload traffic is skewed toward a single partition key (a 'hot partition'), throttling occurs even if the overall table has unused provisioned capacity. Crucially, when an asynchronous Global Secondary Index (GSI) is throttled on writes, DynamoDB throttles writes on the base table to prevent the index from falling indefinitely behind.
Situation: Throttling Despite Low Global Table Utilization
Task: Eliminate Hot Partitions & Balance GSI Provisioning
Action: Partition Key Salting & Sparse Index Optimization
Result: 100% Throttling Elimination & 65% Cost Reduction
- DynamoDB partitions have a hard limit of 1,000 WCU and 3,000 RCU each.
- A single hot partition key or a throttled GSI will throttle writes to the entire base table.
- Apply write salting (partition sharding) and sparse GSIs to distribute throughput and eliminate backpressure.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Using a low-cardinality partition key (e.g., status, boolean, date) in high-throughput DynamoDB tables Low-cardinality partition keys route disproportionate request volume to single storage nodes, triggering ProvisionedThroughputExceededException throttling.
- Neglecting GSI write throttling backpressure on primary DynamoDB table write operations If a GSI lacks sufficient WCU capacity, backpressure from the secondary index throttles writes on the primary table regardless of base table provisioning.
- Omitting exponential backoff and jitter algorithms in client SDK retry configurations during DynamoDB throttling Retrying throttled calls in lockstep causes synchronized retry storms that compound partition exhaustion instead of letting the partition drain.
-
Failing to monitor CloudWatch
ThrottledRequestsand Contributor Insights during DynamoDB latency spikes when configuring Amazon DynamoDB Partition Senior SREs use Contributor Insights to pinpoint the exact partition keys and access patterns causing throttling rather than relying on average latency metrics.