Q: Your AWS bill spiked 3x overnight, but no deployments happened. What is your step-by-step response to identify the leak, stop the bleeding, and prevent recurrence?
Urgent incident response protocol to detect, stop, and remediate a 3x AWS cloud billing spike when zero code deployments occurred.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Immediate Hourly Granularity Cost Breakdown in AWS Cost Explorer
Open AWS Cost Explorer or query the AWS Cost and Usage Report (CUR) with Athena. Group costs by 'Usage Type' and 'API Operation' over the past 24 hours to isolate the exact AWS service driving the surge (e.g. 'USE2-NatGateway-Bytes', 'DataTransfer-Out-Bytes', 'S3-Requests-Tier1', or 'EC2-BoxUsage').
aws ce get-cost-and-usage \
--time-period Start=$(date -u -v-2d +%Y-%m-%d),End=$(date -u +%Y-%m-%d) \
--granularity HOURLY \
--metrics "UnblendedCost" \
--group-by Type=DIMENSION,Key=SERVICECheck Top 4 Silent Cloud Cost Drivers
Investigate common non-deployment spend drivers: 1. **NAT Gateway Data Processing**: Internal pods pulling massive container images or dataset backups from external registries through NAT instead of VPC Endpoints ($0.045/GB). 2. **Cross-AZ Data Transfer**: Microservices communicating across Availability Zones without AZ-affinity ($0.01/GB each way). 3. **S3 List/Put Request Storms**: Recursive scripts or broken backup jobs generating billions of S3 API calls. 4. **Security Breach**: Compromised access keys launching EC2 GPU instances (p4d, g5) across unfamiliar regions (e.g. ap-southeast-2, sa-east-1).
Stop the Financial Bleeding (Emergency Triage)
If runaway SQS/Lambda loop: throttle concurrency or disable event source mappings. If NAT Gateway traffic: deploy AWS Gateway Endpoints for S3/DynamoDB (which are 100% free) or VPC Interface Endpoints. If unauthorized compute: terminate rogue instances, revoke compromised IAM session tokens, and attach explicit deny SCPs.
# Provision free S3 Gateway Endpoint to bypass NAT Gateway immediately:
aws ec2 create-vpc-endpoint \
--vpc-id vpc-0123456789abcdef0 \
--service-name com.amazonaws.us-east-1.s3 \
--route-table-ids rtb-0123456789abcdef0Implement Automated Cost Anomaly Detection & Guardrails
Configure AWS Cost Anomaly Detection with SNS/Slack alerts triggering within 2 hours of deviation. Set AWS Budgets with action policies to automatically shut down non-production accounts when monthly spending limits are exceeded.
aws ce create-anomaly-subscription \
--subscription '{"SubscriptionName":"Sev1-Cost-Spike","Threshold":500,"Frequency":"IMMEDIATE","Subscribers":[{"Type":"EMAIL","Address":"sre-alerts@company.com"}]}'- Query AWS Cost Explorer with hourly granularity grouped by Service and UsageType to find the guilty resource within 5 minutes.
- Verify NAT Gateway bytes and cross-AZ traffic; deploy free S3 Gateway Endpoints to instantly cut data processing fees.
- Audit CloudTrail globally for RunInstances and CreateAccessKey to eliminate credential theft and crypto-mining.
- Establish AWS Cost Anomaly Detection and Budgets with automated SNS/Slack escalations.