EC2 CPU Suddenly Reaches 100% — Troubleshooting Runbook
Triage runbook for handling sudden 100% CPU utilization on production EC2 instances: telemetry triage, identifying culprit processes, handling legit vs maliciou...
Master 131+ battle-tested scenario-based AWS & Cloud Architecture interview questions for Senior DevOps, Cloud, and SRE engineers. Includes incident runbooks, STAR talking points, and CLI commands.
Triage runbook for handling sudden 100% CPU utilization on production EC2 instances: telemetry triage, identifying culprit processes, handling legit vs maliciou...
Systematic OSI and AWS-layer troubleshooting methodology when an EC2 instance shows 'Running' but SSH connection fails, covering network timeouts, con...
Production runbook for isolating Application Load Balancer (ALB) 5xx errors: distinguishing ELB-generated vs Target-generated codes, debugging 502/503/504, and ...
Outside-in OSI network layer troubleshooting model for resolving applications reachable internally on private VPC IPs but inaccessible over the public internet....
Production playbook for migrating mission-critical enterprise workloads from on-premises to AWS with near-zero downtime: discovery, Direct Connect hybrid networ...
Practical portfolio walkthrough of enterprise Microsoft Azure services utilized in modern production DevOps: Key Vault, Azure Container Registry (ACR), Managed ...
End-to-end operational guide for Azure Kubernetes Service (AKS): control plane vs node pools, Azure CNI vs Kubenet, GitOps deployment with Azure DevOps / GitHub...
1. Security Group — does inbound rule allow port 22 from your IP? Check in the EC2 console under Security Groups....
1. CloudWatch metrics — check CPU utilization over time. Is it sustained or spiky?...
The ALB health check is likely misconfigured:...
This sounds like a lifecycle hook or health check issue:...
Use a NAT Gateway (managed by AWS) or NAT Instance (self-managed EC2):...
1. Reserved Instances (RI) — commit to 1 or 3 years for 40-60% discount. Best for steady-state production workloads....
If versioning is enabled: The delete created a "delete marker." You can recover by:...
1. Block Public Access settings — go to S3 → Block Public Access → Enable all four settings. This overrides any bucket/object ACLs and po......
1. IAM permissions — the IAM role/user doesn't have s3:PutObject permission on the bucket....
Use S3 Glacier Deep Archive — cheapest storage class. ~$1/TB/month vs S3 Standard ~$23/TB/month....
Use S3 Pre-Signed URLs:...
1. S3 automatically partitions by prefix — requests are distributed across S3 partitions. More unique prefixes = better parallelism. Add ......
Architectural comparison between AWS Security Groups (stateful, instance-level firewall) and Network ACLs (stateless, subnet-level packet filter)....
1. Same VPC? — Confirm both are in the same VPC. Different VPCs require VPC Peering....
Option 1: VPC Peering...
Overlapping CIDRs are a real problem — traffic routing becomes ambiguous....
VPC Flow Logs captures IP traffic information for network interfaces in your VPC. Logged to CloudWatch Logs or S3....
Option 1: AWS Site-to-Site VPN...
- CLB (Classic Load Balancer) — legacy. Avoid for new projects. Layer 4 and Layer 7 but limited features....
1. Health check path — does the path exist? curl http://<instance-ip>:<health-check-port><health-check-path> from the ALB's subnet....
Lambda functions use an execution role (IAM role). This role needs dynamodb:PutItem (or broader dynamodb:) permission on the target table....
Use IAM Cross-Account Role Assumption:...
This is a security incident. Move fast — bots scan GitHub and abuse leaked keys within minutes....
- Identity policy (attached to IAM user/role/group) — defines what that identity CAN do across AWS....
A permission boundary is a policy attached to an IAM entity that sets the maximum permissions the entity can ever have — even if an ident......
- EC2 launch type — you manage the EC2 instances (the container hosts). You're responsible for patching, scaling the cluster, choosing in......
Exit code 137 = container killed by OOM (Out of Memory) — 128 + 9 (SIGKILL)....
1. Timeout too short — default is 3 seconds. If the function takes 5 seconds, increase the timeout (up to 15 minutes)....
Lambda has a concurrency limit — default 1000 concurrent executions per region per account....
ECS (Elastic Container Service)...
1. Identify the culprit — use RDS Performance Insights to find the top SQL queries consuming CPU....
1. Create a cross-region read replica — in RDS, create a read replica in the target region. It will replicate all data and stay in sync v......
Multi-AZ failover is automatic, but several things can prevent or delay it:...
Each Lambda invocation or serverless function instance opens its own DB connection. At scale, this exhausts the DB connection limit....
Enable Point-In-Time Recovery (PITR):...
1. In CloudWatch, create an Alarm:...
- CloudWatch Logs — stores raw log data (text). Your app, Lambda, ECS, VPC Flow Logs all write here. You can search/query with CloudWatch......
Layer by layer:...
1. Cost Explorer — filter by service CloudWatch to see which API calls cost the most (GetMetricStatistics, PutLogEvents, etc.)....
- CloudTrail — records API calls made to AWS. Who did what, when, from where. "Audit trail." Example: who deleted that S3 bucket? Who cha......
1. CodeCommit or GitHub — source code repository. Push triggers the pipeline....
CodeBuild uses a service role. That role needs ECR permissions:...
Production scenario covering Design a highly available, scalable web application architecture on AW....
AWS and you share responsibility for security:...
1. AWS Cost Explorer — open it, filter by service. Which service increased?...
- CloudTrail — records API actions. "What happened?" Who called ec2:TerminateInstances?...
Static website hosting requires public read. Either make bucket public (then enable static hosting) or use CloudFront with Origin Access ......
Put the Lambda function in the same VPC and private subnet. Add the Lambda's SG to the RDS inbound rules on DB port....
Use VPC Endpoints — Interface Endpoints or Gateway Endpoints for S3 and DynamoDB. Traffic stays within AWS network. Cheaper than NAT....
Route 53 with latency-based or geolocation routing. Application in both regions. Aurora Global Database (primary in one region, read repl......
Objects must be > 128KB for lifecycle transition to Glacier to apply. Also check the prefix filter matches your objects....
Check CloudFormation events in console for the failure reason. Common: IAM permissions, resource limit, invalid property value. Fix the t......
Use CodeDeploy with ECS blue/green. Two target groups (blue=current, green=new). CodeDeploy shifts traffic gradually from blue to green. ......
Scale out consumer EC2/ECS instances. Use ASG scaled on ApproximateNumberOfMessagesVisible CloudWatch metric. Or move to Lambda consumer ......
Check subscription is confirmed (for email, need to click confirmation link). Check subscription filter policy. Check dead-letter queue f......
SNS = pub/sub, one message to many subscribers (fan-out). SQS = queue, message stored until a consumer reads and deletes it (point-to-poi......
Check for hot partitions (one partition key getting all traffic). Use DynamoDB Accelerator (DAX) for microsecond read caching. Check cons......
Each ECS task or Lambda function has its own IAM role with only the permissions it needs. Use task IAM roles for ECS, execution roles for......
Invalidate the CloudFront cache: aws cloudfront create-invalidation --distribution-id <id> --paths "/". Or use cache-control headers and ......
S3 upload → S3 Event Notification → SQS queue → Lambda consumer reads from SQS → resizes image → stores to output S3 bucket → SNS notific......
Route 53 health checks come from specific IP ranges. Ensure Security Group/firewall allows those IPs. Check the health check protocol (HT......
ECS Scheduled Tasks — set a cron expression on the ECS task. ECS runs the Fargate task on schedule and stops it when done....
WAF inspects HTTP requests before they reach the ALB. Rules: AWS Managed Rule Groups (OWASP top 10, bot control), IP rate limiting (preve......
Move model loading code to the Lambda initialization phase (outside the handler function). The runtime container is reused between invoca......
ElastiCache Redis — shared in-memory store that all instances can access. Never store sessions in local EC2 memory (breaks when an instan......
Parameter Store = config and non-sensitive parameters. Free tier available. Secrets Manager = specifically for secrets. Auto-rotation bui......
Use a no-lock migration approach: add new column (no lock), backfill data in batches, add new index concurrently, switch app to use new c......
Spot capacity issue or that AZ/region is out of that instance type. Mitigate: use multiple instance types in the ASG (mixed instances pol......
RDS doesn't allow enabling encryption on a running instance. Steps: take a snapshot → copy the snapshot with encryption enabled → restore......
Elastic Beanstalk console → Environment → Configuration → Software → Environment properties. Or via .ebextensions files in your code. Or ......
Nitro Enclaves create isolated compute environments within EC2 instances for processing highly sensitive data (cryptographic keys, PII). ......
Request a limit increase via AWS Service Quotas. Or redesign to use fewer VPCs with more subnets. Or use a shared VPC (Resource Access Ma......
Based on scaling policies: target tracking (maintain metric at target, e.g., 70% CPU), step scaling (scale by N instances when metric cro......
AWS Athena with Lake Formation for cross-account data access. Or use Amazon Redshift data sharing for analytics. Athena queries S3 data u......
Implement idempotent consumers — use a message ID to track processed messages (store in DynamoDB). If already processed, skip. Also: use ......
Vertical = bigger instance. Horizontal = more instances. AWS encourages horizontal (Auto Scaling Groups, ECS/EKS). Vertical is limited (m......
Remove all implicit trust. Use: IAM everywhere (not network location), security groups per-service (not per-subnet), mutual TLS between s......
Use continue-update-rollback API. It lets you specify resources to skip during rollback so the rollback can complete. After rollback comp......
Enable S3 Versioning + MFA Delete. Enable S3 Object Lock (WORM). Use a bucket policy with Deny for s3:DeleteBucket. Enable AWS Config rul......
Options by RPO/RTO: Backup & Restore (hours RPO/RTO, cheapest) → Pilot Light (critical infra always on, warm data, minutes to hours) → Wa......
Intelligent threat detection service. Analyzes CloudTrail, VPC Flow Logs, DNS logs. Detects: compromised instances communicating with mal......
1) Isolate: change security group to block all outbound. 2) Take a snapshot (forensics). 3) Check VPC Flow Logs for the outbound connecti......
KMS generates and stores encryption keys. You never handle raw key material. AWS-managed keys: automatic rotation, free, no management ne......
EventBridge = event bus that routes events from AWS services (EC2, S3, CodePipeline) and custom apps to Lambda, SQS, SNS, Step Functions.......
Data consistency (cross-region replication has latency), data sovereignty (some data can't leave specific regions), cost (cross-region da......
API Gateway or Kinesis Data Streams (ingestion) → Kinesis Firehose (buffer/batch) → S3 (raw data lake) → Glue crawler (schema discovery) ......
Kinesis: ordered stream, multiple consumers can read same data, data retained 24h-365 days, good for analytics and fan-out. SQS: queue, m......
Check ECS task security group allows ALB security group on the container port. Check the health check path returns 200 on that port. Chec......
AWS Config continuous compliance — detects when actual resource state drifts from desired. CloudFormation Drift Detection — compares stac......
Implement exponential backoff with jitter in API retry logic. Use AWS SDK built-in retry (most SDKs have this). Reduce polling frequency.......
Automated security vulnerability assessment for EC2 and ECR. Scans OS packages and app libraries for CVEs. Integrates with Security Hub. ......
Each AZ is a physically separate data center (separate power, cooling, networking). AZs in same region connected via low-latency links. D......
Recommends best practices across: Cost Optimization (idle resources, unused RIs), Security (open SGs, IAM best practices), Fault Toleranc......
Use AWS Secrets Manager. It stores the password and has a rotation Lambda function that: generates new password, updates it in RDS, updat......
SCP is a guardrail for entire AWS accounts in an Organization. It restricts what IAM policies in those accounts can allow. If SCP doesn't......
In each account: create CloudTrail and send to S3 in the security account. Update the security account S3 bucket policy to allow PutObjec......
Cache the secret in application memory (most secrets don't change frequently). AWS Secrets Manager SDK supports caching. Or use Parameter......
PrivateLink: exposes a specific service (not a whole network) from one VPC to another. Traffic goes through AWS backbone. Supports cross-......
Parallelize independent resources (CFN does this automatically). Use nested stacks to update only changed stacks. Use ChangeSets to previ......
Use API Gateway WebSocket API (scales automatically, no infra to manage). Each connection triggers Lambda functions for connect/disconnec......
You would use ECS Exec (which is powered by AWS Systems Manager Session Manager under the hood)....
Historically, in S3, the AWS account that uploads the object retains explicit ownership and full control of that object, even if the buck......
This happens because you selected an Edge-Optimized API Gateway endpoint....
A CNAME (Canonical Name) essentially maps one domain to another domain. However, the strict global DNS protocol absolutely forbids a CNAM......
Aurora Serverless v2 dynamically scales compute (CPU and RAM) via ACUs (Aurora Capacity Units) in milliseconds. However, it scales the un......
You must implement Origin Access Control (OAC) (the modern replacement for Origin Access Identity, OAI)....
No. They are referencing outdated architecture. ...
This requires a two-part solution utilizing application load balancing and ASG native hooks:...
The SQS Visibility Timeout is misconfigured....
AWS natively provides a 2-Minute Spot Instance Interruption Notice before the instance is forcefully terminated....
Creating a VPC Endpoint simply keeps the traffic on the AWS private backbone; it does not secure it inherently. An attacker could still r......
Local Secondary Indexes (LSIs) are deeply embedded into the physical partition layout of the original DynamoDB table (forcing the same Pa......
This is the immense power of Service Control Policies (SCPs) administered from the AWS Organizations Management (Master) account. ...
The application has exhausted its EFS Burst Credits. ...
You must undeniably use Standard Workflows....
By default, CloudTrail only records Management Events (Control Plane actions). These include creating infrastructure (CreateBucket, RunIn......
Both roles serve entirely different isolation boundaries:...
This is classic SNAT (Source Network Address Translation) Port Exhaustion on the NAT Gateway....
You would append a Condition block to their overarching IAM Policy (or a global SCP) that heavily restricts authentication based on their......
Legacy Redshift nodes (like DC2/DS2) tightly coupled Compute and Storage physically onto the single instance. If you strictly needed 50TB......
Engineering blueprint for designing an enterprise multi-cloud substrate across AWS and GCP covering private BGP interconnects, federated OIDC workload identitie...
Deep architectural trade-off comparison between Multi-AZ and Multi-Region deployments in AWS: synchronous vs asynchronous replication, network latency, data con...
Enterprise AWS IAM security architecture: replacing static credentials with OIDC role assumption, scoping resource ARNs, enforcing IAM Permission Boundaries to ...
Strategic framework for designing disaster recovery across the 4 AWS DR tiers (Backup & Restore, Pilot Light, Warm Standby, Multi-Site Active/Active) while ...