⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All AWS & Cloud Architecture Interview Questions Scenario 34 of 179 in AWS & Cloud Architecture
Senior DevOps / SRE [L2] AWS ECS, EKS, Lambda Production Scenario [L2]
🎯 Target Role / Context: AWS Scenario #27 (L2) · ECS, EKS, Lambda

Q: Your AWS ECS task keeps stopping with exit code 137. What causes this, how do you diagnose task-level vs container-level OOM, and how do you resolve it in production?

Production triage runbook for AWS ECS tasks stopping with exit code 137 (OOMKilled): inspecting ECS StoppedReason, distinguishing task hard limit from container soft limit, CloudWatch Container Insights, and JVM/Node memory tuning.

#exit code 137 ecs #ecs exit code 137 #AWS ECS OOMKilled #exit code 137 #AWS Fargate memory limit #ECS memory reservation #AWS #ECS, EKS, Lambda #SRE #Cloud #Comparisons
🎙️ Candidate Opening & Architectural Context
""In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix.""
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: "In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."

1️⃣

Deciphering Exit Code 137 (128 + 9 = SIGKILL by OOM Killer)

Exit code 137 means the process was killed by signal 9 (128 + 9 = 137). On ECS the most common sender is the Linux OOM killer, but ECS itself also sends SIGKILL when a stop request outlives its timeout, so confirm the cause before resizing memory:

  • Why SIGKILL: When memory exhaustion occurs, the kernel has no time for a graceful SIGTERM shutdown and immediately sends SIGKILL to preserve host stability.
  • ECS Task Status: Run aws ecs describe-tasks --cluster <cluster> --tasks <task-id>. For an OOM kill the container reason reads OutOfMemoryError: Container killed due to memory usage; the task's stopCode and stoppedReason describe every other kind of stop.
2️⃣

Differentiating Task-Level vs Container-Level Memory Limits

AWS ECS task definitions specify memory at two distinct levels:

  • Container Hard Limit (memory): If the specific container exceeds this threshold, the kernel immediately kills that container. If essential=true, the entire task stops.
  • Container Soft Limit (memoryReservation): Memory reserved on the host for the container. The container can exceed this if host memory is available.
  • Task-Level Memory (Fargate): Total memory allocated to the task cgroup. If the aggregate usage of all containers (app + sidecars like Datadog/Envoy) exceeds the task limit, ECS kills the task even if individual containers have not breached their container limits.
Advertisement
3️⃣

Production Remediation & Sizing Strategy

How to fix ECS exit code 137 cleanly:

  • CloudWatch Container Insights: Graph MemoryUtilized vs MemoryReserved to identify baseline consumption and spike patterns.
  • Rightsize Task Definition: Increase Fargate CPU/Memory allocation or increase container memory limits.
  • Runtime Heap Tuning: For JVM applications, ensure container-aware heap sizing flags are set: -XX:MaxRAMPercentage=75.0 so the JVM does not attempt to allocate beyond the container cgroup boundary.
4️⃣

Rule Out Non-OOM Causes of Exit Code 137

Exit code 137 does not always mean memory. Check these before raising limits:

aws ecs describe-tasks --cluster <cluster> --tasks <task-id> \
  --query 'tasks[0].{stopCode:stopCode,stoppedReason:stoppedReason,containers:containers[].{name:name,exitCode:exitCode,reason:reason}}'
  • SIGTERM ignored: ECS sends SIGTERM, then SIGKILL after stopTimeout (30 seconds by default). A shell-form entrypoint or an npm start wrapper swallows SIGTERM, so every deployment ends in 137 with no OOM message. Use exec form or exec, enable initProcessEnabled, and handle SIGTERM in the app.
  • Health check stop: A failed ALB target health check or container health check makes ECS stop the task. The stoppedReason names the health check, not memory.
  • Spot interruption or scale-in: Fargate Spot reclaims and service scale-in stop tasks on purpose. Look at stopCode (for example SpotInterruption or ServiceSchedulerInitiated).
  • EC2 launch type host pressure: Containers without a hard memory limit can push the instance into memory pressure, and the kernel then kills a container that never reached its own limit.
Pro Tip: Triage order: container reason (OOM?), then stopCode and stoppedReason (health check, deployment, Spot), then the gap between SIGTERM and exit (stopTimeout).
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Exit code 137 means SIGKILL, and on ECS the most common sender is the OOM killer, but not the only one. Check the container reason and the task's stopCode and stoppedReason first: OOM points at memory limits, while health-check, deployment, Spot and stopTimeout stops need different fixes."
⚡ 60-Second Elevator Pitch Talking Points
  • Identified exit code 137 as a SIGKILL and used describe-tasks (container reason, stopCode, stoppedReason) to separate OOM kills from SIGTERM-timeout, health-check and Spot stops.
  • Differentiated task-level Fargate allocation limits from individual container memory limits.
  • Monitored CloudWatch Container Insights to pinpoint memory leak spikes in application containers.
  • Configured container-aware JVM flags (-XX:MaxRAMPercentage=75.0) and rightsized task definition memory.
  • Fixed non-OOM cases with exec-form entrypoints, initProcessEnabled and an appropriate stopTimeout.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Raising the memory limit before confirming the exit code 137 was an OOM kill Health-check stops, deployments, Spot reclaims and SIGTERM timeouts also exit 137. Extra memory only raises cost while the restarts continue.
  2. Sizing the JVM or Node.js heap to the full container memory limit Metaspace, thread stacks, direct buffers and sidecars use memory outside the heap. Use -XX:MaxRAMPercentage around 70-75 and leave headroom.
  3. Ignoring sidecar containers (Datadog, Envoy, Fluent Bit) when sizing the task On Fargate the task limit is shared, so the app plus sidecars can exceed it even when every container is under its own limit.
  4. Starting the app through a shell wrapper that swallows SIGTERM ECS waits for stopTimeout and then sends SIGKILL, so every deployment ends in 137 and in-flight requests are dropped. Use exec form or exec, and enable initProcessEnabled.
Advertisement
Want more AWS & Cloud Architecture scenarios?
Explore our complete collection of scenario-based AWS & Cloud Architecture interview runbooks.
Browse All AWS & Cloud Architecture Questions →