Q: Your AWS ECS task keeps stopping with exit code 137. What causes this, how do you diagnose task-level vs container-level OOM, and how do you resolve it in production?
Production triage runbook for AWS ECS tasks stopping with exit code 137 (OOMKilled): inspecting ECS StoppedReason, distinguishing task hard limit from container soft limit, CloudWatch Container Insights, and JVM/Node memory tuning.
Want to master this scenario in a live sandbox? Stephane Maarek's AWS Certified DevOps Engineer Professional Masterclass on Udemy covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: "In a previous role, our monitoring paged me for a similar incident across our AWS VPC infrastructure. When addressing this question, I walk the interviewer through our production incident runbook: isolating the blast radius, checking diagnostic logs and metrics, and applying a safe fix."
Deciphering Exit Code 137 (128 + 9 = SIGKILL by OOM Killer)
Exit code 137 means the process was killed by signal 9 (128 + 9 = 137). On ECS the most common sender is the Linux OOM killer, but ECS itself also sends SIGKILL when a stop request outlives its timeout, so confirm the cause before resizing memory:
- Why SIGKILL: When memory exhaustion occurs, the kernel has no time for a graceful SIGTERM shutdown and immediately sends SIGKILL to preserve host stability.
- ECS Task Status: Run
aws ecs describe-tasks --cluster <cluster> --tasks <task-id>. For an OOM kill the containerreasonreadsOutOfMemoryError: Container killed due to memory usage; the task'sstopCodeandstoppedReasondescribe every other kind of stop.
Differentiating Task-Level vs Container-Level Memory Limits
AWS ECS task definitions specify memory at two distinct levels:
- Container Hard Limit (memory): If the specific container exceeds this threshold, the kernel immediately kills that container. If essential=true, the entire task stops.
- Container Soft Limit (memoryReservation): Memory reserved on the host for the container. The container can exceed this if host memory is available.
- Task-Level Memory (Fargate): Total memory allocated to the task cgroup. If the aggregate usage of all containers (app + sidecars like Datadog/Envoy) exceeds the task limit, ECS kills the task even if individual containers have not breached their container limits.
Production Remediation & Sizing Strategy
How to fix ECS exit code 137 cleanly:
- CloudWatch Container Insights: Graph
MemoryUtilizedvsMemoryReservedto identify baseline consumption and spike patterns. - Rightsize Task Definition: Increase Fargate CPU/Memory allocation or increase container
memorylimits. - Runtime Heap Tuning: For JVM applications, ensure container-aware heap sizing flags are set:
-XX:MaxRAMPercentage=75.0so the JVM does not attempt to allocate beyond the container cgroup boundary.
Rule Out Non-OOM Causes of Exit Code 137
Exit code 137 does not always mean memory. Check these before raising limits:
aws ecs describe-tasks --cluster <cluster> --tasks <task-id> \
--query 'tasks[0].{stopCode:stopCode,stoppedReason:stoppedReason,containers:containers[].{name:name,exitCode:exitCode,reason:reason}}'
- SIGTERM ignored: ECS sends SIGTERM, then SIGKILL after
stopTimeout(30 seconds by default). A shell-form entrypoint or annpm startwrapper swallows SIGTERM, so every deployment ends in 137 with no OOM message. Use exec form orexec, enableinitProcessEnabled, and handle SIGTERM in the app. - Health check stop: A failed ALB target health check or container health check makes ECS stop the task. The
stoppedReasonnames the health check, not memory. - Spot interruption or scale-in: Fargate Spot reclaims and service scale-in stop tasks on purpose. Look at
stopCode(for exampleSpotInterruptionorServiceSchedulerInitiated). - EC2 launch type host pressure: Containers without a hard
memorylimit can push the instance into memory pressure, and the kernel then kills a container that never reached its own limit.
- Identified exit code 137 as a SIGKILL and used describe-tasks (container reason, stopCode, stoppedReason) to separate OOM kills from SIGTERM-timeout, health-check and Spot stops.
- Differentiated task-level Fargate allocation limits from individual container memory limits.
- Monitored CloudWatch Container Insights to pinpoint memory leak spikes in application containers.
- Configured container-aware JVM flags (-XX:MaxRAMPercentage=75.0) and rightsized task definition memory.
- Fixed non-OOM cases with exec-form entrypoints, initProcessEnabled and an appropriate stopTimeout.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Raising the memory limit before confirming the exit code 137 was an OOM kill Health-check stops, deployments, Spot reclaims and SIGTERM timeouts also exit 137. Extra memory only raises cost while the restarts continue.
- Sizing the JVM or Node.js heap to the full container memory limit Metaspace, thread stacks, direct buffers and sidecars use memory outside the heap. Use -XX:MaxRAMPercentage around 70-75 and leave headroom.
- Ignoring sidecar containers (Datadog, Envoy, Fluent Bit) when sizing the task On Fargate the task limit is shared, so the app plus sidecars can exceed it even when every container is under its own limit.
- Starting the app through a shell wrapper that swallows SIGTERM ECS waits for stopTimeout and then sends SIGKILL, so every deployment ends in 137 and in-flight requests are dropped. Use exec form or exec, and enable initProcessEnabled.