Q: A high-volume microservice repeatedly crashes with 'OOMKilled' and Exit Code 137, but the underlying worker node has 48GB of unallocated physical RAM. Explain why this happens, how Linux cgroup memory accounting works in Kubernetes, how Kubelet assigns oom_score_adj based on QoS classes, and how you resolve OOMKilled in JVM and Node.js applications.

Confirm that the termination was indeed an Out-Of-Memory kill and extract the historical termination state:

• Describe the Pod's Last State: Run kubectl describe pod <pod-name> and check Last State: Terminated.
  Look for: Reason: OOMKilled and Exit Code: 137.
• Container OOM vs Node OOM:
  • Container OOM: Node has free RAM. Kubelet condition MemoryPressure is False. The container breached its specific pod memory limit.
  • Node OOM: The worker node ran completely out of physical RAM and swap. Kubelet condition MemoryPressure is True. The host kernel OOM killer sacrificed pods based on oom_score_adj.

Understand how Kubernetes measures memory consumption to prevent false diagnoses:

• The Metric That Triggers OOM: Kubernetes does NOT base OOM kills on Resident Set Size (RSS) alone! It evaluates container_memory_working_set_bytes.
  working_set = memory.current - inactive_file.
• The Page Cache Trap: Linux caches file reads in the page cache (active_file + inactive_file). While inactive page cache can be reclaimed under memory pressure, dirty and active file pages cannot be reclaimed instantly.
• Prometheus PromQL Query:

  container_memory_working_set_bytes{pod="<pod-name>"} / 
  kube_pod_container_resource_limits{resource="memory", pod="<pod-name>"} * 100

How the Linux kernel decides which process to terminate first during node-level memory exhaustion:

• Guaranteed (requests == limits for CPU & Memory): Kubelet assigns oom_score_adj = -997. The kernel will almost never terminate this process unless absolute panic occurs.
• Burstable (requests < limits): Kubelet calculates: oom_score_adj = min(max(2, 1000 - (1000 * memoryRequest) / totalNodeMemory), 999). Evicted before Guaranteed.
• BestEffort (no requests or limits): Kubelet assigns oom_score_adj = 1000. These are the absolute first processes killed by the kernel during host memory pressure.

Why Java applications are the #1 source of OOMKilled incidents in enterprise Kubernetes:

• The Host Core & RAM Blindness: Older JVMs (prior to Java 8u191 / Java 11) query /proc/meminfo instead of cgroup limits. On a 64GB node with a 2GB container limit, the JVM calculates default heap as 25% of 64GB = 16GB, instantly triggering OOMKilled at startup!
• Non-Heap Memory Consumption: Total JVM Memory = Heap + Metaspace + Thread Stacks + Direct Memory + Native Memory. Sizing heap to 1.8GB inside a 2.0GB container will always cause OOMKilled.
• JVM Flag Solution:

  JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0 -XX:+ExitOnOutOfMemoryError"

Tuning other modern language runtimes for container cgroup boundaries:

• Node.js V8 Heap: Node.js defaults to ~1.4GB heap. In smaller containers (e.g. 1GB limit), configure: NODE_OPTIONS="--max-old-space-size=768".
• Golang GOMEMLIMIT (Go 1.19+): Set GOMEMLIMIT=900MiB for a 1GiB container. The Go GC runs aggressively as memory approaches this ceiling, preventing OOMKilled.
• Python Memory Leaks: Profile memory leaks using tracemalloc and verify that data processing pipelines stream files rather than reading entire multi-gigabyte payloads into memory.

Automate memory sizing and practice real OOMKilled incident triage:

• Deploy Vertical Pod Autoscaler (VPA): Run VPA in updateMode: "Off" (recommendation mode) to analyze real-world peak memory usage over 14 days before setting production limits.
• Set Meaningful Memory Requests: Set memory request equal to 95th percentile working set, and memory limit to 125-150% of request to accommodate traffic spikes.
• Lab Runbook: Practice profiling memory leaks, JVM container tuning, and cgroups v2 debugging in DevOps Lab: OOMKilled Hands-On Runbook ↗.