Q: A memory leak in a poorly configured customer container consumed all available host RAM and swap on a shared multi-tenant server. Instead of the kernel OOM killer terminating the rogue container, it evaluated badness heuristics and killed `dockerd` (PID 1420), causing all 80 containers on the host to crash instantly. SRE leadership demands a robust kernel tuning strategy ensuring that core system daemons (`dockerd`, `containerd`, `systemd`) are immune to OOM termination, prioritizing rogue container processes for slaughter.

The kernel assigns each process an OOM score between 0 and 1000 based on memory consumption (`/proc//oom_score`). The process with the highest score is terminated when RAM is exhausted. Administrators adjust this calculation using `oom_score_adj` (-1000 to +1000). A value of -1000 makes a process completely immune to the OOM killer.

<!-- Kernel OOM Scoring -->
Score Calculation: oom_score = (RAM% * 10) + oom_score_adj

- System Daemons (dockerd, kubelet): oom_score_adj = -1000 (Immune, never killed!)
- Core Host Services:               oom_score_adj = -500
- Application Containers:            oom_score_adj = +200 to +1000 (First to be killed!)

Override the systemd unit files for `docker.service` and `containerd.service` to enforce an `OOMScoreAdjust` of `-1000`.

# Create systemd override for Docker daemon
sudo mkdir -p /etc/systemd/system/docker.service.d
cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF

# Create systemd override for containerd
sudo mkdir -p /etc/systemd/system/containerd.service.d
cat <<EOF | sudo tee /etc/systemd/system/containerd.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF

sudo systemctl daemon-reload
sudo systemctl restart docker containerd

Explicitly set high positive `oom_score_adj` values on non-critical container workloads using `--oom-score-adj` to guarantee they are chosen first if memory pressure spikes.

# Launch non-critical container with maximum OOM score
docker run -d \
  --name batch-worker \
  --oom-score-adj=1000 \
  batch-processor:latest

# Verify process OOM adjustment in /proc
CONTAINER_PID=$(docker inspect batch-worker --format '{{.State.Pid}}')
cat /proc/$CONTAINER_PID/oom_score_adj
# Output: 1000

Inspect `/proc/$(pgrep dockerd)/oom_score_adj` to verify value `-1000`. Trigger simulated memory exhaustion using `stress-ng` and verify that the kernel kills only high-scoring containers while the daemon remains fully operational.

# Inspect Docker daemon OOM adjustment
cat /proc/$(pgrep -x dockerd)/oom_score_adj
# Output: -1000 (Immune)