Q: A memory leak in a poorly configured customer container consumed all available host RAM and swap on a shared multi-tenant server. Instead of the kernel OOM killer terminating the rogue container, it evaluated badness heuristics and killed `dockerd` (PID 1420), causing all 80 containers on the host to crash instantly. SRE leadership demands a robust kernel tuning strategy ensuring that core system daemons (`dockerd`, `containerd`, `systemd`) are immune to OOM termination, prioritizing rogue container processes for slaughter.
Configure Linux kernel Out-Of-Memory Killer scoring (oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion.
oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Configure Linux kernel Out-Of-Memory Killer scoring (oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion.
Understand Linux Kernel OOM Killer Scoring Heuristics
The kernel assigns each process an OOM score between 0 and 1000 based on memory consumption (/proc/). The process with the highest score is terminated when RAM is exhausted. Administrators adjust this calculation using oom_score_adj (-1000 to +1000). A value of -1000 makes a process completely immune to the OOM killer.
<!-- Kernel OOM Scoring -->
Score Calculation: oom_score = (RAM% * 10) + oom_score_adj
- System Daemons (dockerd, kubelet): oom_score_adj = -1000 (Immune, never killed!)
- Core Host Services: oom_score_adj = -500
- Application Containers: oom_score_adj = +200 to +1000 (First to be killed!)
Tune systemd Service Units for dockerd and containerd
Override the systemd unit files for docker.service and containerd.service to enforce an OOMScoreAdjust of -1000.
# Create systemd override for Docker daemon
sudo mkdir -p /etc/systemd/system/docker.service.d
cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF
# Create systemd override for containerd
sudo mkdir -p /etc/systemd/system/containerd.service.d
cat <<EOF | sudo tee /etc/systemd/system/containerd.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker containerd
Configure Container OOM Score Adjustment via Docker CLI
Explicitly set high positive oom_score_adj values on non-critical container workloads using --oom-score-adj to guarantee they are chosen first if memory pressure spikes.
# Launch non-critical container with maximum OOM score
docker run -d \
--name batch-worker \
--oom-score-adj=1000 \
batch-processor:latest
# Verify process OOM adjustment in /proc
CONTAINER_PID=$(docker inspect batch-worker --format '{{.State.Pid}}')
cat /proc/$CONTAINER_PID/oom_score_adj
# Output: 1000
Verify Daemon Immunity Under Simulated Memory Stress
Inspect /proc/$(pgrep dockerd)/oom_score_adj to verify value -1000. Trigger simulated memory exhaustion using stress-ng and verify that the kernel kills only high-scoring containers while the daemon remains fully operational.
# Inspect Docker daemon OOM adjustment
cat /proc/$(pgrep -x dockerd)/oom_score_adj
# Output: -1000 (Immune)
OOMScoreAdjust=-1000 in systemd units immunizes dockerd and containerd against kernel OOM kills. Setting positive oom_score_adj on containers ensures the Linux kernel targets rogue microservices rather than terminating the container runtime itself."
- W
- e
- e
- l
- i
- m
- i
- n
- a
- t
- e
- d
- h
- o
- s
- t
- -
- w
- i
- d
- e
- r
- u
- n
- t
- i
- m
- e
- c
- r
- a
- s
- h
- e
- s
- b
- y
- t
- u
- n
- i
- n
- g
- k
- e
- r
- n
- e
- l
- O
- O
- M
- s
- c
- o
- r
- i
- n
- g
- a
- c
- r
- o
- s
- s
- o
- u
- r
- f
- l
- e
- e
- t
- .
- W
- e
- c
- o
- n
- f
- i
- g
- u
- r
- e
- d
- `
- O
- O
- M
- S
- c
- o
- r
- e
- A
- d
- j
- u
- s
- t
- =
- -
- 1
- 0
- 0
- 0
- `
- o
- n
- t
- h
- e
- D
- o
- c
- k
- e
- r
- a
- n
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- s
- y
- s
- t
- e
- m
- d
- u
- n
- i
- t
- s
- ,
- r
- e
- n
- d
- e
- r
- i
- n
- g
- t
- h
- e
- c
- o
- n
- t
- a
- i
- n
- e
- r
- e
- n
- g
- i
- n
- e
- c
- o
- m
- p
- l
- e
- t
- e
- l
- y
- i
- m
- m
- u
- n
- e
- t
- o
- k
- e
- r
- n
- e
- l
- k
- i
- l
- l
- s
- .
- W
- h
- e
- n
- m
- e
- m
- o
- r
- y
- p
- r
- e
- s
- s
- u
- r
- e
- s
- t
- r
- i
- k
- e
- s
- ,
- t
- h
- e
- k
- e
- r
- n
- e
- l
- s
- e
- l
- e
- c
- t
- i
- v
- e
- l
- y
- t
- e
- r
- m
- i
- n
- a
- t
- e
- s
- h
- i
- g
- h
- -
- s
- c
- o
- r
- i
- n
- g
- r
- u
- n
- a
- w
- a
- y
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- w
- h
- i
- l
- e
- p
- r
- e
- s
- e
- r
- v
- i
- n
- g
- h
- o
- s
- t
- s
- t
- a
- b
- i
- l
- i
- t
- y
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Preventing Host Runtime
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Preventing Host Runtime
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Preventing Host Runtime Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Preventing Host Runtime Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.