⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Docker & Containers Interview Questions Scenario 133 of 159 in Docker & Containers
Staff Infrastructure Architect Docker Container Runtime & Systems Engineering Production Scenario
🎯 Target Role / Context: Principal Docker Architect / Staff SRE Scenario

Q: A memory leak in a poorly configured customer container consumed all available host RAM and swap on a shared multi-tenant server. Instead of the kernel OOM killer terminating the rogue container, it evaluated badness heuristics and killed `dockerd` (PID 1420), causing all 80 containers on the host to crash instantly. SRE leadership demands a robust kernel tuning strategy ensuring that core system daemons (`dockerd`, `containerd`, `systemd`) are immune to OOM termination, prioritizing rogue container processes for slaughter.

Configure Linux kernel Out-Of-Memory Killer scoring (oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion.

#Docker #Linux #SRE #Performance #Kernel
🎙️ Candidate Opening & Architectural Context
"Configure Linux kernel Out-Of-Memory Killer scoring (oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Configure Linux kernel Out-Of-Memory Killer scoring (oom_score_adj) to safeguard the Docker daemon, containerd, and kubelet from being killed during extreme host memory exhaustion.

Step 1

Understand Linux Kernel OOM Killer Scoring Heuristics

The kernel assigns each process an OOM score between 0 and 1000 based on memory consumption (/proc//oom_score). The process with the highest score is terminated when RAM is exhausted. Administrators adjust this calculation using oom_score_adj (-1000 to +1000). A value of -1000 makes a process completely immune to the OOM killer.

<!-- Kernel OOM Scoring -->
Score Calculation: oom_score = (RAM% * 10) + oom_score_adj

- System Daemons (dockerd, kubelet): oom_score_adj = -1000 (Immune, never killed!)
- Core Host Services:               oom_score_adj = -500
- Application Containers:            oom_score_adj = +200 to +1000 (First to be killed!)
Pro Tip: Understand Linux Kernel OOM Killer Scoring Heuristics
Step 2

Tune systemd Service Units for dockerd and containerd

Override the systemd unit files for docker.service and containerd.service to enforce an OOMScoreAdjust of -1000.

# Create systemd override for Docker daemon
sudo mkdir -p /etc/systemd/system/docker.service.d
cat <<EOF | sudo tee /etc/systemd/system/docker.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF

# Create systemd override for containerd
sudo mkdir -p /etc/systemd/system/containerd.service.d
cat <<EOF | sudo tee /etc/systemd/system/containerd.service.d/oom.conf
[Service]
OOMScoreAdjust=-1000
EOF

sudo systemctl daemon-reload
sudo systemctl restart docker containerd
Pro Tip: Tune systemd Service Units for dockerd and containerd
Advertisement
Step 3

Configure Container OOM Score Adjustment via Docker CLI

Explicitly set high positive oom_score_adj values on non-critical container workloads using --oom-score-adj to guarantee they are chosen first if memory pressure spikes.

# Launch non-critical container with maximum OOM score
docker run -d \
  --name batch-worker \
  --oom-score-adj=1000 \
  batch-processor:latest

# Verify process OOM adjustment in /proc
CONTAINER_PID=$(docker inspect batch-worker --format '{{.State.Pid}}')
cat /proc/$CONTAINER_PID/oom_score_adj
# Output: 1000
Pro Tip: Configure Container OOM Score Adjustment via Docker CLI
Step 4

Verify Daemon Immunity Under Simulated Memory Stress

Inspect /proc/$(pgrep dockerd)/oom_score_adj to verify value -1000. Trigger simulated memory exhaustion using stress-ng and verify that the kernel kills only high-scoring containers while the daemon remains fully operational.

# Inspect Docker daemon OOM adjustment
cat /proc/$(pgrep -x dockerd)/oom_score_adj
# Output: -1000 (Immune)
Pro Tip: Verify Daemon Immunity Under Simulated Memory Stress
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Setting OOMScoreAdjust=-1000 in systemd units immunizes dockerd and containerd against kernel OOM kills. Setting positive oom_score_adj on containers ensures the Linux kernel targets rogue microservices rather than terminating the container runtime itself."
⚡ 60-Second Elevator Pitch Talking Points
  • W
  • e
  • e
  • l
  • i
  • m
  • i
  • n
  • a
  • t
  • e
  • d
  • h
  • o
  • s
  • t
  • -
  • w
  • i
  • d
  • e
  • r
  • u
  • n
  • t
  • i
  • m
  • e
  • c
  • r
  • a
  • s
  • h
  • e
  • s
  • b
  • y
  • t
  • u
  • n
  • i
  • n
  • g
  • k
  • e
  • r
  • n
  • e
  • l
  • O
  • O
  • M
  • s
  • c
  • o
  • r
  • i
  • n
  • g
  • a
  • c
  • r
  • o
  • s
  • s
  • o
  • u
  • r
  • f
  • l
  • e
  • e
  • t
  • .
  • W
  • e
  • c
  • o
  • n
  • f
  • i
  • g
  • u
  • r
  • e
  • d
  • `
  • O
  • O
  • M
  • S
  • c
  • o
  • r
  • e
  • A
  • d
  • j
  • u
  • s
  • t
  • =
  • -
  • 1
  • 0
  • 0
  • 0
  • `
  • o
  • n
  • t
  • h
  • e
  • D
  • o
  • c
  • k
  • e
  • r
  • a
  • n
  • d
  • c
  • o
  • n
  • t
  • a
  • i
  • n
  • e
  • r
  • d
  • s
  • y
  • s
  • t
  • e
  • m
  • d
  • u
  • n
  • i
  • t
  • s
  • ,
  • r
  • e
  • n
  • d
  • e
  • r
  • i
  • n
  • g
  • t
  • h
  • e
  • c
  • o
  • n
  • t
  • a
  • i
  • n
  • e
  • r
  • e
  • n
  • g
  • i
  • n
  • e
  • c
  • o
  • m
  • p
  • l
  • e
  • t
  • e
  • l
  • y
  • i
  • m
  • m
  • u
  • n
  • e
  • t
  • o
  • k
  • e
  • r
  • n
  • e
  • l
  • k
  • i
  • l
  • l
  • s
  • .
  • W
  • h
  • e
  • n
  • m
  • e
  • m
  • o
  • r
  • y
  • p
  • r
  • e
  • s
  • s
  • u
  • r
  • e
  • s
  • t
  • r
  • i
  • k
  • e
  • s
  • ,
  • t
  • h
  • e
  • k
  • e
  • r
  • n
  • e
  • l
  • s
  • e
  • l
  • e
  • c
  • t
  • i
  • v
  • e
  • l
  • y
  • t
  • e
  • r
  • m
  • i
  • n
  • a
  • t
  • e
  • s
  • h
  • i
  • g
  • h
  • -
  • s
  • c
  • o
  • r
  • i
  • n
  • g
  • r
  • u
  • n
  • a
  • w
  • a
  • y
  • c
  • o
  • n
  • t
  • a
  • i
  • n
  • e
  • r
  • s
  • w
  • h
  • i
  • l
  • e
  • p
  • r
  • e
  • s
  • e
  • r
  • v
  • i
  • n
  • g
  • h
  • o
  • s
  • t
  • s
  • t
  • a
  • b
  • i
  • l
  • i
  • t
  • y
  • .
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Preventing Host Runtime Copying code before package manifests (package.json, go.mod, requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls.
  2. Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Preventing Host Runtime If an application runs as PID 1 without an init system (like tini), it fails to reap zombie child processes and ignores SIGTERM signals, causing unclean 10s timeouts.
  3. Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Preventing Host Runtime Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
  4. Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Preventing Host Runtime Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.
Advertisement
Want more Docker & Containers scenarios?
Explore our complete collection of scenario-based Docker & Containers interview runbooks.
Browse All Docker & Containers Questions →