Q: Your microservices running in Docker and containerd frequently get killed by the Linux kernel Out-Of-Memory (OOM) killer during traffic bursts. In cgroups v1, container memory limits were binary (`memory.limit_in_bytes`), causing instant SIGKILL (exit code 137) as soon as usage exceeded the limit by a single byte. With your Linux hosts upgraded to kernel 6.x and cgroups v2, you want to implement soft memory limits and proactive throttling (`memory.high`) to allow applications to shed caches and throttle gracefully before hitting hard `memory.max` OOM termination.
Configure and tune Linux cgroups v2 memory controllers for containers. Understand the operational mechanics of memory.high proactive throttling versus memory.max hard OOM kills.
memory.high proactive throttling versus memory.max hard OOM kills."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Configure and tune Linux cgroups v2 memory controllers for containers. Understand the operational mechanics of memory.high proactive throttling versus memory.max hard OOM kills.
Verify Host cgroups v2 Unified Hierarchy
Verify that your Linux distribution has mounted the unified cgroups v2 hierarchy (/sys/fs/cgroup) rather than legacy cgroups v1 hybrid mounts.
# Check mounted cgroup filesystem type
stat -fc %T /sys/fs/cgroup/
# Expected output for cgroups v2: cgroup2fs
# Verify containerd/docker cgroup driver
docker info | grep "Cgroup Version"
# Output: Cgroup Version: 2
Compare cgroups v1 vs cgroups v2 Memory Knobs
Understand the architectural shifts: cgroups v1 mixed page cache and anonymous memory into a single rigid ceiling (memory.limit_in_bytes). cgroups v2 separates memory pressure levels into memory.min (hard protection), memory.low (soft protection), memory.high (proactive throttling), and memory.max (hard ceiling causing OOM kill).
<!-- cgroups v2 Memory Hierarchy -->
Host / Container Memory Space
├── memory.max (Hard ceiling: Exceeding triggers kernel OOM killer -> Exit 137)
├── memory.high (Throttling zone: Processes throttled & forced into direct reclaim)
├── memory.low (Best-effort protection: Reclaimed only under extreme host pressure)
└── memory.min (Hard guarantee: Never reclaimed by kernel page scanner)
Configure Docker and containerd with memory.high Throttling
Launch containers specifying both soft throttling thresholds and hard maximum ceilings. When memory crosses --memory-reservation (which maps to memory.high in v2), the kernel slows down the container's processes and triggers aggressive reclaim rather than terminating the container.
# Docker run with memory reservation (memory.high) and hard limit (memory.max)
docker run -d \
--name payment-service \
--memory="2g" \
--memory-reservation="1.6g" \
--memory-swap="2g" \
payment-api:v2.1
# Inspect container cgroup v2 controller files
CONTAINER_ID=$(docker inspect payment-service --format '{{.Id}}')
cat /sys/fs/cgroup/docker/${CONTAINER_ID}/memory.high
cat /sys/fs/cgroup/docker/${CONTAINER_ID}/memory.max
cat /sys/fs/cgroup/docker/${CONTAINER_ID}/memory.events
Monitor Memory Pressure Events and Reclaim Stall Time
Read /sys/fs/cgroup/.../memory.events to track high, max, and oom event counters. Inspect memory.pressure (PSI - Pressure Stall Information) to measure the percentage of time processes stall waiting for memory pages.
# Check container memory events
cat /sys/fs/cgroup/docker/${CONTAINER_ID}/memory.events
# Output:
# low 0
# high 42
# max 0
# oom 0
# oom_kill 0
# Inspect Pressure Stall Information (PSI)
cat /sys/fs/cgroup/docker/${CONTAINER_ID}/memory.pressure
memory.high as a throttling and proactive reclaim boundary. Setting memory.high below memory.max gives containers breathing room to flush caches and throttle throughput under memory spikes instead of suffering catastrophic OOM kills."
- U
- p
- g
- r
- a
- d
- i
- n
- g
- t
- o
- c
- g
- r
- o
- u
- p
- s
- v
- 2
- e
- n
- a
- b
- l
- e
- d
- u
- s
- t
- o
- e
- l
- i
- m
- i
- n
- a
- t
- e
- s
- u
- d
- d
- e
- n
- O
- O
- M
- k
- i
- l
- l
- s
- a
- c
- r
- o
- s
- s
- o
- u
- r
- c
- o
- n
- t
- a
- i
- n
- e
- r
- f
- l
- e
- e
- t
- .
- B
- y
- c
- o
- n
- f
- i
- g
- u
- r
- i
- n
- g
- `
- m
- e
- m
- o
- r
- y
- .
- h
- i
- g
- h
- `
- a
- l
- o
- n
- g
- s
- i
- d
- e
- `
- m
- e
- m
- o
- r
- y
- .
- m
- a
- x
- `
- ,
- t
- h
- e
- L
- i
- n
- u
- x
- k
- e
- r
- n
- e
- l
- a
- p
- p
- l
- i
- e
- s
- p
- r
- o
- g
- r
- e
- s
- s
- i
- v
- e
- t
- h
- r
- o
- t
- t
- l
- i
- n
- g
- a
- n
- d
- d
- i
- r
- e
- c
- t
- p
- a
- g
- e
- r
- e
- c
- l
- a
- i
- m
- w
- h
- e
- n
- m
- e
- m
- o
- r
- y
- s
- p
- i
- k
- e
- s
- o
- c
- c
- u
- r
- ,
- a
- l
- l
- o
- w
- i
- n
- g
- s
- e
- r
- v
- i
- c
- e
- s
- t
- o
- s
- h
- e
- d
- c
- a
- c
- h
- e
- s
- g
- r
- a
- c
- e
- f
- u
- l
- l
- y
- w
- h
- i
- l
- e
- a
- l
- e
- r
- t
- i
- n
- g
- o
- u
- r
- a
- u
- t
- o
- s
- c
- a
- l
- e
- r
- s
- b
- e
- f
- o
- r
- e
- a
- n
- O
- O
- M
- k
- i
- l
- l
- c
- a
- n
- h
- a
- p
- p
- e
- n
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Executing container processes under the default unconstrained root UID instead of an unprivileged user Root container processes dramatically expand the blast radius of container escape vulnerabilities and violate enterprise security baselines.
-
Embedding sensitive API credentials or secret keys directly inside static image layers
Secrets written into intermediate image layers remain permanently recoverable in the image history even if deleted in a subsequent
RUNstep. -
Failing to configure Docker daemon.json logging driver rotation, leading to host disk space exhaustion
By default, Docker's json-file driver retains unlimited log sizes; unconfigured log rotation fills
/var/lib/docker, crashing the host runtime. - Tagging production container images with mutable latest identifiers rather than immutable git commit SHAs in docker-cgroups-v2-memory-throttling Unpinned versions create non-deterministic builds where identical code commits produce differing binary outputs over time.