Q: Several production worker nodes report containers stuck in `Terminating` status. Executing `docker kill -s 9 <id>` or `docker rm -f <id>` has no effect; the CLI hangs or returns successfully while the container remains running. Running `ps aux` reveals application threads in state `D` (Uninterruptible Sleep). A shared NFS volume or AWS EBS disk stopped responding to SCSI requests. You must explain why `SIGKILL` cannot terminate processes in `D` state, safely unfreeze the kernel subsystem, and recover the node.
Diagnose and resolve containers that cannot be stopped or killed with docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O.
docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Diagnose and resolve containers that cannot be stopped or killed with docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O.
Understand the Linux Kernel 'D' State (Uninterruptible Sleep)
A process enters TASK_UNINTERRUPTIBLE (D state) when waiting for hardware I/O (such as a disk read or NFS network RPC). In this state, the kernel guarantees data integrity by ignoring ALL signals, including SIGKILL (kill -9). The process cannot terminate until the kernel I/O call returns or times out.
<!-- Process Wait States -->
Interruptible Sleep ('S'):
Process waiting for timer / event. Kernel delivers SIGTERM/SIGKILL -> Process terminates immediately.
Uninterruptible Sleep ('D'):
Process waiting inside Kernel Driver for Disk / NFS I/O.
Kernel IGNORES ALL SIGNALS! kill -9 has ZERO effect until I/O finishes or fails!
Identify the Blocked Kernel Syscall using /proc and wchan
Locate the stuck process PID and read /proc/ and /proc/ to identify the exact kernel function where the thread is sleeping.
# Find processes in 'D' state
ps -eo pid,stat,wchan:25,comm | grep -w D
# Sample output: 4120 D nfs_wait_client_call app-worker
# Inspect kernel stack trace of the stuck thread
cat /proc/4120/stack
# Reveals: [<0>] nfs4_proc_getattr+0x65/0x90
# [<0>] call_rwsem_down_read_failed+0x18/0x30
Safely Release or Force-Unmount the Hanging Filesystem
If the cause is a frozen NFS share, perform a lazy and forced unmount (umount -l -f /mnt/nfs). This disconnects the mount from the VFS hierarchy, causing pending I/O requests to fail with an error and allowing the kernel to wake the thread.
# Force and lazy unmount hanging storage
sudo umount -f -l /var/lib/docker/volumes/hanging_volume/_data
Handle Blocked Block Devices (EBS / SAN)
For hanging block devices, check dmesg for SCSI I/O timeouts (task blocked for more than 120 seconds). If the block device controller is completely unresponsive, rebooting the node is the only way to clear threads stuck in kernel space.
# Check dmesg for kernel hung task warnings
dmesg -T | grep -i "blocked for more than 120 seconds"
D state cannot be killed by kill -9 because the kernel blocks signal delivery while waiting for hardware I/O to complete. Resolving hanging containers requires force-unmounting the underlying NFS/storage device (umount -f -l) or clearing disk timeouts."
- W
- h
- e
- n
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- a
- r
- e
- s
- t
- u
- c
- k
- i
- n
- `
- T
- e
- r
- m
- i
- n
- a
- t
- i
- n
- g
- `
- a
- n
- d
- i
- g
- n
- o
- r
- e
- `
- k
- i
- l
- l
- -
- 9
- `
- ,
- t
- h
- e
- y
- a
- r
- e
- t
- r
- a
- p
- p
- e
- d
- i
- n
- L
- i
- n
- u
- x
- k
- e
- r
- n
- e
- l
- U
- n
- i
- n
- t
- e
- r
- r
- u
- p
- t
- i
- b
- l
- e
- S
- l
- e
- e
- p
- (
- `
- D
- `
- s
- t
- a
- t
- e
- )
- w
- a
- i
- t
- i
- n
- g
- o
- n
- u
- n
- r
- e
- s
- p
- o
- n
- s
- i
- v
- e
- s
- t
- o
- r
- a
- g
- e
- I
- /
- O
- .
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- t
- h
- i
- s
- b
- y
- c
- h
- e
- c
- k
- i
- n
- g
- `
- /
- p
- r
- o
- c
- /
- <
- p
- i
- d
- >
- /
- s
- t
- a
- c
- k
- `
- t
- o
- i
- d
- e
- n
- t
- i
- f
- y
- t
- h
- e
- b
- l
- o
- c
- k
- e
- d
- k
- e
- r
- n
- e
- l
- c
- a
- l
- l
- .
- F
- o
- r
- c
- e
- -
- u
- n
- m
- o
- u
- n
- t
- i
- n
- g
- t
- h
- e
- u
- n
- d
- e
- r
- l
- y
- i
- n
- g
- v
- o
- l
- u
- m
- e
- (
- `
- u
- m
- o
- u
- n
- t
- -
- f
- -
- l
- `
- )
- a
- b
- o
- r
- t
- s
- t
- h
- e
- p
- e
- n
- d
- i
- n
- g
- I
- /
- O
- a
- n
- d
- i
- n
- s
- t
- a
- n
- t
- l
- y
- r
- e
- l
- e
- a
- s
- e
- s
- t
- h
- e
- t
- r
- a
- p
- p
- e
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Triaging Containers Stuck
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Triaging Containers Stuck
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Triaging Containers Stuck Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Triaging Containers Stuck Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.