Q: Several production worker nodes report containers stuck in `Terminating` status. Executing `docker kill -s 9 <id>` or `docker rm -f <id>` has no effect; the CLI hangs or returns successfully while the container remains running. Running `ps aux` reveals application threads in state `D` (Uninterruptible Sleep). A shared NFS volume or AWS EBS disk stopped responding to SCSI requests. You must explain why `SIGKILL` cannot terminate processes in `D` state, safely unfreeze the kernel subsystem, and recover the node.
Diagnose and resolve containers that cannot be stopped or killed with `docker rm -f` or `kill -9` because processes are trapped in Linux kernel Uninterruptible Sleep (`D` state) due to hanging NFS or block storage I/O.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Understand the Linux Kernel 'D' State (Uninterruptible Sleep)
A process enters `TASK_UNINTERRUPTIBLE` (`D` state) when waiting for hardware I/O (such as a disk read or NFS network RPC). In this state, the kernel guarantees data integrity by ignoring ALL signals, including `SIGKILL` (`kill -9`). The process cannot terminate until the kernel I/O call returns or times out.
<!-- Process Wait States -->
Interruptible Sleep ('S'):
Process waiting for timer / event. Kernel delivers SIGTERM/SIGKILL -> Process terminates immediately.
Uninterruptible Sleep ('D'):
Process waiting inside Kernel Driver for Disk / NFS I/O.
Kernel IGNORES ALL SIGNALS! kill -9 has ZERO effect until I/O finishes or fails!Identify the Blocked Kernel Syscall using /proc and wchan
Locate the stuck process PID and read `/proc/
# Find processes in 'D' state
ps -eo pid,stat,wchan:25,comm | grep -w D
# Sample output: 4120 D nfs_wait_client_call app-worker
# Inspect kernel stack trace of the stuck thread
cat /proc/4120/stack
# Reveals: [<0>] nfs4_proc_getattr+0x65/0x90
# [<0>] call_rwsem_down_read_failed+0x18/0x30Safely Release or Force-Unmount the Hanging Filesystem
If the cause is a frozen NFS share, perform a lazy and forced unmount (`umount -l -f /mnt/nfs`). This disconnects the mount from the VFS hierarchy, causing pending I/O requests to fail with an error and allowing the kernel to wake the thread.
# Force and lazy unmount hanging storage
sudo umount -f -l /var/lib/docker/volumes/hanging_volume/_dataHandle Blocked Block Devices (EBS / SAN)
For hanging block devices, check `dmesg` for SCSI I/O timeouts (`task blocked for more than 120 seconds`). If the block device controller is completely unresponsive, rebooting the node is the only way to clear threads stuck in kernel space.
# Check dmesg for kernel hung task warnings
dmesg -T | grep -i "blocked for more than 120 seconds"- W
- h
- e
- n
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- a
- r
- e
- s
- t
- u
- c
- k
- i
- n
- `
- T
- e
- r
- m
- i
- n
- a
- t
- i
- n
- g
- `
- a
- n
- d
- i
- g
- n
- o
- r
- e
- `
- k
- i
- l
- l
- -
- 9
- `
- ,
- t
- h
- e
- y
- a
- r
- e
- t
- r
- a
- p
- p
- e
- d
- i
- n
- L
- i
- n
- u
- x
- k
- e
- r
- n
- e
- l
- U
- n
- i
- n
- t
- e
- r
- r
- u
- p
- t
- i
- b
- l
- e
- S
- l
- e
- e
- p
- (
- `
- D
- `
- s
- t
- a
- t
- e
- )
- w
- a
- i
- t
- i
- n
- g
- o
- n
- u
- n
- r
- e
- s
- p
- o
- n
- s
- i
- v
- e
- s
- t
- o
- r
- a
- g
- e
- I
- /
- O
- .
- W
- e
- d
- i
- a
- g
- n
- o
- s
- e
- t
- h
- i
- s
- b
- y
- c
- h
- e
- c
- k
- i
- n
- g
- `
- /
- p
- r
- o
- c
- /
- <
- p
- i
- d
- >
- /
- s
- t
- a
- c
- k
- `
- t
- o
- i
- d
- e
- n
- t
- i
- f
- y
- t
- h
- e
- b
- l
- o
- c
- k
- e
- d
- k
- e
- r
- n
- e
- l
- c
- a
- l
- l
- .
- F
- o
- r
- c
- e
- -
- u
- n
- m
- o
- u
- n
- t
- i
- n
- g
- t
- h
- e
- u
- n
- d
- e
- r
- l
- y
- i
- n
- g
- v
- o
- l
- u
- m
- e
- (
- `
- u
- m
- o
- u
- n
- t
- -
- f
- -
- l
- `
- )
- a
- b
- o
- r
- t
- s
- t
- h
- e
- p
- e
- n
- d
- i
- n
- g
- I
- /
- O
- a
- n
- d
- i
- n
- s
- t
- a
- n
- t
- l
- y
- r
- e
- l
- e
- a
- s
- e
- s
- t
- h
- e
- t
- r
- a
- p
- p
- e
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- .