⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Docker & Containers Interview Questions Scenario 149 of 159 in Docker & Containers
Staff Infrastructure Architect Docker Container Runtime & Systems Engineering Production Scenario
🎯 Target Role / Context: Principal Docker Architect / Staff SRE Scenario

Q: Several production worker nodes report containers stuck in `Terminating` status. Executing `docker kill -s 9 <id>` or `docker rm -f <id>` has no effect; the CLI hangs or returns successfully while the container remains running. Running `ps aux` reveals application threads in state `D` (Uninterruptible Sleep). A shared NFS volume or AWS EBS disk stopped responding to SCSI requests. You must explain why `SIGKILL` cannot terminate processes in `D` state, safely unfreeze the kernel subsystem, and recover the node.

Diagnose and resolve containers that cannot be stopped or killed with docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O.

#Docker #Linux #Troubleshooting #Kernel #SRE
🎙️ Candidate Opening & Architectural Context
"Diagnose and resolve containers that cannot be stopped or killed with docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: Diagnose and resolve containers that cannot be stopped or killed with docker rm -f or kill -9 because processes are trapped in Linux kernel Uninterruptible Sleep (D state) due to hanging NFS or block storage I/O.

Step 1

Understand the Linux Kernel 'D' State (Uninterruptible Sleep)

A process enters TASK_UNINTERRUPTIBLE (D state) when waiting for hardware I/O (such as a disk read or NFS network RPC). In this state, the kernel guarantees data integrity by ignoring ALL signals, including SIGKILL (kill -9). The process cannot terminate until the kernel I/O call returns or times out.

<!-- Process Wait States -->
Interruptible Sleep ('S'):
  Process waiting for timer / event. Kernel delivers SIGTERM/SIGKILL -> Process terminates immediately.

Uninterruptible Sleep ('D'):
  Process waiting inside Kernel Driver for Disk / NFS I/O.
  Kernel IGNORES ALL SIGNALS! kill -9 has ZERO effect until I/O finishes or fails!
Pro Tip: Understand the Linux Kernel 'D' State (Uninterruptible Sleep)
Step 2

Identify the Blocked Kernel Syscall using /proc and wchan

Locate the stuck process PID and read /proc//wchan and /proc//stack to identify the exact kernel function where the thread is sleeping.

# Find processes in 'D' state
ps -eo pid,stat,wchan:25,comm | grep -w D
# Sample output: 4120 D   nfs_wait_client_call  app-worker

# Inspect kernel stack trace of the stuck thread
cat /proc/4120/stack
# Reveals: [<0>] nfs4_proc_getattr+0x65/0x90
#          [<0>] call_rwsem_down_read_failed+0x18/0x30
Pro Tip: Identify the Blocked Kernel Syscall using /proc and wchan
Advertisement
Step 3

Safely Release or Force-Unmount the Hanging Filesystem

If the cause is a frozen NFS share, perform a lazy and forced unmount (umount -l -f /mnt/nfs). This disconnects the mount from the VFS hierarchy, causing pending I/O requests to fail with an error and allowing the kernel to wake the thread.

# Force and lazy unmount hanging storage
sudo umount -f -l /var/lib/docker/volumes/hanging_volume/_data
Pro Tip: Safely Release or Force-Unmount the Hanging Filesystem
Step 4

Handle Blocked Block Devices (EBS / SAN)

For hanging block devices, check dmesg for SCSI I/O timeouts (task blocked for more than 120 seconds). If the block device controller is completely unresponsive, rebooting the node is the only way to clear threads stuck in kernel space.

# Check dmesg for kernel hung task warnings
dmesg -T | grep -i "blocked for more than 120 seconds"
Pro Tip: Handle Blocked Block Devices (EBS / SAN)
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Processes in D state cannot be killed by kill -9 because the kernel blocks signal delivery while waiting for hardware I/O to complete. Resolving hanging containers requires force-unmounting the underlying NFS/storage device (umount -f -l) or clearing disk timeouts."
⚡ 60-Second Elevator Pitch Talking Points
  • W
  • h
  • e
  • n
  • c
  • o
  • n
  • t
  • a
  • i
  • n
  • e
  • r
  • s
  • a
  • r
  • e
  • s
  • t
  • u
  • c
  • k
  • i
  • n
  • `
  • T
  • e
  • r
  • m
  • i
  • n
  • a
  • t
  • i
  • n
  • g
  • `
  • a
  • n
  • d
  • i
  • g
  • n
  • o
  • r
  • e
  • `
  • k
  • i
  • l
  • l
  • -
  • 9
  • `
  • ,
  • t
  • h
  • e
  • y
  • a
  • r
  • e
  • t
  • r
  • a
  • p
  • p
  • e
  • d
  • i
  • n
  • L
  • i
  • n
  • u
  • x
  • k
  • e
  • r
  • n
  • e
  • l
  • U
  • n
  • i
  • n
  • t
  • e
  • r
  • r
  • u
  • p
  • t
  • i
  • b
  • l
  • e
  • S
  • l
  • e
  • e
  • p
  • (
  • `
  • D
  • `
  • s
  • t
  • a
  • t
  • e
  • )
  • w
  • a
  • i
  • t
  • i
  • n
  • g
  • o
  • n
  • u
  • n
  • r
  • e
  • s
  • p
  • o
  • n
  • s
  • i
  • v
  • e
  • s
  • t
  • o
  • r
  • a
  • g
  • e
  • I
  • /
  • O
  • .
  • W
  • e
  • d
  • i
  • a
  • g
  • n
  • o
  • s
  • e
  • t
  • h
  • i
  • s
  • b
  • y
  • c
  • h
  • e
  • c
  • k
  • i
  • n
  • g
  • `
  • /
  • p
  • r
  • o
  • c
  • /
  • <
  • p
  • i
  • d
  • >
  • /
  • s
  • t
  • a
  • c
  • k
  • `
  • t
  • o
  • i
  • d
  • e
  • n
  • t
  • i
  • f
  • y
  • t
  • h
  • e
  • b
  • l
  • o
  • c
  • k
  • e
  • d
  • k
  • e
  • r
  • n
  • e
  • l
  • c
  • a
  • l
  • l
  • .
  • F
  • o
  • r
  • c
  • e
  • -
  • u
  • n
  • m
  • o
  • u
  • n
  • t
  • i
  • n
  • g
  • t
  • h
  • e
  • u
  • n
  • d
  • e
  • r
  • l
  • y
  • i
  • n
  • g
  • v
  • o
  • l
  • u
  • m
  • e
  • (
  • `
  • u
  • m
  • o
  • u
  • n
  • t
  • -
  • f
  • -
  • l
  • `
  • )
  • a
  • b
  • o
  • r
  • t
  • s
  • t
  • h
  • e
  • p
  • e
  • n
  • d
  • i
  • n
  • g
  • I
  • /
  • O
  • a
  • n
  • d
  • i
  • n
  • s
  • t
  • a
  • n
  • t
  • l
  • y
  • r
  • e
  • l
  • e
  • a
  • s
  • e
  • s
  • t
  • h
  • e
  • t
  • r
  • a
  • p
  • p
  • e
  • d
  • c
  • o
  • n
  • t
  • a
  • i
  • n
  • e
  • r
  • .
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Triaging Containers Stuck Copying code before package manifests (package.json, go.mod, requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls.
  2. Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Triaging Containers Stuck If an application runs as PID 1 without an init system (like tini), it fails to reap zombie child processes and ignores SIGTERM signals, causing unclean 10s timeouts.
  3. Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Triaging Containers Stuck Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
  4. Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Triaging Containers Stuck Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.
Advertisement
Want more Docker & Containers scenarios?
Explore our complete collection of scenario-based Docker & Containers interview runbooks.
Browse All Docker & Containers Questions →