Q: A production Kubernetes node running containerd suddenly enters `NotReady` status. Kubelet logs report `rpc error: code = DeadlineExceeded desc = context deadline exceeded` while calling `RuntimeService.Status` on `/run/containerd/containerd.sock`. Running `crictl ps` hangs indefinitely, while host CPU and memory appear normal. You must systematically triage the containerd daemon, inspect shim processes, analyze gRPC socket health, and restore container runtime functionality without rebooting the bare-metal worker node.
Diagnose and resolve containerd CRI socket freezes (/run/containerd/containerd.sock), client timeouts in kubelet, and distinguish between low-level ctr vs Kubernetes-focused crictl tooling.
/run/containerd/containerd.sock), client timeouts in kubelet, and distinguish between low-level ctr vs Kubernetes-focused crictl tooling."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Diagnose and resolve containerd CRI socket freezes (/run/containerd/containerd.sock), client timeouts in kubelet, and distinguish between low-level ctr vs Kubernetes-focused crictl tooling.
Differentiate CLI Tools: crictl vs ctr vs nerdctl
Understand the tool boundaries: crictl communicates over the Kubernetes CRI gRPC socket (/run/containerd/containerd.sock) using the k8s.io namespace. ctr is a low-level debugging CLI shipped directly with containerd that bypasses the CRI plugin entirely. nerdctl provides a Docker-compatible CLI for containerd with compose and rootless support.
# crictl targets the Kubernetes CRI plugin
crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps
# ctr operates directly on containerd internal namespaces (bypassing CRI)
ctr -n k8s.io containers list
ctr -n default images list
# Inspect containerd socket availability via curl/socat
echo -e "GET / HTTP/1.0\r\n" | socat - UNIX-CONNECT:/run/containerd/containerd.sock
Inspect containerd Daemon Thread Dumps and Tracing
Send SIGUSR1 to the containerd process to trigger a stack trace dump into journalctl without terminating active workloads. Check for gRPC worker goroutines deadlocked on filesystem I/O locks or unresponsive shims.
# Send SIGUSR1 to generate goroutine stack trace
pkill -SIGUSR1 containerd
# Inspect stack traces in systemd journal
journalctl -u containerd -n 200 --no-pager | grep -A 20 "goroutine"
Triage Hanging containerd-shim-runc-v2 Processes
A frozen container or broken storage volume can cause a specific containerd-shim-runc-v2 process to lock up the containerd event monitor. Identify orphaned or deadlocked shims and inspect their wait state in /proc.
# Find shims without active runc parent or in 'D' state (uninterruptible sleep)
ps -eo pid,ppid,stat,wchan:20,comm | grep -E 'containerd-shim|D'
# Trace hanging system calls on the containerd daemon
strace -p $(pgrep -x containerd) -f -e trace=network,futex,epoll_wait -s 256
Repair Socket and Safely Recover Without Pod Eviction
If containerd's state database (/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db) or containerd socket is deadlocked, restart containerd. Because containerd uses decoupled shims (shim-v2), running containers continue executing uninterrupted during daemon restart.
# Restart containerd daemon (running containers remain alive thanks to shims)
systemctl restart containerd
# Verify CRI endpoint responsiveness
crictl --runtime-endpoint unix:///run/containerd/containerd.sock info | jq .status
containerd-shim-runc-v2. When the CRI socket hangs, crictl hangs because it uses the CRI gRPC plugin, whereas ctr can verify if containerd's core runtime is functional. Restarting containerd preserves running workloads."
- W
- h
- e
- n
- a
- n
- o
- d
- e
- e
- n
- t
- e
- r
- s
- N
- o
- t
- R
- e
- a
- d
- y
- d
- u
- e
- t
- o
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- C
- R
- I
- s
- o
- c
- k
- e
- t
- t
- i
- m
- e
- o
- u
- t
- s
- ,
- w
- e
- s
- e
- n
- d
- `
- S
- I
- G
- U
- S
- R
- 1
- `
- t
- o
- c
- a
- p
- t
- u
- r
- e
- g
- o
- r
- o
- u
- t
- i
- n
- e
- d
- e
- a
- d
- l
- o
- c
- k
- s
- a
- n
- d
- i
- d
- e
- n
- t
- i
- f
- y
- b
- l
- o
- c
- k
- e
- d
- s
- h
- i
- m
- s
- i
- n
- u
- n
- i
- n
- t
- e
- r
- r
- u
- p
- t
- i
- b
- l
- e
- s
- l
- e
- e
- p
- (
- '
- D
- '
- s
- t
- a
- t
- e
- )
- .
- B
- e
- c
- a
- u
- s
- e
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- d
- e
- l
- e
- g
- a
- t
- e
- s
- c
- o
- n
- t
- a
- i
- n
- e
- r
- l
- i
- f
- e
- c
- y
- c
- l
- e
- s
- t
- o
- d
- e
- c
- o
- u
- p
- l
- e
- d
- s
- h
- i
- m
- p
- r
- o
- c
- e
- s
- s
- e
- s
- ,
- r
- e
- s
- t
- a
- r
- t
- i
- n
- g
- `
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- `
- r
- e
- c
- o
- n
- n
- e
- c
- t
- s
- t
- h
- e
- C
- R
- I
- s
- o
- c
- k
- e
- t
- t
- o
- e
- x
- i
- s
- t
- i
- n
- g
- p
- o
- d
- s
- w
- i
- t
- h
- o
- u
- t
- t
- e
- r
- m
- i
- n
- a
- t
- i
- n
- g
- c
- u
- s
- t
- o
- m
- e
- r
- w
- o
- r
- k
- l
- o
- a
- d
- s
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring containerd Socket Hangs
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring containerd Socket Hangs
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring containerd Socket Hangs Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring containerd Socket Hangs Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.