Q: A Kubernetes worker node loses connectivity with the control plane and kubelet becomes unresponsive. SREs logging into the node attempt to run `docker ps` but realize the node runs containerd. Running `ctr containers list` shows dozens of obscure UUIDs with no pod names. You must guide the team on how to use `crictl` for Kubernetes pod-aware triage, `nerdctl` for Docker-compatible developer tooling, and low-level `ctr` to directly inspect containerd runtime internals.
Master node-level container debugging using crictl, nerdctl, and ctr. Troubleshoot Pod sandboxes, pull images directly into the k8s.io namespace, and inspect container PID/cgroup states during kubelet outages.
crictl, nerdctl, and ctr. Troubleshoot Pod sandboxes, pull images directly into the k8s.io namespace, and inspect container PID/cgroup states during kubelet outages."
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: Master node-level container debugging using crictl, nerdctl, and ctr. Troubleshoot Pod sandboxes, pull images directly into the k8s.io namespace, and inspect container PID/cgroup states during kubelet outages.
Master the CRI Tooling Landscape: crictl vs nerdctl vs ctr
Map tool roles: crictl is the Kubernetes CRI validation and debugging tool; it understands Pods, Pod sandboxes, and K8s namespaces. ctr is containerd's bare-bones engineering CLI. nerdctl is the modern containerd-native drop-in replacement for Docker CLI.
# Tool Summary:
# crictl -> CRI client, speaks to /run/containerd/containerd.sock (K8s pod-aware)
# nerdctl -> Docker-compatible CLI for containerd (supports compose, rootless, buildkit)
# ctr -> Low-level containerd internal client (requires explicit -n <namespace>)
Debug Pod Sandboxes and Containers Using crictl
Configure /etc/crictl.yaml and inspect pod sandboxes, container states, and container runtime logs directly from the node without requiring a functioning kubelet or api-server.
# Configure crictl endpoint
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
# List Pod Sandboxes
crictl pods
# Inspect pod sandbox network status and IP
crictl inspectp <POD_ID> | jq .status.network
# Inspect failed container logs directly from node disk
crictl logs <CONTAINER_ID>
Pull and Debug Images in the k8s.io Namespace with nerdctl
Use nerdctl to pull private images directly into the Kubernetes containerd namespace (k8s.io) and execute interactive debugging containers sharing the pod's network namespace.
# Pull image directly into Kubernetes namespace
nerdctl -n k8s.io pull registry.internal/debug-tools:latest
# Run interactive container inside the existing Kubernetes pod network namespace
nerdctl -n k8s.io run --rm -it \
--net=container:<CONTAINER_ID> \
nicolaka/netshoot
Triage Low-Level containerd Tasks with ctr
When CRI is completely deadlocked, use ctr to check whether the underlying containerd daemon and runc tasks are alive.
# Check containerd tasks in k8s.io namespace
ctr -n k8s.io tasks list
# Check containerd plugins health
ctr plugins list | grep -E 'cri|io.containerd'
crictl is the proper CLI for triaging Kubernetes nodes because it understands Pod sandboxes and CRI abstractions. nerdctl provides a Docker-compatible UX atop containerd, while ctr provides raw runtime access when the CRI plugin itself is unresponsive."
- W
- h
- e
- n
- t
- r
- i
- a
- g
- i
- n
- g
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- -
- b
- a
- s
- e
- d
- K
- u
- b
- e
- r
- n
- e
- t
- e
- s
- n
- o
- d
- e
- s
- d
- u
- r
- i
- n
- g
- c
- o
- n
- t
- r
- o
- l
- p
- l
- a
- n
- e
- o
- u
- t
- a
- g
- e
- s
- ,
- w
- e
- r
- e
- l
- y
- o
- n
- `
- c
- r
- i
- c
- t
- l
- `
- t
- o
- i
- n
- s
- p
- e
- c
- t
- p
- o
- d
- s
- a
- n
- d
- b
- o
- x
- e
- s
- a
- n
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- t
- a
- t
- e
- s
- w
- i
- t
- h
- o
- u
- t
- d
- e
- p
- e
- n
- d
- i
- n
- g
- o
- n
- k
- u
- b
- e
- l
- e
- t
- .
- F
- o
- r
- c
- o
- m
- p
- l
- e
- x
- t
- r
- o
- u
- b
- l
- e
- s
- h
- o
- o
- t
- i
- n
- g
- ,
- `
- n
- e
- r
- d
- c
- t
- l
- `
- a
- l
- l
- o
- w
- s
- d
- e
- v
- e
- l
- o
- p
- e
- r
- s
- t
- o
- r
- u
- n
- n
- e
- t
- s
- h
- o
- o
- t
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- s
- h
- a
- r
- i
- n
- g
- p
- o
- d
- n
- e
- t
- w
- o
- r
- k
- n
- a
- m
- e
- s
- p
- a
- c
- e
- s
- ,
- w
- h
- i
- l
- e
- `
- c
- t
- r
- `
- s
- e
- r
- v
- e
- s
- a
- s
- o
- u
- r
- l
- a
- s
- t
- -
- r
- e
- s
- o
- r
- t
- t
- o
- o
- l
- f
- o
- r
- v
- e
- r
- i
- f
- y
- i
- n
- g
- u
- n
- d
- e
- r
- l
- y
- i
- n
- g
- r
- u
- n
- c
- t
- a
- s
- k
- s
- .
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Structuring Dockerfile instructions inefficiently by copying app source before package manifests when configuring Container Diagnostics crictl
Copying code before package manifests (
package.json,go.mod,requirements.txt) invalidates the build cache on every commit, forcing slow full reinstalls. -
Neglecting the PID 1 process reap problem and omitting proper SIGTERM signal forwarding when configuring Container Diagnostics crictl
If an application runs as PID 1 without an init system (like
tini), it fails to reap zombie child processes and ignoresSIGTERMsignals, causing unclean 10s timeouts. - Assembling single-stage images containing compilers, build SDKs, and header files in production when configuring Container Diagnostics crictl Single-stage builds bloat image size by gigabytes and expose extensive package vulnerabilities; multi-stage builds isolate artifacts into minimal distroless images.
- Confusing container port forwarding on bridge networks with unisolated host networking mode when configuring Container Diagnostics crictl Bridge networking utilizes iptables NAT translations which add latency overhead, whereas host networking bypasses the network namespace entirely.