Q: A Kubernetes worker node loses connectivity with the control plane and kubelet becomes unresponsive. SREs logging into the node attempt to run `docker ps` but realize the node runs containerd. Running `ctr containers list` shows dozens of obscure UUIDs with no pod names. You must guide the team on how to use `crictl` for Kubernetes pod-aware triage, `nerdctl` for Docker-compatible developer tooling, and low-level `ctr` to directly inspect containerd runtime internals.
Master node-level container debugging using `crictl`, `nerdctl`, and `ctr`. Troubleshoot Pod sandboxes, pull images directly into the `k8s.io` namespace, and inspect container PID/cgroup states during kubelet outages.
Want to master this scenario in a live sandbox? KodeKloud's Docker Certified Associate (DCA) Hands-On Lab Course covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Master the CRI Tooling Landscape: crictl vs nerdctl vs ctr
Map tool roles: `crictl` is the Kubernetes CRI validation and debugging tool; it understands Pods, Pod sandboxes, and K8s namespaces. `ctr` is containerd's bare-bones engineering CLI. `nerdctl` is the modern containerd-native drop-in replacement for Docker CLI.
# Tool Summary:
# crictl -> CRI client, speaks to /run/containerd/containerd.sock (K8s pod-aware)
# nerdctl -> Docker-compatible CLI for containerd (supports compose, rootless, buildkit)
# ctr -> Low-level containerd internal client (requires explicit -n <namespace>)Debug Pod Sandboxes and Containers Using crictl
Configure `/etc/crictl.yaml` and inspect pod sandboxes, container states, and container runtime logs directly from the node without requiring a functioning kubelet or api-server.
# Configure crictl endpoint
cat <<EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF
# List Pod Sandboxes
crictl pods
# Inspect pod sandbox network status and IP
crictl inspectp <POD_ID> | jq .status.network
# Inspect failed container logs directly from node disk
crictl logs <CONTAINER_ID>Pull and Debug Images in the k8s.io Namespace with nerdctl
Use `nerdctl` to pull private images directly into the Kubernetes containerd namespace (`k8s.io`) and execute interactive debugging containers sharing the pod's network namespace.
# Pull image directly into Kubernetes namespace
nerdctl -n k8s.io pull registry.internal/debug-tools:latest
# Run interactive container inside the existing Kubernetes pod network namespace
nerdctl -n k8s.io run --rm -it \
--net=container:<CONTAINER_ID> \
nicolaka/netshootTriage Low-Level containerd Tasks with ctr
When CRI is completely deadlocked, use `ctr` to check whether the underlying containerd daemon and runc tasks are alive.
# Check containerd tasks in k8s.io namespace
ctr -n k8s.io tasks list
# Check containerd plugins health
ctr plugins list | grep -E 'cri|io.containerd'- W
- h
- e
- n
- t
- r
- i
- a
- g
- i
- n
- g
- c
- o
- n
- t
- a
- i
- n
- e
- r
- d
- -
- b
- a
- s
- e
- d
- K
- u
- b
- e
- r
- n
- e
- t
- e
- s
- n
- o
- d
- e
- s
- d
- u
- r
- i
- n
- g
- c
- o
- n
- t
- r
- o
- l
- p
- l
- a
- n
- e
- o
- u
- t
- a
- g
- e
- s
- ,
- w
- e
- r
- e
- l
- y
- o
- n
- `
- c
- r
- i
- c
- t
- l
- `
- t
- o
- i
- n
- s
- p
- e
- c
- t
- p
- o
- d
- s
- a
- n
- d
- b
- o
- x
- e
- s
- a
- n
- d
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- t
- a
- t
- e
- s
- w
- i
- t
- h
- o
- u
- t
- d
- e
- p
- e
- n
- d
- i
- n
- g
- o
- n
- k
- u
- b
- e
- l
- e
- t
- .
- F
- o
- r
- c
- o
- m
- p
- l
- e
- x
- t
- r
- o
- u
- b
- l
- e
- s
- h
- o
- o
- t
- i
- n
- g
- ,
- `
- n
- e
- r
- d
- c
- t
- l
- `
- a
- l
- l
- o
- w
- s
- d
- e
- v
- e
- l
- o
- p
- e
- r
- s
- t
- o
- r
- u
- n
- n
- e
- t
- s
- h
- o
- o
- t
- c
- o
- n
- t
- a
- i
- n
- e
- r
- s
- s
- h
- a
- r
- i
- n
- g
- p
- o
- d
- n
- e
- t
- w
- o
- r
- k
- n
- a
- m
- e
- s
- p
- a
- c
- e
- s
- ,
- w
- h
- i
- l
- e
- `
- c
- t
- r
- `
- s
- e
- r
- v
- e
- s
- a
- s
- o
- u
- r
- l
- a
- s
- t
- -
- r
- e
- s
- o
- r
- t
- t
- o
- o
- l
- f
- o
- r
- v
- e
- r
- i
- f
- y
- i
- n
- g
- u
- n
- d
- e
- r
- l
- y
- i
- n
- g
- r
- u
- n
- c
- t
- a
- s
- k
- s
- .