Q: You want to run a one-off debug pod on a specific node. How?
kubectl debug node/<node-name> -it --image=ubuntu or use nodeName field in pod spec.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>To run a one-off debug pod on a specific node, use <code>kubectl debug node/<node-name> -it --image=busybox</code>, or pin an ordinary pod with <code>nodeName</code> or a <code>kubernetes.io/hostname</code> nodeSelector.</strong> The first form creates a pod already placed on that node with the host filesystem mounted at <code>/host</code>. Interviewers ask because debugging node-level problems (disk, DNS, CNI, kubelet) needs a shell next to the node, yet SSH is often unavailable on managed clusters. A good answer compares the options, explains how taints and the scheduler interact, and remembers cleanup and security.
Mental model: three ways to land a pod on a node
Each option makes a different trade-off between convenience, scheduler involvement and host access.
- kubectl debug node: creates a pod in the current namespace, scheduled directly onto the node, using host namespaces and mounting the node root filesystem read-write at
/host. Needs kubectl 1.20+ for the node form, and profiles on newer versions. - nodeName: setting
spec.nodeNameassigns the pod directly and bypasses the scheduler, so scheduler constraints, including NoSchedule taints, are not evaluated. A NoExecute taint is still enforced by the kubelet. - nodeSelector:
kubernetes.io/hostname: <node>goes through the scheduler, so taints, resource requests and quotas still apply, and the pod stays Pending if they do not fit. - Managed clusters: on EKS, GKE and AKS you often have no SSH, which makes
kubectl debug nodethe standard approach. - Typical use cases: checking disk pressure under
/var/lib/kubelet, reading container runtime logs, testing DNS from the node's network, or inspecting mounted volumes that only exist on that host.
Hands-on: run a one-off debug pod on a specific node with kubectl debug
This is the fastest method and requires only API access plus permission to create pods.
# Interactive shell on node worker-1
kubectl debug node/worker-1 -it --image=nicolaka/netshoot
# Inside the container: the node filesystem is at /host
chroot /host /bin/sh
journalctl -u kubelet --no-pager | tail -n 50
# When done
exit
kubectl get pods | grep node-debugger
kubectl delete pod node-debugger-worker-1-abcde
- Start a session: pick an image that has your tools.
busyboxis tiny, whilenicolaka/netshootadds network utilities. - Use the host filesystem: inside the container run
chroot /hostto work as if logged into the node, then read/var/logor inspect kubelet files. - Pick a profile: on kubectl 1.30 and later,
--profile=sysadmingives a privileged container and--profile=generala limited one. Profile availability depends on kubectl version, so checkkubectl debug --help. - Cleanup: the debug pod is not removed automatically. Delete it when finished, since it is a normal pod named
node-debugger-<node>-xxxxx. - Image choice: pin a version or digest of the debug image. A tool image that includes
tcpdump,digandcurlsaves time, but a smaller image pulls faster on a node with limited bandwidth.
Pinning an ordinary pod with nodeName or nodeSelector
Use a manifest when you need a specific image, resources, volumes or a service account that the debug command does not give you.
apiVersion: v1
kind: Pod
metadata:
name: debug-worker-1
spec:
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: worker-1
tolerations:
- operator: Exists # tolerate any taint for this debug pod
containers:
- name: debug
image: busybox:1.36
command: ["sleep", "3600"]
# Run it, then exec in and clean up
# kubectl apply -f debug.yaml
# kubectl exec -it debug-worker-1 -- sh
# kubectl delete pod debug-worker-1 --now
- Short-lived: set
restartPolicy: Neverand usekubectl run --rm -itstyle behaviour so the pod disappears on exit. - Scheduler path: prefer
nodeSelectorwith the hostname label. If the node is tainted, add a matchingtolerationsentry. - Bypass path:
nodeNameworks on a cordoned node, which is handy while investigating, because cordon only adds the unschedulable taint that the scheduler honours. - Host access on demand: only add
hostNetwork,hostPIDor ahostPathvolume when needed, because each widens what the pod can see. - Namespace matters: the debug pod is created in the namespace you specify, so ResourceQuota, LimitRange and Pod Security there still apply. Pick a namespace where that is acceptable, such as a dedicated debug namespace.
Verify placement and handle failures
Always confirm the pod landed where you intended and that it really sees the node state you want to inspect.
- Check placement:
kubectl get pod debug-worker-1 -o wideshows the NODE column, andkubectl describe podshows scheduling events. - Pending pod: usually a taint without a toleration, insufficient resources on a full node, or a typo in the hostname label. The node name and its hostname label can differ on some clouds, so verify with
kubectl get node worker-1 --show-labels. - Permission denied: a restricted Pod Security Admission level blocks host namespaces and privileged containers. Use a namespace such as a dedicated debug namespace with relaxed policy.
- NotReady node: API-created pods do not start on a node whose kubelet is dead, so fall back to cloud provider tools like SSM or serial console.
- Verify host view: after chroot,
hostnameshould print the node name andps auxshould show kubelet and containerd, confirming you are looking at the intended host and not just the container.
Production practice: controlled, audited node access
A node debug pod is root on the host, so govern it like SSH access.
- RBAC: restrict who can create pods in the debug namespace and who can use
pods/execthere. - Audit logging: API audit logs record pod creation and exec, which gives you a trail that SSH often lacks.
- Time-box: use
activeDeadlineSecondson pinned manifests so forgotten debug pods terminate themselves. - Prefer less access: if only network checks are needed, avoid mounting the host filesystem and use a normal pod pinned to the node.
- I use kubectl debug node/<name> with a tool-rich image, which creates a pod on that node with the host filesystem under /host.
- For custom needs I pin a pod using a hostname nodeSelector or nodeName, add tolerations if tainted, and verify with get pod -o wide.
- I treat it as root access: restrict RBAC, rely on audit logs, set deadlines and delete the pod afterward.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Using nodeName and expecting scheduler checks nodeName bypasses the scheduler, so resource fit and NoSchedule taints are not evaluated.
- Leaving node-debugger pods running They are privileged-capable pods with host filesystem access and are not cleaned up automatically.
- Debugging a NotReady node with kubectl debug The kubelet must be alive to start the pod, so use cloud console or SSM instead.
- Assuming hostname label equals node name On some providers they differ, which leaves the pod Pending.