⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 99 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #89 (L2) · Advanced Scenarios

Q: You want to run a one-off debug pod on a specific node. How?

kubectl debug node/<node-name> -it --image=ubuntu or use nodeName field in pod spec.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>To run a one-off debug pod on a specific node, use <code>kubectl debug node/&lt;node-name&gt; -it --image=busybox</code>, or pin an ordinary pod with <code>nodeName</code> or a <code>kubernetes.io/hostname</code> nodeSelector.</strong> The first form creates a pod already placed on that node with the host filesystem mounted at <code>/host</code>. Interviewers ask because debugging node-level problems (disk, DNS, CNI, kubelet) needs a shell next to the node, yet SSH is often unavailable on managed clusters. A good answer compares the options, explains how taints and the scheduler interact, and remembers cleanup and security."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>To run a one-off debug pod on a specific node, use <code>kubectl debug node/&lt;node-name&gt; -it --image=busybox</code>, or pin an ordinary pod with <code>nodeName</code> or a <code>kubernetes.io/hostname</code> nodeSelector.</strong> The first form creates a pod already placed on that node with the host filesystem mounted at <code>/host</code>. Interviewers ask because debugging node-level problems (disk, DNS, CNI, kubelet) needs a shell next to the node, yet SSH is often unavailable on managed clusters. A good answer compares the options, explains how taints and the scheduler interact, and remembers cleanup and security.

1️⃣

Mental model: three ways to land a pod on a node

Each option makes a different trade-off between convenience, scheduler involvement and host access.

  • kubectl debug node: creates a pod in the current namespace, scheduled directly onto the node, using host namespaces and mounting the node root filesystem read-write at /host. Needs kubectl 1.20+ for the node form, and profiles on newer versions.
  • nodeName: setting spec.nodeName assigns the pod directly and bypasses the scheduler, so scheduler constraints, including NoSchedule taints, are not evaluated. A NoExecute taint is still enforced by the kubelet.
  • nodeSelector: kubernetes.io/hostname: <node> goes through the scheduler, so taints, resource requests and quotas still apply, and the pod stays Pending if they do not fit.
  • Managed clusters: on EKS, GKE and AKS you often have no SSH, which makes kubectl debug node the standard approach.
  • Typical use cases: checking disk pressure under /var/lib/kubelet, reading container runtime logs, testing DNS from the node's network, or inspecting mounted volumes that only exist on that host.
2️⃣

Hands-on: run a one-off debug pod on a specific node with kubectl debug

This is the fastest method and requires only API access plus permission to create pods.

# Interactive shell on node worker-1
kubectl debug node/worker-1 -it --image=nicolaka/netshoot

# Inside the container: the node filesystem is at /host
chroot /host /bin/sh
journalctl -u kubelet --no-pager | tail -n 50

# When done
exit
kubectl get pods | grep node-debugger
kubectl delete pod node-debugger-worker-1-abcde
  • Start a session: pick an image that has your tools. busybox is tiny, while nicolaka/netshoot adds network utilities.
  • Use the host filesystem: inside the container run chroot /host to work as if logged into the node, then read /var/log or inspect kubelet files.
  • Pick a profile: on kubectl 1.30 and later, --profile=sysadmin gives a privileged container and --profile=general a limited one. Profile availability depends on kubectl version, so check kubectl debug --help.
  • Cleanup: the debug pod is not removed automatically. Delete it when finished, since it is a normal pod named node-debugger-<node>-xxxxx.
  • Image choice: pin a version or digest of the debug image. A tool image that includes tcpdump, dig and curl saves time, but a smaller image pulls faster on a node with limited bandwidth.
Advertisement
3️⃣

Pinning an ordinary pod with nodeName or nodeSelector

Use a manifest when you need a specific image, resources, volumes or a service account that the debug command does not give you.

apiVersion: v1
kind: Pod
metadata:
  name: debug-worker-1
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/hostname: worker-1
  tolerations:
    - operator: Exists          # tolerate any taint for this debug pod
  containers:
    - name: debug
      image: busybox:1.36
      command: ["sleep", "3600"]

# Run it, then exec in and clean up
# kubectl apply -f debug.yaml
# kubectl exec -it debug-worker-1 -- sh
# kubectl delete pod debug-worker-1 --now
  • Short-lived: set restartPolicy: Never and use kubectl run --rm -it style behaviour so the pod disappears on exit.
  • Scheduler path: prefer nodeSelector with the hostname label. If the node is tainted, add a matching tolerations entry.
  • Bypass path: nodeName works on a cordoned node, which is handy while investigating, because cordon only adds the unschedulable taint that the scheduler honours.
  • Host access on demand: only add hostNetwork, hostPID or a hostPath volume when needed, because each widens what the pod can see.
  • Namespace matters: the debug pod is created in the namespace you specify, so ResourceQuota, LimitRange and Pod Security there still apply. Pick a namespace where that is acceptable, such as a dedicated debug namespace.
4️⃣

Verify placement and handle failures

Always confirm the pod landed where you intended and that it really sees the node state you want to inspect.

  • Check placement: kubectl get pod debug-worker-1 -o wide shows the NODE column, and kubectl describe pod shows scheduling events.
  • Pending pod: usually a taint without a toleration, insufficient resources on a full node, or a typo in the hostname label. The node name and its hostname label can differ on some clouds, so verify with kubectl get node worker-1 --show-labels.
  • Permission denied: a restricted Pod Security Admission level blocks host namespaces and privileged containers. Use a namespace such as a dedicated debug namespace with relaxed policy.
  • NotReady node: API-created pods do not start on a node whose kubelet is dead, so fall back to cloud provider tools like SSM or serial console.
  • Verify host view: after chroot, hostname should print the node name and ps aux should show kubelet and containerd, confirming you are looking at the intended host and not just the container.
5️⃣

Production practice: controlled, audited node access

A node debug pod is root on the host, so govern it like SSH access.

  • RBAC: restrict who can create pods in the debug namespace and who can use pods/exec there.
  • Audit logging: API audit logs record pod creation and exec, which gives you a trail that SSH often lacks.
  • Time-box: use activeDeadlineSeconds on pinned manifests so forgotten debug pods terminate themselves.
  • Prefer less access: if only network checks are needed, avoid mounting the host filesystem and use a normal pod pinned to the node.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: kubectl debug node gives host-level access without SSH, but nodeName bypasses the scheduler; use a nodeSelector when you want taints and quotas respected, and always delete the debugger pod."
⚡ 60-Second Elevator Pitch Talking Points
  • I use kubectl debug node/<name> with a tool-rich image, which creates a pod on that node with the host filesystem under /host.
  • For custom needs I pin a pod using a hostname nodeSelector or nodeName, add tolerations if tainted, and verify with get pod -o wide.
  • I treat it as root access: restrict RBAC, rely on audit logs, set deadlines and delete the pod afterward.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Using nodeName and expecting scheduler checks nodeName bypasses the scheduler, so resource fit and NoSchedule taints are not evaluated.
  2. Leaving node-debugger pods running They are privileged-capable pods with host filesystem access and are not cleaned up automatically.
  3. Debugging a NotReady node with kubectl debug The kubelet must be alive to start the pod, so use cloud console or SSM instead.
  4. Assuming hostname label equals node name On some providers they differ, which leaves the pod Pending.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →