⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 106 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #96 (L2) · Advanced Scenarios

Q: How do you run a privileged debug container on a running pod without modifying the pod spec?

kubectl debug -it <pod> --image=ubuntu --share-processes --copy-to=debug-pod — creates a copy of the pod with an extra debug container.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s
🎙️ Candidate Opening & Architectural Context
"<strong>To run a privileged debug container on a running pod without modifying the pod spec, use <code>kubectl debug -it &lt;pod&gt; --image=&lt;tools-image&gt; --target=&lt;container&gt;</code> with a privileged profile.</strong> This adds an ephemeral container to the live pod through the <code>ephemeralcontainers</code> subresource, so your workload's containers are not restarted and the Deployment template is untouched. Interviewers ask because production images are often distroless with no shell, and you still need to inspect processes, network and files. A strong answer covers profiles, namespace sharing, the Pod Security and RBAC limits, and cleanup."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>To run a privileged debug container on a running pod without modifying the pod spec, use <code>kubectl debug -it &lt;pod&gt; --image=&lt;tools-image&gt; --target=&lt;container&gt;</code> with a privileged profile.</strong> This adds an ephemeral container to the live pod through the <code>ephemeralcontainers</code> subresource, so your workload's containers are not restarted and the Deployment template is untouched. Interviewers ask because production images are often distroless with no shell, and you still need to inspect processes, network and files. A strong answer covers profiles, namespace sharing, the Pod Security and RBAC limits, and cleanup.

1️⃣

Mental model: ephemeral containers

Ephemeral containers are a special container type added to an existing pod for troubleshooting only.

  • What they are: containers added through a dedicated subresource to a running pod. They have no ports, probes or resource guarantees and are never restarted.
  • What is not changed: the pod's regular containers keep running, and the owning Deployment or StatefulSet template is not modified. The Pod object itself does record the ephemeral container in its status and spec.
  • Availability: generally available since Kubernetes 1.25, and requires kubectl with the debug command.
  • Permanent record: an ephemeral container cannot be removed or edited. It lives until the pod is deleted, so the pod stays 'dirty' afterwards.
  • Why not edit the Deployment: changing the template to add a debug sidecar triggers a rollout and replaces the very pod you want to investigate, destroying in-memory state and the symptom.
2️⃣

Hands-on: run a privileged debug container on a running pod

Attach a tools image, share the target container's process namespace, and choose a profile that grants the privileges you need.

# Privileged ephemeral container attached to the 'app' container
kubectl -n prod debug -it my-app-7d9c6b5f4-x2k8p \
  --image=nicolaka/netshoot \
  --target=app \
  --profile=sysadmin

# Inside the debug container
ps aux                      # sees the target's processes
ls /proc/1/root/etc         # target filesystem via /proc
ss -tnlp; tcpdump -i any -c 20 port 8080

# Older kubectl: custom security context
echo '{"securityContext":{"privileged":true}}' > custom.json
kubectl -n prod debug -it my-app-7d9c6b5f4-x2k8p --image=busybox --target=app --custom=custom.json
  • --target: shares the process namespace with the named container, so ps and /proc/<pid>/root show the application's processes and files.
  • --profile: in kubectl 1.30 and later, sysadmin gives a privileged container, netadmin adds network capabilities, and general is the default for most uses. Check kubectl debug --help for your version.
  • Custom security context: on older versions or special needs, use --custom with a JSON file that sets the container securityContext.
  • Copy alternative: --copy-to makes a new pod, which leaves the original pod untouched but loses the original's runtime state.
  • Stateful investigation: because the app container keeps running, you can capture live behaviour such as open sockets, memory maps and DNS answers at the moment of the problem, which a restarted copy could not show.
Advertisement
3️⃣

Requirements: RBAC, Pod Security and runtime

Several controls can stop a privileged debug container, which is intentional.

# Check your rights
kubectl auth can-i patch pods/ephemeralcontainers -n prod
kubectl auth can-i create pods/attach -n prod

# See the ephemeral container recorded on the pod
kubectl -n prod get pod my-app-7d9c6b5f4-x2k8p \
  -o jsonpath='{.spec.ephemeralContainers[*].name}'
  • RBAC: you need patch or update on pods/ephemeralcontainers, plus pods/attach or pods/exec to attach.
  • Pod Security Admission: in a namespace enforcing baseline or restricted, privileged ephemeral containers are rejected. Use a controlled exception, not weaker namespace labels for everyone.
  • Runtime support: the container runtime must support ephemeral containers, which containerd and CRI-O do on supported versions.
  • Target limits: shared process namespace via --target needs runtime support. If unsupported, you will see only the debug container's processes.
  • Image tooling: choose an image with the tools you need, for example strace, tcpdump, dig and curl. Tracing another process may need extra capabilities such as SYS_PTRACE, which the profiles can add.
4️⃣

Limits and alternatives

Know when this technique does not fit.

  • Pod can't be cleaned: once added, the container remains in the pod spec until the pod is replaced. A rollout restart or deleting the pod removes it.
  • Node-level issues: if you need the host (kubelet, disk, CNI), use kubectl debug node/<name> instead.
  • Cannot reproduce state: for crash loops, --copy-to with a changed command (for example sleep) lets you debug a failing container.
  • Evidence handling: be careful with tools that dump memory or traffic, as they may expose secrets and customer data.
  • Resource impact: ephemeral containers have no resource guarantees, but they still consume node CPU and memory. A heavy capture on a busy pod can affect the application, so time-box it.
Pro Tip: Debugging with a privileged ephemeral container is effectively root on that pod. Treat access as break-glass, with approvals and audit logging.
5️⃣

Production practice: break-glass debugging

Make debugging possible without weakening everyday security.

  • Dedicated role: create a ClusterRole for ephemeral container access and bind it to an on-call group, ideally with time-limited access.
  • Audit: API audit logs record patches to pods/ephemeralcontainers, so alert on use in production namespaces.
  • Approved images: use a vetted internal debug image from your registry, scanned and pinned by digest, not an arbitrary public image.
  • Clean up: delete or restart the pod after investigations so an unusual privileged container does not remain.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: kubectl debug with --target and --profile=sysadmin gives root-level visibility into a live pod without touching its template, but it needs pods/ephemeralcontainers RBAC and a permissive Pod Security level."
⚡ 60-Second Elevator Pitch Talking Points
  • I use kubectl debug to add an ephemeral container with a tools image, targeting the app container so it shares the process namespace.
  • For privileges I choose the sysadmin profile or a custom security context, subject to RBAC and Pod Security Admission, then verify with ps and /proc.
  • I treat it as audited break-glass access and restart the pod afterwards because ephemeral containers cannot be removed.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Assuming the pod is not modified at all The ephemeral container is recorded on the Pod and stays until the pod is deleted, though workload templates are unchanged.
  2. Omitting --target Without it you do not share the app's process namespace and cannot see its processes.
  3. Weakening namespace Pod Security labels permanently to allow debugging It lowers the security of every workload there; grant narrow break-glass exceptions.
  4. Using an unvetted public image A privileged container running unknown code is a serious supply chain risk.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →