Q: How do you run a privileged debug container on a running pod without modifying the pod spec?
kubectl debug -it <pod> --image=ubuntu --share-processes --copy-to=debug-pod — creates a copy of the pod with an extra debug container.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: <strong>To run a privileged debug container on a running pod without modifying the pod spec, use <code>kubectl debug -it <pod> --image=<tools-image> --target=<container></code> with a privileged profile.</strong> This adds an ephemeral container to the live pod through the <code>ephemeralcontainers</code> subresource, so your workload's containers are not restarted and the Deployment template is untouched. Interviewers ask because production images are often distroless with no shell, and you still need to inspect processes, network and files. A strong answer covers profiles, namespace sharing, the Pod Security and RBAC limits, and cleanup.
Mental model: ephemeral containers
Ephemeral containers are a special container type added to an existing pod for troubleshooting only.
- What they are: containers added through a dedicated subresource to a running pod. They have no ports, probes or resource guarantees and are never restarted.
- What is not changed: the pod's regular containers keep running, and the owning Deployment or StatefulSet template is not modified. The Pod object itself does record the ephemeral container in its status and spec.
- Availability: generally available since Kubernetes 1.25, and requires kubectl with the
debugcommand. - Permanent record: an ephemeral container cannot be removed or edited. It lives until the pod is deleted, so the pod stays 'dirty' afterwards.
- Why not edit the Deployment: changing the template to add a debug sidecar triggers a rollout and replaces the very pod you want to investigate, destroying in-memory state and the symptom.
Hands-on: run a privileged debug container on a running pod
Attach a tools image, share the target container's process namespace, and choose a profile that grants the privileges you need.
# Privileged ephemeral container attached to the 'app' container
kubectl -n prod debug -it my-app-7d9c6b5f4-x2k8p \
--image=nicolaka/netshoot \
--target=app \
--profile=sysadmin
# Inside the debug container
ps aux # sees the target's processes
ls /proc/1/root/etc # target filesystem via /proc
ss -tnlp; tcpdump -i any -c 20 port 8080
# Older kubectl: custom security context
echo '{"securityContext":{"privileged":true}}' > custom.json
kubectl -n prod debug -it my-app-7d9c6b5f4-x2k8p --image=busybox --target=app --custom=custom.json
- --target: shares the process namespace with the named container, so
psand/proc/<pid>/rootshow the application's processes and files. - --profile: in kubectl 1.30 and later,
sysadmingives a privileged container,netadminadds network capabilities, andgeneralis the default for most uses. Checkkubectl debug --helpfor your version. - Custom security context: on older versions or special needs, use
--customwith a JSON file that sets the containersecurityContext. - Copy alternative:
--copy-tomakes a new pod, which leaves the original pod untouched but loses the original's runtime state. - Stateful investigation: because the app container keeps running, you can capture live behaviour such as open sockets, memory maps and DNS answers at the moment of the problem, which a restarted copy could not show.
Requirements: RBAC, Pod Security and runtime
Several controls can stop a privileged debug container, which is intentional.
# Check your rights
kubectl auth can-i patch pods/ephemeralcontainers -n prod
kubectl auth can-i create pods/attach -n prod
# See the ephemeral container recorded on the pod
kubectl -n prod get pod my-app-7d9c6b5f4-x2k8p \
-o jsonpath='{.spec.ephemeralContainers[*].name}'
- RBAC: you need
patchorupdateonpods/ephemeralcontainers, pluspods/attachorpods/execto attach. - Pod Security Admission: in a namespace enforcing
baselineorrestricted, privileged ephemeral containers are rejected. Use a controlled exception, not weaker namespace labels for everyone. - Runtime support: the container runtime must support ephemeral containers, which containerd and CRI-O do on supported versions.
- Target limits: shared process namespace via
--targetneeds runtime support. If unsupported, you will see only the debug container's processes. - Image tooling: choose an image with the tools you need, for example
strace,tcpdump,digandcurl. Tracing another process may need extra capabilities such asSYS_PTRACE, which the profiles can add.
Limits and alternatives
Know when this technique does not fit.
- Pod can't be cleaned: once added, the container remains in the pod spec until the pod is replaced. A rollout restart or deleting the pod removes it.
- Node-level issues: if you need the host (kubelet, disk, CNI), use
kubectl debug node/<name>instead. - Cannot reproduce state: for crash loops,
--copy-towith a changed command (for examplesleep) lets you debug a failing container. - Evidence handling: be careful with tools that dump memory or traffic, as they may expose secrets and customer data.
- Resource impact: ephemeral containers have no resource guarantees, but they still consume node CPU and memory. A heavy capture on a busy pod can affect the application, so time-box it.
Production practice: break-glass debugging
Make debugging possible without weakening everyday security.
- Dedicated role: create a ClusterRole for ephemeral container access and bind it to an on-call group, ideally with time-limited access.
- Audit: API audit logs record patches to
pods/ephemeralcontainers, so alert on use in production namespaces. - Approved images: use a vetted internal debug image from your registry, scanned and pinned by digest, not an arbitrary public image.
- Clean up: delete or restart the pod after investigations so an unusual privileged container does not remain.
- I use kubectl debug to add an ephemeral container with a tools image, targeting the app container so it shares the process namespace.
- For privileges I choose the sysadmin profile or a custom security context, subject to RBAC and Pod Security Admission, then verify with ps and /proc.
- I treat it as audited break-glass access and restart the pod afterwards because ephemeral containers cannot be removed.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Assuming the pod is not modified at all The ephemeral container is recorded on the Pod and stays until the pod is deleted, though workload templates are unchanged.
- Omitting --target Without it you do not share the app's process namespace and cannot see its processes.
- Weakening namespace Pod Security labels permanently to allow debugging It lowers the security of every workload there; grant narrow break-glass exceptions.
- Using an unvetted public image A privileged container running unknown code is a serious supply chain risk.