⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Kubernetes Interview Questions Scenario 91 of 207 in Kubernetes
Senior DevOps / SRE [L2] Kubernetes Advanced Scenarios Production Scenario [L2]
🎯 Target Role / Context: Kubernetes Scenario #81 (L2) · Advanced Scenarios

Q: A developer accidentally deleted a namespace. How do you recover?

From backup/GitOps. There's no undo in kubectl. This is why GitOps (ArgoCD/Flux) matters — re-apply the Git state.

#Kubernetes #Advanced Scenarios #L2 #Container Orchestration #K8s #ArgoCD
🎙️ Candidate Opening & Architectural Context
"To <strong>recover a deleted namespace</strong> in Kubernetes, first stop further damage and protect any surviving volumes, then rebuild the namespace from a backup (Velero or an etcd-derived snapshot) or from Git via GitOps, and restore data from volume snapshots. There is no undelete: deleting a namespace removes every namespaced object in it, including Deployments, ConfigMaps, Secrets and PersistentVolumeClaims. Interviewers ask this to test incident discipline, backup literacy and whether you know what happens to persistent volumes."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: To <strong>recover a deleted namespace</strong> in Kubernetes, first stop further damage and protect any surviving volumes, then rebuild the namespace from a backup (Velero or an etcd-derived snapshot) or from Git via GitOps, and restore data from volume snapshots. There is no undelete: deleting a namespace removes every namespaced object in it, including Deployments, ConfigMaps, Secrets and PersistentVolumeClaims. Interviewers ask this to test incident discipline, backup literacy and whether you know what happens to persistent volumes.

1️⃣

Contain the Incident and Understand What Was Lost

Spend the first minutes limiting damage and measuring the blast radius, not restoring blindly.

  • What is deleted: all namespaced resources: workloads, Services, Ingresses, ConfigMaps, Secrets, PVCs, RBAC bindings, custom resources. Cluster-scoped objects and PersistentVolumes are not namespaced. Write down exactly what existed, using Git, Helm releases, and Velero or Argo CD inventories, because that list becomes your restore checklist.
  • Volumes depend on reclaim policy: a PV with Retain survives as Released; with Delete the backing disk is removed along with the PVC, unless you intervene first.
  • Still Terminating? if kubectl get ns my-ns shows Terminating, controllers are still cleaning up. Quickly patch PVs to Retain before the volumes are deleted.
  • Find the cause: Kubernetes audit logs or cloud audit logs show who issued the delete. Freeze the pipeline or credentials involved so it is not repeated.
Pro Tip: Do not force-remove namespace finalizers as a first reaction. That skips cleanup controllers and can orphan cloud resources such as load balancers and disks.
2️⃣

Hands-on: Recover a Deleted Namespace with Velero

With a recent backup, restore is a single command and the quickest path to a known-good state.

# Protect surviving volumes first
kubectl get pv | grep my-ns
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

# Restore from the latest suitable backup
velero backup get
velero restore create restore-my-ns \
  --from-backup daily-20260101020000 \
  --include-namespaces my-ns
velero restore describe restore-my-ns --details
velero restore logs restore-my-ns | tail -30
  • List backups: pick the newest backup that contains the namespace and predates the delete.
  • Restore selectively: include only the namespace; Velero recreates objects and, with CSI or file-system backup, volume data. Prefer restoring into the original name only after confirming the namespace is fully gone, otherwise a half-terminated namespace causes conflicts and partial restores.
  • Watch the restore: velero restore describe shows warnings and errors for skipped objects.
  • Cross-cluster option: Velero can restore into a different cluster, which is useful if the damage is wider or you want to validate the restore before touching production.
Advertisement
3️⃣

Rebuild from Git, Secrets and Data

If infrastructure is declared in Git, the namespace is rebuilt from source of truth rather than from a backup.

# Rebuild from Git instead of backup
kubectl create namespace my-ns
kubectl label namespace my-ns team=payments
argocd app sync my-app-prod      # or: flux reconcile kustomization my-app -n flux-system --with-source
kubectl get all,cm,secret,pvc -n my-ns
  • GitOps resync: recreate the namespace and sync the Argo CD Application or Flux Kustomization. Declared Deployments, Services and Ingresses return exactly as defined.
  • Secrets: if they come from an external store (External Secrets Operator, Vault, cloud secret manager) they are regenerated automatically. Secrets created by hand or by kubectl create are gone unless backed up.
  • Persistent data: restore from VolumeSnapshots or cloud disk snapshots, create PVs and PVCs that bind to them, then start the workloads. For databases, prefer restoring the application's own logical backups over raw volume copies.
  • Last resort: an etcd snapshot restored into a separate scratch cluster, with etcdutl snapshot restore, lets you export the needed objects with kubectl and re-apply them. Never restore etcd over a live production cluster for one namespace.
4️⃣

Verify the Recovery

Confirm both control plane objects and application behaviour.

  • Objects back: kubectl get all,cm,secret,pvc,ingress -n my-ns compared against Git or a previous inventory.
  • Pods ready and data intact: check kubectl rollout status, application health endpoints and a data sanity check such as a row count. Run an application-level check, not just kubectl: log in, read a known record, and confirm background jobs resume.
  • Dependencies: verify DNS, certificates (cert-manager reissues them), and Ingress addresses, which may have changed after recreation.
  • RBAC and quotas: RoleBindings, ResourceQuotas and NetworkPolicies are namespaced, so confirm they were restored too.
  • Cloud leftovers: load balancers and DNS records created by controllers may have been deleted too; check ExternalDNS and the cloud console for orphaned or missing resources.
5️⃣

Prevention: Backups, Guardrails and Access

Make the next accident hard or harmless.

  • Scheduled backups: velero schedule create daily --schedule="0 2 * * *" --ttl 720h, and test restores regularly in a scratch cluster.
  • Retain volumes: use a StorageClass with reclaimPolicy: Retain for critical data and keep cloud disk snapshots.
  • Admission guardrail: a ValidatingAdmissionPolicy (stable in 1.30) or Kyverno policy can deny DELETE on namespaces carrying a protection label.
  • Least privilege: developers rarely need namespace delete; grant it only to platform automation and require review for prod.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Patch the PersistentVolumes to Retain the moment you notice the delete, since a Delete reclaim policy removes the underlying disks while the namespace is still terminating."
⚡ 60-Second Elevator Pitch Talking Points
  • Deleting a namespace removes all namespaced objects and there is no undelete, so recovery means restoring from backup or Git.
  • I protect PVs with Retain first, then restore with Velero or resync GitOps, and rebuild secrets and data from snapshots.
  • I prevent repeats with scheduled and tested backups, Retain classes, admission policy and least-privilege RBAC.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Forcing namespace finalizers off immediately It bypasses controller cleanup and can orphan load balancers and disks.
  2. Not checking PV reclaim policy A Delete policy removes the backing disk with the PVC, turning an object loss into a data loss.
  3. Assuming Git contains everything Secrets, manually created resources and data are not in manifests unless backed up elsewhere.
  4. Never testing restores An untested backup often fails on missing CRDs, storage classes or cross-namespace dependencies during the real incident.
Advertisement
📥 FREE DOWNLOAD · 101-PAGE COMPANION HANDBOOK
Studying for Kubernetes & SRE Technical Rounds?
Download the complete 100-question PDF field guide covering all 11 core modules with offline diagnostic runbooks.
📥 Download PDF (Free) Read Online Guide →
Want more Kubernetes scenarios?
Explore our complete collection of scenario-based Kubernetes interview runbooks.
Browse All Kubernetes Questions →