⚡ ~/naveed Interview Prep
⚡ Portfolio Home ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All Platform Engineering & IDP Interview Questions Scenario 30 of 50 in Platform Engineering & IDP
Senior Platform Engineer Platform Engineering Ephemeral Environments & Preview Apps Resource Leakage
🎯 Target Role / Context: Senior Platform Engineer Interview · Ephemeral Lifecycle Track

Q: Your team creates preview environments on PR open and runs `kubectl delete namespace preview-pr-123` on PR close. While the Kubernetes namespace disappears, dozens of AWS Application Load Balancers and Elastic IPs remain active in AWS, accumulating $8,000/month in zombie charges. Why did this happen and how do you fix it?

Diagnosing and preventing cloud resource leaks (ALBs, Elastic IPs, Security Groups) when ephemeral preview Kubernetes namespaces are deleted.

#Platform Engineering #AWS ALB #Kubernetes #Ephemeral Environments #FinOps #Cost Optimization
🎙️ Candidate Opening & Architectural Context
"When a Kubernetes namespace is deleted forcefully while cloud controllers (like the AWS Load Balancer Controller) are reconciling or when finalizers hang, Kubernetes may forcefully remove the Ingress/Service resource before the cloud controller can execute the AWS API call to delete the physical ALB, creating orphaned cloud resources."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

1

Root Cause: Finalizer Deadlocks and Forceful Namespace Deletion

Inspect AWS Load Balancer Controller logs. When `kubectl delete ns` is issued, if the ingress finalizer (`ingress.k8s.aws/resources`) is stripped or the controller fails to authenticate, Kubernetes deletes the Ingress metadata, leaving the physical AWS ALB running forever.

# Check for orphaned finalizers
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.finalizers}{"\n"}{end}'
2

Graceful Teardown Sequence in CI Pipeline

In the GitHub Actions PR close workflow, delete Ingress and Service resources explicitly first and wait for the controller to release cloud assets before deleting the namespace.

# Graceful deletion step in GitHub Actions
kubectl delete ingress -n preview-pr-${{ github.event.pull_request.number }} --all --timeout=120s
kubectl wait --for=delete ingress --all -n preview-pr-${{ github.event.pull_request.number }} --timeout=180s
kubectl delete namespace preview-pr-${{ github.event.pull_request.number }}
Advertisement
3

Deploy Cloud Janitor (AWS-Nuke or Custom Controller)

Deploy an automated nightly janitor (using AWS-Nuke or a Python Lambda) that queries AWS ALBs tagged with `Environment: preview` and checks if the corresponding Kubernetes namespace still exists; if not, it terminates the orphaned ALB.

Pro Tip: Architectural Prevention: Use shared Ingress controllers with path-based routing (e.g. `pr-42.preview.acme.com`) rather than provisioning a dedicated AWS Load Balancer for every ephemeral PR.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Prevent orphaned cloud load balancers by gracefully deleting Ingresses before deleting namespaces, and sharing a single ingress controller across all preview environments."
⚡ 60-Second Elevator Pitch Talking Points
  • Avoid dedicated ALBs per PR; share a single wildcard ingress controller across all preview namespaces.
  • Ensure CI/CD teardown scripts explicitly wait for Ingress finalizers to complete before namespace deletion.
  • Run an automated cloud janitor to detect and delete orphaned load balancers with missing namespace tags.
Advertisement
Want more Platform Engineering & IDP scenarios?
Explore our complete collection of scenario-based Platform Engineering & IDP interview runbooks.
Browse All Platform Engineering & IDP Questions →