Q: Your team creates preview environments on PR open and runs `kubectl delete namespace preview-pr-123` on PR close. While the Kubernetes namespace disappears, dozens of AWS Application Load Balancers and Elastic IPs remain active in AWS, accumulating $8,000/month in zombie charges. Why did this happen and how do you fix it?
Diagnosing and preventing cloud resource leaks (ALBs, Elastic IPs, Security Groups) when ephemeral preview Kubernetes namespaces are deleted.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Root Cause: Finalizer Deadlocks and Forceful Namespace Deletion
Inspect AWS Load Balancer Controller logs. When `kubectl delete ns` is issued, if the ingress finalizer (`ingress.k8s.aws/resources`) is stripped or the controller fails to authenticate, Kubernetes deletes the Ingress metadata, leaving the physical AWS ALB running forever.
# Check for orphaned finalizers
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.finalizers}{"\n"}{end}'
Graceful Teardown Sequence in CI Pipeline
In the GitHub Actions PR close workflow, delete Ingress and Service resources explicitly first and wait for the controller to release cloud assets before deleting the namespace.
# Graceful deletion step in GitHub Actions
kubectl delete ingress -n preview-pr-${{ github.event.pull_request.number }} --all --timeout=120s
kubectl wait --for=delete ingress --all -n preview-pr-${{ github.event.pull_request.number }} --timeout=180s
kubectl delete namespace preview-pr-${{ github.event.pull_request.number }}
Deploy Cloud Janitor (AWS-Nuke or Custom Controller)
Deploy an automated nightly janitor (using AWS-Nuke or a Python Lambda) that queries AWS ALBs tagged with `Environment: preview` and checks if the corresponding Kubernetes namespace still exists; if not, it terminates the orphaned ALB.
- Avoid dedicated ALBs per PR; share a single wildcard ingress controller across all preview namespaces.
- Ensure CI/CD teardown scripts explicitly wait for Ingress finalizers to complete before namespace deletion.
- Run an automated cloud janitor to detect and delete orphaned load balancers with missing namespace tags.