Q: Your team creates preview environments on PR open and runs `kubectl delete namespace preview-pr-123` on PR close. While the Kubernetes namespace disappears, dozens of AWS Application Load Balancers and Elastic IPs remain active in AWS, accumulating $8,000/month in zombie charges. Why did this happen and how do you fix it?
Diagnosing and preventing cloud resource leaks (ALBs, Elastic IPs, Security Groups) when ephemeral preview Kubernetes namespaces are deleted.
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: When a Kubernetes namespace is deleted forcefully while cloud controllers (like the AWS Load Balancer Controller) are reconciling or when finalizers hang, Kubernetes may forcefully remove the Ingress/Service resource before the cloud controller can execute the AWS API call to delete the physical ALB, creating orphaned cloud resources.
Root Cause: Finalizer Deadlocks and Forceful Namespace Deletion
Inspect AWS Load Balancer Controller logs. When kubectl delete ns is issued, if the ingress finalizer (ingress.k8s.aws/resources) is stripped or the controller fails to authenticate, Kubernetes deletes the Ingress metadata, leaving the physical AWS ALB running forever.
# Check for orphaned finalizers
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.finalizers}{"\n"}{end}'
Graceful Teardown Sequence in CI Pipeline
In the GitHub Actions PR close workflow, delete Ingress and Service resources explicitly first and wait for the controller to release cloud assets before deleting the namespace.
# Graceful deletion step in GitHub Actions
kubectl delete ingress -n preview-pr-${{ github.event.pull_request.number }} --all --timeout=120s
kubectl wait --for=delete ingress --all -n preview-pr-${{ github.event.pull_request.number }} --timeout=180s
kubectl delete namespace preview-pr-${{ github.event.pull_request.number }}
Deploy Cloud Janitor (AWS-Nuke or Custom Controller)
Deploy an automated nightly janitor (using AWS-Nuke or a Python Lambda) that queries AWS ALBs tagged with Environment: preview and checks if the corresponding Kubernetes namespace still exists; if not, it terminates the orphaned ALB.
- Avoid dedicated ALBs per PR; share a single wildcard ingress controller across all preview namespaces.
- Ensure CI/CD teardown scripts explicitly wait for Ingress finalizers to complete before namespace deletion.
- Run an automated cloud janitor to detect and delete orphaned load balancers with missing namespace tags.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
- Confusing ALB 502 Bad Gateway (target connection reset) with 504 Gateway Timeout (upstream response timeout) when configuring Automated Preview Environment 502 indicates the backend target closed the TCP connection prematurely or returned malformed HTTP, whereas 504 means the target failed to respond before idle timeout.
- Setting target deregistration delay to zero instead of allowing connection draining when configuring Automated Preview Environment Instant target deregistration severs in-flight HTTP requests during auto-scaling scale-in or rolling deployments, causing immediate user-facing drops.
-
Failing to query CloudWatch ELB metrics (
HTTPCode_ELB_5XXvsHTTPCode_Target_5XX) during load balancer outages when configuring Automated Preview Environment Comparing load balancer generated errors against target generated errors immediately isolates whether the issue lies in the proxy tier or application runtime. - Overlooking Security Group rules on backend EC2/ECS instances that block health check traffic from the ALB when configuring Automated Preview Environment Target instances must explicitly permit inbound HTTP/HTTPS traffic from the ALB security group on the configured container/application port.