Q: AKS deployment succeeds and the new image exists in ACR, but users still see the previous application version. What would you check?
Investigating why an Azure Kubernetes Service (AKS) cluster continues serving the previous application revision despite successful pipeline execution and image verification in Azure Container Registry (ACR).
#Kubernetes #AKS #ACR #Rollout #Troubleshooting
🎙️ Candidate Opening & Architectural Context
"This common production issue is almost always caused by using mutable tags (like :latest) with imagePullPolicy: IfNotPresent, stalled pod rollouts due to unready health checks, or Ingress routing pointing to a secondary service."
Advertisement
🛠️ Production Runbook & Step-by-Step Resolution
1
Check Pod Image Digest and imagePullPolicy
Inspect running pods in AKS. If the manifest uses myapp:latest and imagePullPolicy: IfNotPresent, existing nodes will never pull the new layer from ACR because the tag already exists locally.
kubectl get pods -l app=frontend -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'
2
Inspect Deployment Rollout Status
Check if the deployment actually rolled out or if new pods are stuck in CrashLoopBackOff or Pending, preventing old pods from being terminated.
kubectl rollout status deployment/frontend -n production
kubectl get replicaset -l app=frontend
3
Verify Ingress and Azure Application Gateway Routing
If new pods are Running and healthy, verify whether the Ingress controller or Azure Application Gateway Ingress Controller (AGIC) backend pool updated its endpoints.
Pro Tip: Best Practice: Never use :latest in production. Always tag images with immutable git commit SHAs (e.g., :git-c7e12d4) to ensure deterministic rollouts.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Never use :latest with IfNotPresent. Tag images with git commit SHAs, verify kubectl rollout status, and confirm Ingress backend endpoints."
⚡ 60-Second Elevator Pitch Talking Points
- Check imagePullPolicy and ensure immutable image tags (git SHA) instead of :latest.
- Verify kubectl rollout status: ensure new replica pods passed readiness checks.
- Compare running pod container imageID hashes against the new ACR digest.
- Inspect Ingress / Application Gateway backend pool health to ensure traffic routes to the new ReplicaSet.
Advertisement