Q: AKS deployment succeeds and the new image exists in ACR, but users still see the previous application version. What would you check?
Investigating why an Azure Kubernetes Service (AKS) cluster continues serving the previous application revision despite successful pipeline execution and image verification in Azure Container Registry (ACR).
Want to master this scenario in a live sandbox? KodeKloud's CKA & CKAD Hands-On Certification Track covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Direct Answer: This common production issue is almost always caused by using mutable tags (like :latest) with imagePullPolicy: IfNotPresent, stalled pod rollouts due to unready health checks, or Ingress routing pointing to a secondary service.
Check Pod Image Digest and imagePullPolicy
Inspect running pods in AKS. If the manifest uses myapp:latest and imagePullPolicy: IfNotPresent, existing nodes will never pull the new layer from ACR because the tag already exists locally.
kubectl get pods -l app=frontend -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\t"}{.status.containerStatuses[*].imageID}{"\n"}{end}'
Inspect Deployment Rollout Status
Check if the deployment actually rolled out or if new pods are stuck in CrashLoopBackOff or Pending, preventing old pods from being terminated.
kubectl rollout status deployment/frontend -n production
kubectl get replicaset -l app=frontend
Verify Ingress and Azure Application Gateway Routing
If new pods are Running and healthy, verify whether the Ingress controller or Azure Application Gateway Ingress Controller (AGIC) backend pool updated its endpoints.
- Check imagePullPolicy and ensure immutable image tags (git SHA) instead of :latest.
- Verify kubectl rollout status: ensure new replica pods passed readiness checks.
- Compare running pod container imageID hashes against the new ACR digest.
- Inspect Ingress / Application Gateway backend pool health to ensure traffic routes to the new ReplicaSet.
Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:
-
Draining cluster worker nodes without auditing PodDisruptionBudget availability and replica headroom when addressing application
Unconditional node draining without checking PDB
maxUnavailablecan evict quorum-critical pods, causing cascading service outages. - Skipping deprecated API manifest validation tools (Pluto, Kubent) prior to initiating cluster upgrades when addressing deployment Kubernetes regularly graduates and removes beta APIs; deploying modern cluster versions with outdated manifest API versions causes immediate deployment rejections.
- Initiating control plane upgrades without verifying etcd snapshot backups and quorum health If control plane state becomes corrupted during an upgrade, restoring etcd snapshots is the sole disaster recovery mechanism.
- Ignoring kubelet daemon configuration deprecations across newly provisioned worker node instances Kubelet flags and container runtime interfaces evolve across versions; unvalidated flags prevent worker node registration with the API server.