Senior DevOps / SRE CI/CD Pipelines & Delivery Pipeline Triage

Q: Pipeline succeeds but the new version isn't deployed — how would you debug?

Diagnostic guide for when CI/CD logs show green checkmarks but the target production environment continues running old code: image tag caching, branch rules, and GitOps sync gaps.

#CI/CD #Docker #Image Tagging #GitOps #Kubernetes #GitHub Actions

🎙️ Candidate Opening & Architectural Context

"When a CI/CD pipeline shows green but production is unchanged, the issue lies in artifact immutability, deployment trigger conditions, or GitOps reconciliation gaps."

🛠️ Production Runbook & Step-by-Step Resolution

1️⃣

Verify What is Actually Running in Production

Start by inspecting the live cluster/server before checking pipeline scripts:

Run: kubectl get deployment <app> -o jsonpath='{.spec.template.spec.containers[0].image}'.
Check the image tag and digest. Does it match the newly built Git commit SHA?
Hit the application's version endpoint: curl https://app.example.com/version.

2️⃣

The ':latest' Tag & imagePullPolicy Trap

The single most common root cause in container CI/CD:

If the pipeline pushes myapp:latest and the Kubernetes Deployment manifest says image: myapp:latest with imagePullPolicy: IfNotPresent:
Kubernetes checks if a tag named latest exists locally on the node. If yes, it never pulls the new image from the registry!
Furthermore, Kubernetes detects no change in the Deployment manifest (the image string is still myapp:latest), so it triggers zero rollout!
Fix: Always use immutable image tags based on Git SHA or semantic release (e.g. myapp:sha-7f3a9b2).

3️⃣

Pipeline Step Conditions & Environment Mismatch

Audit pipeline execution steps:

Skipped Deploy Step: Did the build/test job succeed, but the deploy job was skipped because of a condition like if: github.ref == 'refs/heads/main' when building a feature branch?
Target Environment Mismatch: Did the pipeline deploy to Staging instead of Production due to environment variable configuration?
Manual Approval Gate: Is the pipeline waiting on manual approval in GitHub Actions Environments / GitLab Protected Environments?

4️⃣

GitOps Manifest Repo & Controller Audit

If using a separate manifest repository (ArgoCD / Flux):

Did the CI pipeline successfully commit and push the updated image tag to the config repo? (Check git credentials and branch protection rules).
Check ArgoCD sync status: Is the application in OutOfSync or Sync Failed state? Is auto-sync paused?
Check for Kubernetes manifest validation failure (e.g. invalid YAML or unaccepted CPU limit).

💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)

"Check the live running image tag first. Avoid mutable ':latest' tags that bypass k8s rollouts. Verify pipeline conditions, approval gates, and GitOps manifest commit chains."

⚡ 60-Second Elevator Pitch Talking Points

Verify running container image: 'kubectl get deploy <app> -o jsonpath={..image}'.
Check image tagging: Avoid ':latest' with 'imagePullPolicy: IfNotPresent' which ignores new image pushes.
Audit CI logs: Ensure the deploy job actually executed and was not skipped by branch/tag conditions.
Check GitOps repo: Verify CI successfully pushed new image tag commit to the manifest repository.
Check ArgoCD/Flux: Inspect sync status, controller errors, or paused auto-sync.