⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 52 of 187 in CI/CD & GitOps
Senior DevOps / SRE [L2] CI/CD Additional CI/CD Scenarios Production Scenario [L2]
🎯 Target Role / Context: CI/CD Scenario #43 (L2) · Additional CI/CD Scenarios

Q: How do you version Docker images in a way that's safe for rollbacks?

To version Docker images so rollbacks are safe, give every build an immutable, unique identity: tag with the git commit SHA (and a semantic version for releases), enforce immutable tags in the registry, and deploy by digest.

#CI/CD #Additional CI/CD Scenarios #L2 #DevOps #Automation #Pipelines
🎙️ Candidate Opening & Architectural Context
"To <strong>version Docker images</strong> so rollbacks are safe, give every build an immutable, unique identity: tag with the git commit SHA (and a semantic version for releases), enforce immutable tags in the registry, and deploy by digest. Never rely on <code>latest</code>. The goal is that any running pod can be traced to a commit and that an older version can be redeployed byte for byte. Interviewers ask this because many rollback failures are caused by mutable tags, pruned images or ambiguity about what was actually running."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: To <strong>version Docker images</strong> so rollbacks are safe, give every build an immutable, unique identity: tag with the git commit SHA (and a semantic version for releases), enforce immutable tags in the registry, and deploy by digest. Never rely on <code>latest</code>. The goal is that any running pod can be traced to a commit and that an older version can be redeployed byte for byte. Interviewers ask this because many rollback failures are caused by mutable tags, pruned images or ambiguity about what was actually running.

1️⃣

Mental model: tag for humans, digest for machines

A tag is a movable label; a digest is the content hash of the manifest and cannot change. Think of the tag as the name on the box and the digest as the serial number stamped on the contents; rollback safety comes from relying on the serial number.

  • Tag: my-app:1.4.2 is a pointer in the registry that can be repointed unless the registry enforces immutability.
  • Digest: my-app@sha256:... identifies exact bytes. Deploying by digest guarantees what runs.
  • Two tags per build: A SHA tag such as my-app:3f9c2ab for traceability and a semver tag such as 1.4.2 for release communication.
  • Allowed characters: Docker tags allow letters, digits, underscore, period and hyphen and are at most 128 characters; semver build metadata with + is not valid, so use a hyphen.
  • Why not only semver: A semver tag says what a release is called but not which commit built it. The SHA closes that gap, and pre-release builds can use 1.5.0-rc.1 style tags.
Pro Tip: The tag you promote must be the tag that you tested. Version at build time and never rebuild per environment.
2️⃣

Hands-on: version Docker images in CI

Build once, label with metadata and push both tags. This takes seconds in CI and means every image carries its own provenance, which saves time during incident triage.

SHA=$(git rev-parse --short=7 HEAD)
VERSION=1.4.2
IMG=registry.example.com/my-app

docker build \
  --label org.opencontainers.image.revision=$SHA \
  --label org.opencontainers.image.version=$VERSION \
  -t $IMG:$SHA -t $IMG:$VERSION .
docker push $IMG:$SHA
docker push $IMG:$VERSION

docker buildx imagetools inspect $IMG:$VERSION | grep Digest
# Digest: sha256:<digest>   <- put this in the Deployment
  • Compute the identity: Short SHA from git rev-parse --short=7 HEAD and the version from a git tag or VERSION file.
  • Add OCI labels: org.opencontainers.image.revision and .version make the image self-describing.
  • Push and record the digest: Capture the digest after push and write it into your deployment manifest or values file.
  • Use metadata-action in GitHub: docker/metadata-action generates tags and labels consistently.
  • Dockerfile ARGs: Pass --build-arg GIT_SHA=$SHA if the application should report its own version at runtime through a /version endpoint, which makes support and incident triage faster.
Advertisement
3️⃣

Deploy by digest and roll back

Reference the digest in manifests and treat rollback as redeploying an earlier digest.

kubectl set image deployment/my-app \
  my-app=registry.example.com/my-app@sha256:<previous-digest> -n prod
kubectl rollout status deployment/my-app -n prod
kubectl get pods -n prod -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'
  • Manifest reference: image: registry.example.com/my-app:1.4.2@sha256:<digest> keeps a readable tag while the digest is what Kubernetes actually pulls.
  • Rollback: Revert the commit that changed the digest in GitOps, or run kubectl rollout undo for an emergency.
  • Verify what runs: Check .status.containerStatuses[*].imageID on pods against your record.
  • Trace back: docker inspect --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}' <image> returns the commit.
  • Helm and Kustomize: Store the digest in values.yaml or a Kustomize image override, so a Git revert rolls the whole release back with the same review process.
4️⃣

Failure modes and edge cases

These choices look fine until a rollback is needed. Each of these shows up only when you need to recover quickly, which is the worst time to discover them.

  • latest in production: Two nodes can pull different content under the same tag, and the previous version is unrecoverable by name.
  • Mutable semver tags: Rebuilding 1.4.2 to fix a typo breaks every record that referred to it. Publish 1.4.3 instead.
  • Registry cleanup: Lifecycle policies must keep the last N release images and anything referenced by a live deployment.
  • Multi-arch images: A tag points to an index of per-platform manifests; pin the index digest so every node architecture resolves consistently.
  • Pull policy: With a digest reference imagePullPolicy matters less, but a tag-only reference with IfNotPresent can leave nodes on stale cached content; this is another reason to pin digests.
5️⃣

Production practice

Enforce the rules in the registry and pipeline, not by convention.

  • Immutable tags: Enable tag immutability on ECR or an equivalent setting on other registries.
  • Admission policy: Reject pods that use latest or tag-only references with Kyverno or Gatekeeper.
  • Automated versions: Tools like semantic-release or release-please derive semver from commit history.
  • Sign and attest: Sign digests with cosign so you can prove the artifact that rolled back is the one you built.
  • Retention rule of thumb: Keep every released version indefinitely or per compliance, keep the last N commit-SHA builds, and expire untagged manifests after a grace period.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Publish a commit-SHA tag for traceability and a semver tag for humans, but let the digest in Git be the single source of truth for what runs."
⚡ 60-Second Elevator Pitch Talking Points
  • Safe versioning means every build gets an immutable identity: commit SHA and semver tags, with deployments pinned by digest.
  • I verify by comparing the running imageID with the recorded digest and reading the revision label from the image.
  • I automate semver generation, immutable registry tags, admission rules against latest, and retention that protects released images.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Deploying latest The tag moves, so you cannot identify or reproduce what is running.
  2. Re-pushing an existing version tag Rollback targets and audit records no longer match the actual bytes.
  3. Using + from semver build metadata in a tag Docker tags do not allow +, so the push fails or tags get mangled.
  4. Pruning old images by age alone The previous version may be deleted just when you need to roll back.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →