⚡ ~/naveed Interview Prep
⚡ Portfolio Home 👤 About Naveed Bio & Credentials 📫 Contact Naveed Get In Touch ✍️ Engineering Blog Deep Dives 🎯 Interview Hub 1,000+ Scenarios ☸️ Kubernetes Mastery Hub 24 Modules 🎮 DevOps Arcade & Quizzes Subnet Blitz ⚡ 🗺️ DevOps Roadmaps PDFs & Guides 🤖 Morpheus Analysis AI Quant ↗ 🛠️ Developer Tools Utilities 🧪 Labs & Experiments 📄 Interactive CV & Certs 🔗 All Links & Socials ⚡ Join The Dispatch (Weekly SRE Newsletter) →
← Back to All CI/CD & GitOps Interview Questions Scenario 57 of 187 in CI/CD & GitOps
Staff SRE / Principal Architect [L3] CI/CD Additional CI/CD Scenarios Staff SRE Scenario [L3]
🎯 Target Role / Context: CI/CD Scenario #48 (L3) · Additional CI/CD Scenarios

Q: How do you implement progressive delivery?

Progressive delivery releases a change to a growing slice of traffic while automated analysis decides whether to continue or roll back.

#CI/CD #Additional CI/CD Scenarios #L3 #DevOps #Automation #Pipelines
🎙️ Candidate Opening & Architectural Context
"<strong>Progressive delivery</strong> releases a change to a growing slice of traffic while automated analysis decides whether to continue or roll back. You implement it with a controller such as Argo Rollouts or Flagger, a traffic-shifting layer (ingress, service mesh or Gateway API), and metrics that define success. It goes beyond a plain rolling update because the decision is driven by real user-facing signals, not only pod readiness. Interviewers ask to see whether you can design the loop: exposure steps, metrics, thresholds, abort behaviour and the data compatibility that makes it safe."
Advertisement
⚡ Recommended Practice Lab

Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.

🛠️ Production Runbook & Step-by-Step Resolution

Direct Answer: <strong>Progressive delivery</strong> releases a change to a growing slice of traffic while automated analysis decides whether to continue or roll back. You implement it with a controller such as Argo Rollouts or Flagger, a traffic-shifting layer (ingress, service mesh or Gateway API), and metrics that define success. It goes beyond a plain rolling update because the decision is driven by real user-facing signals, not only pod readiness. Interviewers ask to see whether you can design the loop: exposure steps, metrics, thresholds, abort behaviour and the data compatibility that makes it safe.

1️⃣

Mental model: exposure, measurement, decision

Progressive delivery is a feedback loop around a release. Without that loop you only have a slower deployment, not a safer one.

  • Strategies: Canary (small percentage first), blue-green (full parallel stack and switch), and feature flags (decouple deploy from release).
  • Control loop: Shift weight, wait, query metrics, then promote or abort automatically.
  • Traffic control: Without a mesh or ingress integration, a canary weight is approximated by replica ratio, so 10 percent needs enough pods to be meaningful.
  • Decision data: Success rate, latency percentiles and saturation compared against a baseline or fixed threshold.
  • Compared with rolling update: A Kubernetes rolling update gates on readiness only. Progressive delivery adds weighted exposure, metric analysis and automatic rollback on top.
2️⃣

Hands-on: progressive delivery with Argo Rollouts

A Rollout replaces the Deployment and defines canary steps and analysis. Keep the Rollout, AnalysisTemplate and Service in the same Git path so they are reviewed and promoted together.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: {name: my-app, namespace: prod}
spec:
  replicas: 5
  selector: {matchLabels: {app: my-app}}
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: {duration: 5m}
        - analysis:
            templates: [{templateName: success-rate}]
            args: [{name: service-name, value: my-app}]
        - setWeight: 50
        - pause: {duration: 10m}
  template:
    metadata: {labels: {app: my-app}}
    spec:
      containers:
        - name: my-app
          image: registry.example.com/my-app:1.5.0
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: {name: success-rate, namespace: prod}
spec:
  args: [{name: service-name}]
  metrics:
    - name: success-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m]))
            / sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))
  • Steps: setWeight shifts traffic, pause waits, and analysis runs checks inline.
  • AnalysisTemplate: A reusable query with a success condition and failure limit.
  • Traffic routing: Add trafficRouting for NGINX, Istio, Gateway API or other supported providers for exact percentages.
  • Operate it: The kubectl plugin shows live status and supports promote and abort.
  • Blue-green variant: Setting strategy.blueGreen with autoPromotionEnabled: false runs the new version beside the old one, lets you test the preview Service, and then switches on promotion.
Advertisement
3️⃣

Operate, observe and verify the rollout

Watch each step and confirm both the controller state and the user-facing metrics.

kubectl argo rollouts get rollout my-app -n prod --watch
kubectl argo rollouts promote my-app -n prod
kubectl argo rollouts abort my-app -n prod
kubectl argo rollouts undo my-app -n prod
  • Watch: kubectl argo rollouts get rollout my-app -n prod --watch shows the current step, weights and analysis runs.
  • Promote or abort: promote skips a pause, and abort returns traffic to the stable version immediately.
  • Cross-check dashboards: Compare canary and stable error rate and latency, since an analysis query can be wrong or too lenient.
  • Confirm final state: After the last step the new ReplicaSet becomes stable and the old one scales down after the configured delay.
  • Pre-promotion checks: You can also run analysis before any traffic shifts, using smoke-test Jobs as a metric provider, to catch obvious breakage cheaply.
4️⃣

Failure modes and design trade-offs

The mechanism only protects you if signals and data are sound. A good design review asks what happens when each of these goes wrong, and answers it before production does.

  • Low traffic: If 10 percent of traffic is a handful of requests, metrics are noise. Use longer pauses or synthetic probes for low-volume services.
  • Bad metrics: A query that returns no data may pass or fail depending on configuration; test analysis templates against a deliberately broken release.
  • Stateful and schema changes: Two versions run at once, so database changes must be backward compatible.
  • Sticky sessions and caches: Users may flip between versions; make APIs and cached payloads compatible across both.
  • Rollback speed: Abort is fast for canary because stable pods are still running, whereas in-place rolling updates must pull and start old pods again.
5️⃣

Production practice

Standardise it so teams get safe defaults.

  • GitOps integration: Argo CD manages the Rollout manifest; the controller manages the traffic steps.
  • Flagger alternative: Flagger watches a normal Deployment and creates the canary resources for you, with provider support for Istio, Linkerd, NGINX and Gateway API.
  • Feature flags: Ship code dark, then release gradually per cohort without redeploying.
  • Notifications: Send abort events to chat and paging so a failed canary is investigated rather than silently retried.
  • Define SLO-based gates: Tie thresholds to your error budget so promotion rules reflect what the business tolerates and not arbitrary numbers.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Pro-Tip: Test your analysis template by deploying a deliberately failing version; a progressive delivery pipeline that has never aborted is unproven."
⚡ 60-Second Elevator Pitch Talking Points
  • Progressive delivery shifts traffic to a new version in steps while automated analysis on real metrics decides whether to promote or abort.
  • I verify with the rollout watch output, canary versus stable dashboards and a deliberate failure test.
  • I automate it with Argo Rollouts or Flagger, Prometheus analysis, GitOps and abort notifications.
⛔ Critical Candidate Mistakes to Avoid in This Scenario

Senior hiring managers and bar raisers evaluate how you prioritize during outages. Avoid these common anti-patterns:

  1. Using replica ratio as exact traffic percentage Without traffic routing a 10 percent canary of 3 pods is really 33 percent.
  2. Analysis queries that never produce data Empty results can be treated as pass or inconclusive, so a broken release is promoted.
  3. Running incompatible schema changes with canary Both versions share the database, so a breaking migration affects stable users too.
  4. Promoting on a timer alone Pauses without analysis are just a slow rollout and catch nothing.
Advertisement
Want more CI/CD & GitOps scenarios?
Explore our complete collection of scenario-based CI/CD & GitOps interview runbooks.
Browse All CI/CD & GitOps Questions →