Q: Your microservices communicate over an Istio service mesh with mutual TLS (mTLS). You need to roll out new backend versions using A/B testing (routing internal employee requests with header 'X-Canary: always' to the new version) before gradually shifting 10% -> 50% -> 100% of public traffic. How do you implement Flagger with Istio VirtualServices to automate this progressive rollout?
Engineering a zero-downtime progressive delivery pipeline on Kubernetes using Flagger, Istio VirtualServices, automated webhooks, and A/B testing based on HTTP request headers.
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
Deploy Flagger Controller & Declare Canary Custom Resource
Establish the automated delivery controller and target deployment spec:
- Flagger Installation: Deployed Flagger controller with Istio provider enabled in
istio-system. - Canary Custom Resource: Created
CanaryCR targetingorder-servicedeployment withprovider: istio,progressDeadlineSeconds: 60, andinterval: 1m.
Configure Header-Based A/B Testing Routing in Istio VirtualService
Direct specific users or QA testers to canary pods before public release:
- A/B Testing Match Rule: Configured Flagger
route: { match: [ { headers: { 'x-canary': { exact: 'always' } } } ] }. - Internal Validation: Internal employees and automated E2E test suites pass
X-Canary: always, testing the live canary pods in production with zero risk to general customers.
Transition to Weighted Traffic Shifting & Prometheus Analysis
Shift public traffic in controlled increments while measuring telemetry:
- Traffic Shifting: Configured
stepWeight: 10,maxWeight: 50, shifting traffic 10% -> 20% -> 30% -> 40% -> 50%. - Istio Metrics: Flagger queries Istio Prometheus telemetry:
istio_requests_totalandistio_request_duration_milliseconds_bucket, validating error rate < 1% and p99 latency < 500ms.
Integrate Acceptance Test Webhooks & Final Promotion
Execute automated smoke tests before promoting canary to primary:
- Pre-Rollout Webhook: Flagger calls a test runner pod executing k6 integration tests against
order-service-canary:8080. - Final Promotion: When 50% step succeeds with zero errors, Flagger scales the primary deployment to the new image version, points 100% traffic to primary, and scales canary pods down.
- Deploy Flagger to manage Istio VirtualServices and canary deployments declaratively.
- Test live production canary pods using X-Canary header matching for internal teams.
- Shift public traffic in 10% increments while verifying Istio Prometheus error rates.
- Execute synthetic k6 test webhooks and automate final primary promotion seamlessly.