Q: Deploying a new microservice version requires running a database schema migration before new application pods boot. If you deploy pods and run migrations simultaneously, new pods crash due to missing database columns. If the migration fails, application pods must NOT be updated. How do you design and execute this orchestration in Argo CD using Sync Waves and Resource Hooks?
Engineering a fault-tolerant GitOps deployment lifecycle in Argo CD using Sync Waves, PreSync database migration jobs, and PostSync notification hooks with automated rollback on migration failures.
#CI/CD #Argo CD #Sync Waves #Resource Hooks #Database Migrations #GitOps
🎙️ Candidate Opening & Architectural Context
"Kubernetes reconciles manifests in arbitrary order. Deploying application pods before schema migrations complete causes immediate cascading application crashes. We designed a deterministic GitOps deployment lifecycle using Argo CD Sync Waves and PreSync Job hooks."
Advertisement
⚡ Recommended Practice Lab
Want to master this scenario in a live sandbox? KodeKloud's Enterprise GitOps with ArgoCD & Kubernetes Rollouts covers this exact problem with hands-on terminal drills.
🛠️ Production Runbook & Step-by-Step Resolution
1️⃣
Configure Database Migration as an Argo CD PreSync Job Hook
Execute database schema updates before applying application workloads:
- PreSync Hook Annotation: Annotated Kubernetes migration Job with
argocd.argoproj.io/hook: PreSync. - Hook Deletion Policy: Configured
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation, HookSucceededto automatically clean up completed migration jobs. - Migration Execution: Job runs Flyway / Liquibase database schema migration against production PostgreSQL.
Pro Tip: PreSync hooks execute and MUST reach Completed status before Argo CD begins applying any other manifests in the synchronization.
2️⃣
Sequence Infrastructure & Workload Manifests via Sync Waves
Order resource application deterministically across execution phases:
- Wave 0 (Foundations): Namespaces, ServiceAccounts, RBAC roles:
argocd.argoproj.io/sync-wave: '0'. - Wave 1 (Configuration & PreSync): ConfigMaps, ExternalSecrets, and DB Migration PreSync Job:
sync-wave: '1'. - Wave 2 (Workloads): Deployments and StatefulSets:
sync-wave: '2'. - Wave 3 (Ingress & Routing): Ingress and Istio VirtualServices:
sync-wave: '3'.
Pro Tip: Argo CD waits for all resources in Wave N to reach Healthy and Synced state before advancing to Wave N+1.
Advertisement
3️⃣
Enforce Automated Sync Abort on PreSync Migration Failure
Guarantee application pods remain untouched if schema migrations fail:
- Sync Failure Behavior: If the PreSync migration Job fails (exit code 1), Argo CD halts the synchronization immediately.
- Workloads Untouched: Wave 2 (Deployments) is NEVER executed; existing running application pods continue serving traffic against the unmodified database.
- SyncFail Hook: Triggered
SyncFailnotification hook alerting the on-call engineer in Slack with the failed migration container logs.
Pro Tip: Halting the sync on PreSync failure prevents broken application pods from deploying onto a partially migrated database.
4️⃣
Deploy PostSync Verification & Slack Notification Hooks
Verify end-to-end service availability after successful sync:
- PostSync Hook: Configured lightweight test job annotated with
argocd.argoproj.io/hook: PostSyncrunning smoke tests against/healthz. - Deployment Success: Posts deployment success confirmation to Slack with git commit details and author.
- Reliability Posture: 100% elimination of deployment outages caused by schema migration race conditions.
Pro Tip: PostSync hooks provide automated smoke testing before marking the GitOps application as fully healthy.
💡 The Senior SRE Gold Nugget (Key Architectural Takeaway)
"Argo CD Sync Waves and PreSync Hooks enforce deterministic deployment ordering, executing database migrations and configuration updates before deploying application pods, and aborting rollouts if migrations fail."
⚡ 60-Second Elevator Pitch Talking Points
- Annotate database migration Jobs with argocd.argoproj.io/hook: PreSync.
- Order resources into sequential phases using sync waves: configs (Wave 1) -> pods (Wave 2) -> ingress (Wave 3).
- Automatically abort the rollout if PreSync migration jobs fail, leaving running pods safe.
- Run automated smoke tests in PostSync hooks to verify production health before marking success.
Advertisement