Q: Your organization uses Argo CD with automated self-healing (`selfHeal: true`) across 200 clusters. During a major P1 outage, an on-call SRE manually updated a Deployment replica count and memory limit via `kubectl edit` to stop an outage. 45 seconds later, Argo CD automatically detected the drift and reverted the live cluster state back to the Git repository, restarting the crash loop and extending the outage. SRE leadership wants an architecture that prevents rogue manual drifts during normal operations but provides safe, auditable 'break-glass' emergency pauses with automated expiration.

Update Argo CD Application configurations with `ignoreDifferences` for fields mutated dynamically by runtime controllers, such as Horizontal Pod Autoscalers (`spec.replicas`) and mutating admission webhooks.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: core-api-service
  namespace: argocd
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
    - group: ""
      kind: Service
      jsonPointers:
        - /spec/clusterIP
        - /metadata/annotations/service.beta.kubernetes.io~1aws-load-balancer-arn

Create a standardized CLI tool or Slack ChatOps command (`/break-glass pause core-api-service --reason 'P1 outage' --ttl 2h`) that annotates the Argo CD Application to disable `selfHeal` and `automated` sync policies, recording the audit event.

# Emergency Pause Script / Hook
kubectl annotate application core-api-service -n argocd \
  argocd.argoproj.io/break-glass-active="true" \
  argocd.argoproj.io/break-glass-by="sre-oncall" \
  argocd.argoproj.io/break-glass-expires="$(date -u -v+2H +%Y-%m-%dT%H:%M:%SZ)" --overwrite

# Patch Application to disable self-healing
kubectl patch application core-api-service -n argocd --type merge -p '{
  "spec": {
    "syncPolicy": {
      "automated": null
    }
  }
}'

Deploy a controller or CronJob that inspects Application break-glass annotations. If the break-glass TTL expires without the SRE committing changes back to Git, the controller sends high-priority alerts to the team Slack channel and re-enables self-healing.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: break-glass-watchdog
  namespace: argocd
spec:
  schedule: '*/5 * * * *'
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: argocd-audit-sa
          restartPolicy: OnFailure
          containers:
            - name: watchdog
              image: ghcr.io/myorg/gitops-watchdog:latest
              command: ["/bin/watchdog", "--re-enable-expired"]

Establish a post-incident automated PR generator that detects differences between the cluster's live state and Git, creating an urgent 'Post-Incident Reconciliation' PR for developer approval.