Q: Deploying newly fine-tuned foundation models directly to production risks catastrophic regressions in reasoning or safety. How do you engineer an automated GitOps CI/CD pipeline that runs quantitative eval benchmarks, mirrors live production traffic via shadow routing, and executes automated canary promotion?

When a new model artifact is registered in MLflow or Hugging Face, trigger a Kubernetes batch evaluation job running the Language Model Evaluation Harness (`lm-eval-harness`). Test against standardized benchmark suites: MMLU (knowledge), GSM8K (reasoning), and internal compliance test sets. Block deployment unless the candidate model passes strict threshold gates compared to the current production baseline.

# Model eval job step
lm_eval --model vllm \
  --model_args pretrained=/models/candidate-v2,tensor_parallel_size=4 \
  --tasks mmlu,gsm8k \
  --batch_size auto \
  --output_path /results/eval.json

Once eval gates pass, deploy the candidate model into a shadow environment. Configure Envoy Gateway / Istio VirtualService with traffic shadowing (`mirror: candidate-model-svc`, `mirror_percentage: 100`). Real production user prompts are duplicated asynchronously to the candidate model: candidate outputs are logged and evaluated for latency (TTFT, ITL) and safety violations without affecting user responses.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: llm-gateway
spec:
  hosts: ["api.ai.company.com"]
  http:
  - route:
    - destination: { host: llm-prod-v1, port: { number: 8000 } }
    mirror: { host: llm-candidate-v2, port: { number: 8000 } }
    mirrorPercentage: { value: 100.0 }

Promote the model to live traffic using Argo Rollouts. Configure automated canary steps (10% -> 25% -> 50% -> 100%) with automated analysis templates. Argo Rollouts monitors real-time Prometheus metrics (5xx error rate < 0.1%, p99 TTFT < 800ms). If metrics degrade, Argo automatically aborts the rollout and rolls back traffic instantly.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: llm-serving-rollout
spec:
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: { duration: 1h }
      - setWeight: 50
      - pause: { duration: 2h }
      analysis:
        templates:
        - templateName: llm-latency-and-error-metrics