Q: A team reserves 32 H100 GPUs at $3.50/hr each ($80k/month), but their model code is bottlenecked on CPU preprocessing, resulting in only 8% actual GPU SM utilization. How do you design an automated GPU FinOps system that tracks allocated cost vs utilized cost and enforces reclamation policies?

Deploy OpenCost across all AI clusters. Define precise hourly amortization rates for GPU instance types (e.g. AWS p4de.24xlarge = $40.96/hr, p5.48xlarge = $98.32/hr). Tag every pod and namespace with mandatory cost center labels (`cost-center: computer-vision`, `project: llama-rag`).

# OpenCost configuration mapping GPU costs per node type
node_costs:
  "p4de.24xlarge": 40.96
  "p5.48xlarge": 98.32
  "g5.12xlarge": 5.67

Join OpenCost financial allocation metrics with Prometheus DCGM metrics: `DCGM_FI_PROF_SM_ACTIVE`. Calculate the 'Financial Efficiency Ratio': `Efficiency = (DCGM_SM_ACTIVE / 100) * Allocated_Cost`. This reveals the exact dollar waste of underutilized GPUs per engineering department.

# PromQL calculating wasted GPU spend per namespace per month:
sum(opencost_gpu_cost_hourly * (1 - (DCGM_FI_PROF_SM_ACTIVE / 100))) by (namespace) * 730

Deploy a governance controller that monitors running GPU pods. If a pod maintains `< 5% DCGM_FI_PROF_SM_ACTIVE` for more than 4 consecutive hours in non-production namespaces, emit an automated Slack alert to the team owner. If unacknowledged within 2 hours, automatically evict the pod and release the node back to Karpenter for termination.

# Zombie GPU alert rule
- alert: ZombieGPUAllocation
  expr: avg_over_time(DCGM_FI_PROF_SM_ACTIVE[4h]) < 5 and on(pod) kube_pod_status_phase{phase="Running"} == 1
  for: 30m
  labels: { severity: high }
  annotations:
    summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} has wasted GPU capacity (<5% SM for 4h)"