Q: Your AKS cluster running batch data processing, machine learning training, and test environments accounts for $40,000/month in cloud compute bills. How do you integrate Azure Spot Node Pools to reduce compute costs by 60-80% without failing mission-critical batch workflows when Azure reclaims spot capacity?

Add an elastic Spot node pool to the existing production cluster:

• CLI Pool Creation: Provisioned spot pool with az aks nodepool add --resource-group rg-aks --cluster-name aks-prod --name spotpool --priority Spot --eviction-policy Delete --spot-max-price -1 --enable-cluster-autoscaler --min-count 0 --max-count 50 --node-vm-size Standard_D8s_v5 --node-taints kubernetes.azure.com/scalesetpriority=spot:NoSchedule.
• Spot Max Price: Configured --spot-max-price -1 to indicate willingness to pay up to standard on-demand price, minimizing eviction frequency to pure capacity reclaim events.

Explicitly schedule batch and non-critical services onto Spot infrastructure:

• Toleration Block: Added toleration in pod spec for key kubernetes.azure.com/scalesetpriority with operator Equal and value spot.
• Preferred Node Affinity: Configured preferredDuringSchedulingIgnoredDuringExecution targeting spot nodes, allowing pods to fall back to regular on-demand nodes if no Spot capacity is available in the region.

Gracefully intercept Azure's 30-second pre-eviction notice to drain pods safely:

• Termination Handler DaemonSet: Deployed azure-scheduled-events-handler querying the IMDS endpoint (http://169.254.169.254/metadata/scheduledevents).
• Cordon & Drain: Upon detecting a Preempt scheduled event, the controller immediately marks the node unschedulable (cordon) and triggers a graceful pod drain with --ignore-daemonsets --delete-emptydir-data.

Quantify compute cost reduction and monitor job failure metrics:

• Cost Impact: Cut batch processing compute cost from $42,000/month to $11,800/month (72% net savings).
• SLO Health: Batch job completion rate remained steady at 99.98% due to automated Kubernetes Job retries.