Q: Standard Cluster Autoscaler takes 12-15 minutes to provision a p4de.24xlarge node and pull a 140GB model image. How do you design Karpenter NodePools with heterogeneous instance types, pre-warmed AMIs, and automated NVMe instance store RAID0 mounting to slash scale-up time to under 2 minutes?

Configure Karpenter NodePools to allow flexible instance selection across compatible GPU families (e.g., g5.12xlarge, g5.24xlarge, g6e.12xlarge) across multiple availability zones. Configure consolidation policies and spot fallback rules to maximize instance acquisition probability during capacity shortages.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-inference
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand", "spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["g5.12xlarge", "g5.24xlarge", "g6e.12xlarge"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu-nodeclass

GPU instances include high-speed ephemeral NVMe disks (e.g., 4x 900GB NVMe). In EC2NodeClass userdata, run a bash bootstrap script that discovers all local NVMe devices, creates an mdadm RAID0 array, formats it with ext4 or XFS with noatime, and mounts it to /mnt/fast-scratch and /var/lib/containerd for blistering 10GB/s I/O throughput.

# EC2NodeClass userData snippet
#!/bin/bash
DEVICES=$(lsblk -d -n -o NAME,TYPE | grep -E '^nvme[1-9]' | awk '{print "/dev/"$1}')
if [ -n "$DEVICES" ]; then
  mdadm --create --verbose /dev/md0 --level=0 --raid-devices=$(echo $DEVICES | wc -w) $DEVICES
  mkfs.ext4 -F /dev/md0
  mkdir -p /mnt/models
  mount -o noatime /dev/md0 /mnt/models
fi

Pre-bake the NVIDIA driver and base PyTorch/vLLM runtime into custom Packer AMIs. Mount the /mnt/models host directory into inference pods using hostPath or local PVs. Inference containers mount the local RAID0 array where model checkpoints are cached and shared across pod replicas, eliminating repeated Hugging Face or S3 downloads.

volumes:
- name: model-cache
  hostPath:
    path: /mnt/models
    type: DirectoryOrCreate