Q: What is the resource deadlock problem in multi-tenant distributed AI training on standard Kubernetes, and how do gang-scheduling engines like Volcano or Kueue enforce 'all-or-nothing' pod scheduling for PyTorchJob resources?

When multiple multi-node PyTorch distributed training jobs are submitted concurrently to a shared cluster, kube-scheduler schedules pods one by one. If Job 1 (requesting 8 nodes) gets 4 nodes, and Job 2 (requesting 8 nodes) gets the other 4 nodes, both PyTorch jobs block in NCCL init waiting for their remaining peers, starving the cluster and deadlocking all progress.

# Deadlock indicator: Multiple PyTorchJobs in 'Running' phase, but all worker pods stuck in NCCL init timeout

Deploy Volcano as a custom Kubernetes scheduler. The Kubeflow Training Operator creates a Volcano `PodGroup` CRD specifying `minMember: N` matching the exact number of distributed workers. Volcano enforces all-or-nothing scheduling: pods for the job are held in pending queue until the full quota of GPU nodes is available simultaneously, binding all pods in a single transactional scheduling cycle.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pytorch-llama-train
spec:
  minMember: 32
  queue: production-ai
  priorityClassName: high-priority

Adopt Kueue (Kubernetes-native job queuing system developed by SIG-Scheduling). Kueue operates as an admission controller and queue manager above kube-scheduler. It inspects PyTorchJob custom resources, manages ClusterQueues and LocalQueues with fair sharing and preemption, and releases jobs to the cluster only when the full cohort of GPU resources is free.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-cluster-queue
spec:
  resourceGroups:
  - coveredResources: ["nvidia.com/gpu"]
    flavors:
    - name: h100-gpu
      resources:
      - name: nvidia.com/gpu
        nominalQuota: 256