Q: How does Mixture-of-Experts (MoE) model serving differ from dense LLM serving? What is the All-to-All communication bottleneck during expert routing, and how do you architect GPU clusters to handle unbalanced token dispatching?

In dense models, Tensor Parallelism shards weight matrices across GPUs. In MoE models, Expert Parallelism (EP) places different expert sub-networks on different GPUs (e.g. Expert 0-3 on GPU 0, Expert 4-7 on GPU 1). When the gating network scores tokens, it assigns token $T_1$ to Expert 1 and token $T_2$ to Expert 5. The cluster must execute an All-to-All collective to dispatch tokens to the GPUs hosting the selected experts, and another All-to-All to gather output activations back.

# MoE Token Dispatch Collective:
# Input: tokens on Rank 0-N
# Step 1: All-to-All dispatch -> send tokens to GPUs owning top-k experts
# Step 2: Compute expert MLP locally
# Step 3: All-to-All combine -> return computed activations to original ranks

Unlike dense models where workload is evenly divided, MoE gating routers often develop 'popular experts' that receive 5x more tokens than others. The GPU hosting the hot expert becomes a severe bottleneck, stalling all other ranks. Implement capacity factor clamping (`expert_capacity_factor = 1.25`) with token dropping or auxiliary loss balancing during serving, and deploy replicated hot experts across multiple GPUs.

# vLLM MoE launch parameters for DeepSeek / Mixtral
python3 -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mixtral-8x7B-Instruct-v0.1 \
  --tensor-parallel-size 4 \
  --pipeline-parallel-size 1

Because All-to-All collectives transmit different data between every pair of GPUs simultaneously, network bisection bandwidth is the primary performance limiter. If Expert Parallelism spans across nodes without high-bandwidth InfiniBand or RoCE v2 fabrics, All-to-All latency dwarfs computation time. Confine EP within high-speed NVLink domains or ensure full non-blocking leaf-spine network topologies with zero oversubscription.

# Check All-to-All performance via NCCL tests
./build/alltoall_perf -b 8M -e 256M -f 2 -g 8