Q: A 64-node distributed training cluster writing checkpoints and loading dataset shards over Amazon FSx for Lustre experiences I/O hangs and Lustre client timeouts. How do you tune Lustre directory striping, metadata caching, and S3 Data Repository Associations (DRA) for 100GB/s sustained throughput?

Lustre partitions data across Object Storage Targets (OSTs). By default, a file is written to only 1 OST (`stripe_count = 1`). For large training checkpoint files or multi-gigabyte dataset archives (e.g. 50GB tar files), set directory striping to stripe across all available OSTs (`-c -1` or `-c 16`) with a 4MB or 8MB stripe size (`-S 4M`). This distributes parallel I/O across the entire storage cluster simultaneously.

# Stripe checkpoint directory across all available OSTs
lfs setstripe -c -1 -S 4M /mnt/fsx/checkpoints
# Verify striping layout on file
lfs getstripe /mnt/fsx/checkpoints/model-step-1000.pt

Tune client mount options on GPU worker nodes: increase Lustre maximum RPC in flight (`max_rpcs_in_flight = 32`), increase dirty cache memory limits (`max_dirty_mb = 2048`), and enable metadata caching (`statahead = 1`) to accelerate directory scans and avoid saturating Metadata Targets (MDTs).

# Lustre client performance tuning via sysctl / lctl
sudo lctl set_param osc.*.max_rpcs_in_flight=32
sudo lctl set_param osc.*.max_dirty_mb=2048
sudo lctl set_param llite.*.statahead_max=32

Configure FSx for Lustre Data Repository Association (DRA) linked to Amazon S3. Pre-load dataset files into Lustre cache before jobs start (`lfs hsm_restore` or FSx Data Repository Task) to ensure GPUs read directly from high-speed NVMe/SSD storage rather than triggering lazy-loading read stalls during the first epoch.

# Trigger FSx data import task from S3 via AWS CLI
aws fsx create-data-repository-task \
  --file-system-id fs-0123456789abcdef0 \
  --type IMPORT_METADATA_FROM_REPOSITORY \
  --paths "s3://my-training-data/"