System Design AI
All designs

Classic design · Junior → Staff · by System Design AI

Design a Distributed Training Platform

Train 1 B to 400 B models on 8 to 10,240 H100s: DDP, ZeRO and FSDP, tensor and pipeline parallel over NVLink and EFA, checkpoints, hot spares, MFU and goodput: one-hour boards for junior, senior and staff, with the theory behind them.

ai-infratraininggpuparallelismawsinterview-board
Outline of the design layout

Checking your sign-in…

  • A walkthrough with a diagram that grows level by level
  • A one-hour interview board for each level it covers
  • Notes behind every component: what to say, and what goes wrong