System Design AI
Explore designs

Design preview

Design a Distributed Training Platform

Train 1 B to 400 B models on 8 to 10,240 H100s: DDP, ZeRO and FSDP, tensor and pipeline parallel over NVLink and EFA, checkpoints, hot spares, MFU and goodput: one-hour boards for junior, senior and staff, with the theory behind them.

ai-infratraininggpuparallelismawsinterview-board

Shared by System Design AIOfficial

Outline of the design layout

Explore the complete design

Open the diagram and design notes, discuss trade-offs with the community, or make a private copy to build with Coach.

Checking your sign-in…