102. Design a GPU Cluster Scheduler for Training Jobs
Thousands of GPUs shared by many teams: gang admission, topology-aware placement, quotas with lending, checkpointed preemption and failure recovery.
ClassicMedium
Designs
Pick a system. Work through the problem. Compare your approach.
Classic designs 21
Ordered by design numberCompany tags are community-reported. Counts on cards show how many people reported that design.
AI infrastructureClear filters
103. Design Model Weight Distribution to a GPU Fleet
A 500 GB model on 1,000 GPU servers in minutes: chunks and hashes, a topology-aware swarm, signed manifests, bandwidth budgets and waves.
ClassicMedium104. Design a Feature Store
Define a feature once, train on point-in-time correct history and serve 200 fresh values in under 10 ms.
ClassicMedium105. Design a Distributed Training Platform
Train 1 B to 400 B models on 8 to 10,240 H100s: DDP, ZeRO and FSDP, tensor and pipeline parallel over NVLink and EFA, checkpoints, hot spares, MFU and goodput.
ClassicMedium107. Design an ML Model Serving and Deployment Platform
From registry to traffic for hundreds of models: gates, shadow and canary with automatic rollback, packed CPU and GPU pools, LLM serving, cells and audit.
ClassicMedium108. Design an LLM Inference Serving Platform
Open-weight LLMs from 8B to 405B served to many products over a streaming OpenAI-style API on AWS GPUs.
ClassicMedium109. Design an Embedding Generation and Indexing Pipeline
Embed a billion chunks on GPUs, keep a k-NN index fresh through CDC with versioned writes and provable deletes, and migrate models blue-green.
ClassicMedium111. Design Checkpointing for a 10,000-GPU Training Run
Save the 1 TB state of a 70 B model on 10,000 GPUs often enough that a failure costs minutes, without stalling training, and restore fast.
ClassicMedium112. Design ML Model Monitoring and Drift Detection
Know within hours when hundreds of production models see broken inputs, drift or falling quality, before the labels arrive.
ClassicMedium113. Design a Data Labeling Platform
Route images, text, audio and model replies to thousands of annotators, buy quality with gold and consensus, cut cost with models, and ship versioned datasets.
ClassicMedium114. Design a Web-Scale Training Data Pipeline
Petabytes of crawled pages into trillions of clean, deduplicated, tokenized and versioned training tokens, with opt-outs honoured.
ClassicMedium115. Design an LLM Batch Inference Service
Millions of LLM prompts in JSONL files, one result per custom_id within 24 hours at about half the online price, on GPUs that come and go.
ClassicMedium116. Design an LLM Pretraining Pipeline
Pretraining a base LLM on a fixed budget: sizing by 6ND and scaling laws, the data factory, tokenizer, stable runs, failures and evals.
ClassicMedium117. Design a Self-Healing GPU Fleet
Detect, classify and repair failing GPU nodes (Xid errors, ECC, NVLink and EFA faults, stragglers, silent corruption) without draining the fleet.
ClassicMedium118. Design an LLM Gateway
One API in front of every model: token quotas, routing and fallbacks, caching, cost attribution, masked logs and guardrails, streamed without buffering.
ClassicMedium121. Design an LLM Evaluation Platform
GPU batch generation behind a shared cache, sandboxed pass@k, calibrated judges, paired statistics, release gates, contamination checks.
ClassicMedium122. Design an Experiment Tracking and Model Registry Service
Experiment tracking and a model registry: non-blocking logging, chunked metric curves, content-addressed artifacts, lineage and a promotion gate. One-hour boards for junior, senior and staff, with the theory behind them.
ClassicMedium123. Design a Hyperparameter Tuning Service
A tuning service like Vizier: random and Bayesian search, ASHA and population-based training, trials on a shared GPU cluster.
ClassicMedium124. Design Multi-Tenant LoRA Adapter Serving
Serve 10,000 LoRA adapters for 2,000 tenants on one shared base model, with mixed-adapter batches, tiered adapter caches, affinity routing and base upgrades.
ClassicMedium125. Design a Prompt and Semantic Cache for LLMs
Exact, semantic and prefix caching in front of LLMs: scoped keys, distances with an error budget, tenant isolation, invalidation and measured savings.
ClassicMedium129. Design an Image-Text Training Data Pipeline
Billions of image-text pairs from Common Crawl: polite fetching, CLIP scoring, dedup, recaptioning, WebDataset shards and takedowns.
ClassicMedium