Classic design · Junior → Staff · by System Design AI

Design Disaggregated Prefill and Decode Serving

Llama 3.1 70B with prefill and decode on separate GPUs and the KV cache handed over NVLink or EFA, two pools sized from the token mix: one-hour boards for junior, senior and staff, with the theory behind them.