Classic design · Junior → Staff · by System Design AI

Design a Batch Processing Framework like MapReduce

Map, shuffle and reduce on one machine, a 1,000-node cluster and a multi-tenant YARN platform on EMR: splits, locality, sorted spills, reruns, backup tasks, Spot and S3 commits, with the theory behind them.