All designs
Classic design · Junior → Staff · by System Design AI
Design a Web Crawler
Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.
crawlingsqss3politenessdeduplicationawsinterview-board
Checking your sign-in…
- A walkthrough with a diagram that grows level by level
- A one-hour interview board for each level it covers
- Notes behind every component: what to say, and what goes wrong