System Design AI
Explore designs

Design preview

Design a Web Crawler

Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.

crawlingsqss3politenessdeduplicationawsinterview-board

Shared by System Design AIOfficial

Outline of the design layout

Explore the complete design

Open the diagram and design notes, discuss trade-offs with the community, or make a private copy to build with Coach.

Checking your sign-in…