46. Design a Web Crawler
Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.
ClassicMedium
Start with a template. Work through each step. Ask Coach when you need a second opinion.
Company tags are community-reported. Counts on cards show how many people reported that design.
Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.
Every committed change in Aurora and DynamoDB, read from the log and delivered in order per row to search, caches, Redshift and an S3 lake.