Guided practice

Company tags are community-reported. Counts on cards show how many people reported that design.

• 

  46. Design a Web Crawler

  Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.

  ClassicMedium

  QueuesObject storageAWS

  +5

  CrawlingDeduplicationPolitenessAmazon S3Amazon SQS

  Worked design

  Practice
• 

  114. Design a Web-Scale Training Data Pipeline

  Petabytes of crawled pages into trillions of clean, deduplicated, tokenized and versioned training tokens, with opt-outs honoured.

  ClassicMedium

  Generative AIAI infrastructureAWS

  +3

  Data pipelineDeduplicationLLMs

  Worked design

  Practice

Self-paced by default. Your answers and canvas stay private. AI feedback runs only when you ask for it.