Designs

Company tags are community-reported. Counts on cards show how many people reported that design.

• 

  46. Design a Web Crawler

  Ten billion pages in five days, politely, without losing progress: fetchers and parsers, domain locks, bandwidth math, deduplication and a crawl that stays fresh. One-hour boards for junior, senior and staff.

  ClassicMedium

  QueuesObject storageAWS

  +5

  CrawlingDeduplicationPolitenessAmazon S3Amazon SQS

  Worked design

  Practice
• 

  85. Design a Change Data Capture Pipeline

  Every committed change in Aurora and DynamoDB, read from the log and delivered in order per row to search, caches, Redshift and an S3 lake.

  ClassicMedium

  QueuesDatabasesStreaming

  +8

  TransactionsDistributed systemsAWSChange data captureDebeziumKafkaOutboxPostgreSQL

  Worked design

  Practice

Each design brings together the brief, practice, examples and discussion.