System Design AI

Design a Web Crawler: Junior, Senior, and Staff Answers

When you’re asked to design a web crawler, the interviewer is looking for three things: how you think about scale, how you model the data, and how you keep the system reliable. They want to see that you can turn a simple request-response loop into a pipeline that can run continuously at millions of pages per day.

What the interviewer is testing

  • Scalability mindset – can you estimate load and break the problem into independent pieces?
  • Data modeling – do you know what metadata you need to store and how you’ll query it?
  • Failure handling – how will you detect and recover from network errors, throttling, or a node crash?
  • Trade-off awareness – are you comfortable discussing latency vs. freshness, consistency vs. throughput?

Core requirements and driving numbers

RequirementTypical ballpark (hedged)
Pages crawled per secondRoughly 100 pages/s (≈8.6 M pages/day)
Storage for raw HTML per dayAbout 1 TB (assuming roughly 100 KB per page)
Crawl depth3-5 hops from seed URLs

These numbers are not exact; they give you a sense of the magnitude you need to handle. From them you can derive downstream capacities – e.g., a queue that must sustain 100 req/s, or a storage layer that writes ~1 TB per day.

Request path, one hop at a time

  1. Seed manager – stores the initial set of URLs and periodically injects them into the system.
  2. Front-end API – a lightweight HTTP endpoint (e.g., API Gateway + Lambda) that receives URLs from the seed manager or from external triggers.
  3. URL queue – an SQS queue holds URLs awaiting fetch. It decouples the front-end from the fetch workers and provides at-least-once delivery.
  4. Fetcher workers – EC2 instances or Fargate tasks that pull a URL, respect robots.txt, fetch the page, and push the result to a downstream stream.
  5. Stream processor – Kafka on MSK receives raw HTML. A consumer extracts links, normalizes them, and writes new URLs back to the queue.
  6. Metadata store – DynamoDB holds URL status (fetched, pending, error), timestamps, and fingerprint hashes for deduplication.
  7. Blob storage – S3 stores the raw HTML and any extracted assets. CloudFront can serve the stored pages for debugging.
  8. Scheduler / politeness controller – a Redis (ElastiCache) cache tracks per-host request rates to avoid overloading a single site.

Data model snapshot

  • URL table (DynamoDB): url (PK), status, lastFetched, etag, fingerprint.
  • Page object (S3): key = pages/{hash}.html, metadata = {url, fetchTime, size}.
  • Link table (optional, Aurora): stores adjacency for graph-analysis features.

Key choice: SQS vs. Kafka for the URL queue

We pick SQS for the initial queue because it gives simple scaling, built-in dead-letter handling, and no need to manage partitions. Kafka is introduced later as the high-throughput stream for raw pages, where ordering and replay are valuable. A junior answer might keep everything in SQS; a senior answer adds Kafka for the bulk data path.

What a complete junior answer covers

  • High-level diagram of the flow above.
  • Rough estimates for request rate and storage.
  • Mention of robots.txt and basic politeness.
  • Simple URL queue (SQS) and storage (S3 + DynamoDB).
  • No deep discussion of sharding, consistency, or operational metrics.

What a senior answer adds

  • Sharding strategy – partition the URL queue by host hash to avoid hot spots.
  • Deduplication – store a fingerprint (e.g., SHA-256 of content) in DynamoDB and skip re-fetches.
  • Back-pressure handling – use SQS visibility timeout and Lambda concurrency limits to throttle fetchers.
  • Metrics & monitoring – CloudWatch dashboards for queue depth, fetch latency, error rates.
  • Consistency trade-off – explain why eventual consistency in DynamoDB is acceptable for crawl status.

What a staff answer adds

  • Failure modes – network timeouts, DNS failures, storage throttling. Show how dead-letter queues, exponential back-off, and automatic retries mitigate them.
  • Operational concerns – rolling deployments of fetcher containers, canary testing of new parsing logic, and alerting on crawl-rate anomalies.
  • Cost estimation – rough monthly cost: S3 (~$23 for 1 TB), DynamoDB reads/writes (tens of dollars), SQS (few dollars), EC2/Fargate (depends on instance type, roughly $0.10 per hour per worker). Emphasize that cost scales linearly with pages per second.
  • Trade-off defense – argue for higher politeness latency to stay within target sites’ rate limits, even if it reduces overall throughput. Discuss choosing DynamoDB over Aurora for its low-latency key-value access and auto-scaling.
  • Scalability roadmap – move from a single fetcher fleet to a region-wide deployment, add a CDN edge cache for static assets, and consider using AWS Glue for periodic analytics.

Mistakes that sink this question

MistakeWhy it hurts
Ignoring robots.txt and politenessShows lack of real-world awareness; interviewers expect you to respect crawl etiquette.
Under-estimating storageLeads to unrealistic designs; 1 TB per day is easy to miss if you assume 1 KB per page.
No deduplication strategyCrawlers quickly waste bandwidth re-fetching identical content.
Using a single queue without shardingCreates hot-spot on popular domains and limits scalability.
Forgetting monitoringWithout metrics you cannot tell if the crawler is stuck or throttled.

How to practice this

  1. Study the worked design in the System Design AI library. It walks through the same components with notes on what to say at each step.
  2. Draw your own canvas: start with the seed manager, add the API, queue, fetchers, stream, and storage. Label each node with the AWS service you would use.
  3. Run a mock interview: explain the flow, answer follow-up questions on scaling, and get feedback from the AI coach on missing pieces.

Questions people ask

What is the typical throughput for a production web crawler?

A large-scale crawler often handles on the order of 100 pages per second, which translates to roughly 8-9 million pages per day. The exact number depends on the target domain and politeness constraints.

Why use DynamoDB for URL status instead of a relational database?

DynamoDB offers low-latency key-value access and auto-scaling, matching the high read/write rate of URL status updates. A relational store would add unnecessary complexity for this pattern.

How do you ensure you don’t overload a single website?

Store per-host request timestamps in ElastiCache (Redis) and enforce a minimum interval (e.g., 1 second) before issuing another request to the same host. This implements the politeness policy.

Can the crawler be built without any AWS services?

Yes, the architecture is portable, but AWS services like SQS, Kafka on MSK, DynamoDB, and S3 provide managed scaling and reduce operational overhead, letting you focus on the design rather than infrastructure.