System Design AI

Design an Ad Click Aggregator: The Junior, Senior and Staff Answer

When an interviewer asks you to design an ad click aggregator, they want to see how you handle a high-throughput, low-latency pipeline that turns a firehose of click events into useful metrics. They are looking for:

  • Understanding of traffic patterns and how they drive capacity decisions.
  • Ability to break a request path into discrete hops and justify each component.
  • Awareness of data consistency, durability, and operational concerns at scale.
  • Clear communication of trade-offs and failure handling.

1. Core Requirements and Driving Numbers

RequirementTypical Target
ThroughputRoughly 100 million clicks per day (≈1 200 clicks per second)
LatencyEnd-to-end aggregation visible within a few seconds; per-click processing < 100 ms
DurabilityNo loss of click data; each click is counted exactly once
ScalabilityAbility to grow to billions of clicks per day without major redesign
QueryabilityReal-time dashboards and batch reports; reads may be up to 10× the write rate

The numbers are only ball-park, but they set the stage for choosing services that can handle the expected load.

2. Request Path – One Hop at a Time

Below is a step-by-step walk-through of a single click event from the browser to the final stored aggregate.

  1. Client → Edge – The ad click is sent via HTTPS to a CloudFront distribution. CloudFront terminates TLS and forwards the request to the nearest regional endpoint, reducing latency for the user.
  2. Edge → API Layer – An AWS API Gateway endpoint receives the request. API Gateway validates the payload (adId, campaignId, timestamp, userId) and throttles traffic if needed.
  3. API → Ingestion Service – The request is placed onto an Amazon SQS queue (or an MSK Kafka topic for higher durability). This decouples the front-end from downstream processing and smooths burst traffic.
  4. Ingestion → Processing Workers – A fleet of Amazon ECS tasks (or Lambda functions for smaller loads) poll the queue, parse the event, and enrich it (e.g., resolve campaign metadata from DynamoDB). The worker then writes a small record to an Amazon Kinesis Data Stream for real-time aggregation and also persists the raw event to S3 for audit.
  5. Real-time Aggregation – An Apache Flink application (on Amazon Managed Service for Apache Flink, the service formerly named Kinesis Data Analytics) consumes the stream, groups by adId/campaignId per minute, and writes the incremental counters to an Amazon DynamoDB table keyed by (adId, minute).
  6. Batch Aggregation – A nightly AWS Glue job reads the raw S3 logs, re-aggregates over longer windows (hourly, daily) and writes the results to Amazon Aurora for ad-hoc reporting.
  7. Read Path – Dashboards query DynamoDB (through a read-through cache in ElastiCache) for recent metrics, while deeper reports hit Aurora.

Each hop is intentional: edge caching for latency, queue for elasticity, stream for ordered processing, DynamoDB for low-latency key-value reads, and Aurora for complex analytics.

3. Data Model Sketch

  • Raw Click Record (S3): { adId, campaignId, userId, ts, metadata }
  • Realtime Counter (DynamoDB): Partition key = adId, sort key = minuteBucket. Attributes: clickCount, uniqueUserCount (using HyperLogLog stored as a binary blob).
  • Batch Summary (Aurora): Table ad_clicks_summary(ad_id, day, hour, click_cnt, unique_user_cnt).

The DynamoDB table is the core of the real-time path; it gives sub-second reads for the most recent minutes. Aurora is added only because relational queries (joins with campaign budgets, etc.) become cheaper when the data set grows beyond what DynamoDB can scan efficiently.

4. What a Complete Junior Answer Covers

ElementExpected Junior Coverage
ScopeLists functional requirements (track clicks, aggregate per ad) and non-functional goals (throughput, latency).
ArchitectureSketches a simple three-tier flow: client → load balancer → stateless app servers → a single relational DB (e.g., Aurora).
ScalingMentions adding more app servers behind an ELB and vertical scaling of the DB.
Data ModelShows a basic clicks(ad_id, ts, user_id) table.
BottlenecksIdentifies the DB write rate as a potential choke point.
Trade-offsBriefly notes that a relational store is simple but may not sustain 1 200 cps without sharding.

A junior answer stops at a monolithic design and acknowledges that more work is needed to handle the load.

5. What a Senior Answer Adds

ElementSenior Enhancements
DecouplingIntroduces a queue (SQS) and stream (Kinesis) to absorb spikes.
CachingAdds ElastiCache (Redis) as a read-through cache for recent aggregates.
ShardingDescribes partitioning DynamoDB by adId and using on-demand capacity to auto-scale.
DurabilityStores raw events in S3 for replay and compliance.
MetricsShows CloudWatch dashboards for queue depth, Lambda errors, and DynamoDB throttles.
Cost AwarenessMentions on-demand vs. provisioned capacity trade-off for DynamoDB.

The senior design demonstrates an end-to-end pipeline that can sustain the target 1 200 cps while keeping latency low and providing durability.

6. What a Staff Answer Looks Like

A staff-level answer expands on the senior design with operational depth and defensive reasoning.

6.1 Failure Modes and Mitigations

  • Queue backlog – Auto-scale the ECS worker fleet; alert on SQS depth > 10 × average.
  • DynamoDB hot partition – Use a composite key with a random suffix or enable adaptive capacity; monitor ConsumedReadCapacityUnits.
  • Stream processing lag – Deploy a checkpoint-based consumer that can replay from S3 if the Kinesis application crashes.
  • Data loss – Enable S3 versioning; use DynamoDB point-in-time recovery (PITR) for the counter table.

6.2 Operational Concerns

  • Observability – Instrument each hop with X-Ray traces; push custom metrics (clicks per ad, error rates) to CloudWatch.
  • Deployment – Use CloudFormation or CDK pipelines; blue-green deployments for the processing workers to avoid downtime.
  • Security – Enforce IAM least-privilege; encrypt data at rest (S3 SSE-KMS, DynamoDB encryption) and in transit (TLS).
  • Cost – Estimate daily cost: S3 storage for raw logs (~10 GB/day ≈ $0.023/GB), DynamoDB on-demand reads/writes (~1 200 cps × 86 400 s ≈ 100 M reads ≈ $0.25 per million reads), Kinesis shards (1 200 cps ≈ 2 shards ≈ $0.015 per shard-hour). Rough total under $100/day, well within typical budgets.

6.3 Trade-off Defense

  • Why DynamoDB over Redis for the primary store? DynamoDB provides durability and automatic scaling without manual eviction logic, which is crucial for a write-heavy workload. Redis is kept as a cache for read speed, not as the source of truth.
  • Why SQS instead of direct Lambda invocation? SQS buffers bursts and gives visibility into backlog, whereas invoking Lambda directly ties ingestion to Lambda’s concurrency limit: synchronous calls are throttled once it is reached, and asynchronous ones are retried only for a limited time before they are discarded.

7. Common Mistakes That Sink This Question

  • Ignoring write scalability – Proposing a single relational DB without sharding or write-offloading.
  • Over-engineering the pipeline – Adding unnecessary components (e.g., a full-text search engine) before the core aggregation problem is solved.
  • Missing durability guarantees – Forgetting to persist raw events or to enable replay mechanisms.
  • No latency plan – Failing to explain how aggregates become visible within a few seconds; simply stating "the system is fast".
  • Skipping operational metrics – No mention of monitoring, alerting, or capacity planning.

Avoiding these pitfalls shows that you can design a system that not only works on paper but also survives real-world traffic.

8. Comparison of the Three Levels

AspectJuniorSeniorStaff
Core flowClient → LB → app → single DBAdds queue, stream, cache layersAdds observability, replay, and failover loops
StorageAurora onlyAurora + DynamoDB + S3Same + PITR, versioning, encryption
ScalabilityVertical scaling, manual shardingAuto-scaling services (ECS, DynamoDB on-demand)Capacity planning, cost modeling, adaptive capacity
ReliabilityBasic retry logicDurable queue, checkpointed streamFull disaster recovery, data loss prevention
OpsBasic monitoringCloudWatch dashboards, alertsX-Ray tracing, CI/CD pipelines, security audits

How to practice this

  1. Study the worked design for an ad click aggregator in the System Design AI library; note how each hop is justified.
  2. On a blank canvas, sketch the request path yourself, labeling each AWS service and the data model.
  3. Use the practice brief rubric to evaluate your answer against junior, senior and staff expectations, then iterate.

Questions people ask

What is the typical throughput for an ad click aggregator?

Roughly 100 million clicks per day, which translates to about 1 200 clicks per second. Designs should handle bursts several times higher than the average.

Why use both DynamoDB and Aurora?

DynamoDB gives low-latency key-value reads for recent aggregates, while Aurora handles complex relational queries and long-term analytics that are inefficient on a NoSQL store.

Can I replace SQS with Kafka?

Yes. Kafka (via MSK) provides higher durability and ordering guarantees, but it adds operational complexity. SQS is simpler and sufficient for most click-stream workloads.

How do I ensure no click data is lost?

Persist the raw event to S3 immediately, enable versioning, and use DynamoDB point-in-time recovery. Workers can replay from S3 if downstream components fail.