Design a Real-time Analytics Dashboard: The Junior, Senior and Staff Answer
When you hear real-time analytics dashboard, the interviewer is looking for three things: how you break a continuous stream into a queryable store, how you keep the UI fresh without overloading the backend, and how you think about reliability and cost at scale. The problem is a microcosm of any high-throughput, low-latency system – think ad-tech, IoT telemetry, or monitoring dashboards.
Core Requirements
| Requirement | Why it matters |
|---|---|
| Ingestion rate – roughly 10 k events per second (≈864 M per day) | Drives the choice of message broker and write path. |
| Query latency – sub-second end-to-end for a user-initiated chart refresh | Determines cache strategy and read path. |
| Concurrent viewers – about 1 k active dashboards at peak | Sets the read throughput needed from the store and CDN. |
| Data retention – 30 days of raw events, aggregated metrics for 90 days | Affects storage tiering and compaction strategy. |
| Durability – no loss of events, but occasional stale data is acceptable | Guides replication and replay mechanisms. |
These numbers are ball-park. You can adjust them during the interview to match the interviewer's scenario.
Request Path – One Hop at a Time
- Client → API Gateway – The browser sends a REST or WebSocket request for a chart. API Gateway validates the token and forwards to a Lambda function.
- Lambda → Kafka (MSK) – For write-heavy paths (e.g., an event source like clickstream), the Lambda publishes the event to an MSK topic. Kafka provides ordered, durable logs and can handle the 10 k msg/s load.
- Kafka → Stream Processor (Amazon Managed Service for Apache Flink, or Flink on EMR) – A stream job consumes the topic, performs windowed aggregations (e.g., per minute, per hour), and writes two outputs:
- Raw events → S3 (partitioned by date) for long-term storage.
- Aggregated metrics → DynamoDB (or Aurora) for low-latency reads.
- Cache Layer – ElastiCache (Redis) – The Lambda that serves UI queries first checks Redis for the latest pre-computed bucket. If a miss, it falls back to DynamoDB.
- Response → CloudFront – The final JSON payload is cached at the edge for a few seconds, reducing repeat reads from the origin.
Data Model
- Raw events – JSON blobs stored in S3, partitioned
year/month/day/hour. This is cheap, immutable, and easy to replay. - Aggregated table – DynamoDB with a composite key
(metric_id, bucket_timestamp). Each item holds count, sum, min, max, and a pre-computed percentile. - Cache entry – Redis key
metric:{id}:{bucket}with TTL of 5 seconds. This gives the sub-second UI experience.
What a Junior Answer Covers
- High-level pipeline – Mention API Gateway, Lambda, Kafka, DynamoDB, and a front-end chart library.
- Basic sizing – Roughly 10 k writes/sec → a small three-broker MSK cluster; DynamoDB read capacity of a few thousand RCUs.
- Simple data flow – One write path, one read path, no fan-out.
- Missing pieces – No cache, no CDN, no discussion of partition keys, and no operational concerns.
What a Senior Answer Adds
- Fan-out and buffering – Introduce SQS between Lambda and Kafka to smooth spikes.
- Caching – Explain why Redis is needed for sub-second latency and how TTL aligns with the refresh interval.
- Read scaling – Use DynamoDB on-demand or provisioned with auto-scaling, and add a read replica if using Aurora for complex queries.
- Metrics granularity – Discuss window sizes (e.g., 1 min, 5 min) and how they affect storage and query cost.
- Operational monitoring – CloudWatch alarms on lag, Lambda error rates, and Kafka consumer lag.
What a Staff Answer Includes
- Failure modes – Detail how to handle Kafka broker loss (replication factor = 3), Lambda throttling (burst concurrency limits), and Redis node failure (cluster mode with automatic failover).
- Cost analysis – Rough estimate: MSK ~ $0.10 per broker-hour, DynamoDB on-demand ~ $0.625 per million writes, Redis ~ $0.15 per GB-hour. Show trade-offs between provisioned vs on-demand.
- Operational runbook – Steps for scaling the Kafka partition count, rebalancing consumers, and purging old S3 data with lifecycle policies.
- Defended trade-off – Why you chose DynamoDB over Aurora for point reads: predictable latency, automatic scaling, and lower operational overhead despite higher per-read cost.
- Security – IAM role separation, VPC endpoints for MSK and DynamoDB, and encryption at rest.
Common Mistakes That Sink the Question
- Skipping the ingestion layer – Jumping straight to DynamoDB without a durable log leads to data loss under spikes.
- Over-engineering – Proposing a full data lake with Spark jobs for a dashboard that needs only minutes-level aggregates adds unnecessary latency.
- Ignoring cache invalidation – Forgetting TTL or stale-read scenarios makes the UI appear laggy.
- No scaling plan – Not addressing how to increase Kafka partitions or DynamoDB throughput as traffic grows.
- Hard-coding numbers – Using exact figures without hedging (“exactly 10,000 writes/sec”) makes the design look brittle.
Comparison of Expected Answers
| Level | Core Components | Added Detail | Depth of Ops & Cost |
|---|---|---|---|
| Junior | API Gateway, Lambda, small Kafka cluster, DynamoDB (basic) | Simple data flow, no cache, no CDN | Minimal – only mention of provisioning basics |
| Senior | Adds SQS buffer, Redis cache, CloudFront CDN, auto-scaling policies | Windowed aggregation, latency targets, monitoring metrics | Discusses scaling limits, basic cost drivers |
| Staff | Full fault-tolerant topology, replication factors, runbooks, security zones | Failure modes, cost breakdown, trade-off justification, operational playbook | Deep dive into cost, ops, and why each AWS service was chosen |
How to Practice This
- Study a related worked design in the System Design AI library, such as one built on streaming aggregation. Observe how the diagram grows from the junior answer to the staff answer, and click each component for its notes.
- Draw your own canvas on the practice board. Start with the ingestion layer, then add storage, cache, and CDN. Label each hop and write a short note on why you chose it.
- Run a mock interview with a peer, or ask the AI coach to review your canvas. Explain the junior, senior, and staff versions, then look for missing failure modes or cost considerations.
Questions people ask
What is the typical ingestion rate for a real-time dashboard?
Roughly 10 k events per second, which translates to about 864 million events per day. The exact number depends on the product, but this ball-park drives the need for a durable, partitioned log like Kafka.
Why use DynamoDB instead of Aurora for metric storage?
DynamoDB offers single-digit millisecond latency, automatic scaling, and a simpler operational model for point reads. Aurora can handle complex joins but adds replica lag and requires more DBA effort.
How does Redis improve UI latency?
By caching the most recent aggregated bucket, Redis serves reads in microseconds. The UI can poll every few seconds and hit the edge cache, keeping the end-to-end latency well under a second.
What should I watch for in production?
Monitor Kafka consumer lag, Lambda error rates, Redis eviction metrics, and DynamoDB throttling. Set alarms on any metric that exceeds a small percentage of the provisioned capacity.