Design a Recommendation System: Junior, Senior, and Staff Answers
When an interviewer asks you to design a recommendation system, they are looking for three things: your ability to break a complex pipeline into clear steps, your sense of the data-driven trade-offs (latency vs. freshness vs. cost), and how you think about operations at scale. The problem is a good litmus test because it touches storage, compute, messaging, and monitoring – all core building blocks of modern web services.
Core Requirements and Driving Numbers
Functional
- Generate a ranked list of items (e.g., videos, products) for a user on each page view.
- Support personalization based on recent activity and long-term preferences.
- Allow engineers to update the model without taking the service offline.
Non-functional
- QPS: Roughly 100 K requests per second (RPS) for a popular consumer app. That is about 8.6 billion requests per day if sustained.
- Latency: End-to-end response time ≤ 200 ms for 95 % of requests.
- Data freshness: Recent user actions should influence recommendations within a few minutes.
- Storage: User-item interaction logs on the order of 10 TB per month.
These numbers drive the choice of services: you need a fast key-value store for user profiles, a durable log for event ingestion, and a batch compute layer for model training.
Request Path – One Hop at a Time
- API Gateway receives the HTTP request. It validates the user token and forwards the call to a Lambda (or an EC2 container) that acts as the recommendation service.
- The service looks up the user’s profile in DynamoDB. The profile contains aggregated features (e.g., category scores, recent clicks) that are kept small enough to fit in a single item.
- To enrich the profile with the latest events, the service reads the most recent activity from ElastiCache (Redis), where a stream of user actions is cached for a few minutes.
- The service calls a model scoring microservice (running on ECS/Fargate) that takes the profile and a list of candidate items (pulled from a pre-computed candidate pool stored in S3) and returns a ranked list.
- The ranked list is returned to the API Gateway, which sends it back to the client.
Each hop is deliberately short: API Gateway → Lambda → DynamoDB → Redis → Scoring Service → S3. The diagram on the canvas would show arrows for each hop, with notes on why the service was chosen.
Data Model Overview
- UserProfile (DynamoDB): Partition key =
user_id. Attributes = aggregated feature vector, last update timestamp. - ItemFeatures (S3): Parquet files partitioned by category, refreshed nightly after model training.
- EventLog (MSK Kafka): Streams of raw click, view, and purchase events. Consumers write to DynamoDB (for long-term aggregates) and to Redis (for recent activity).
- CandidatePool (S3): Pre-filtered list of ~10 K items per user, generated daily by a batch job.
The key design choice is to store the real-time profile in DynamoDB and cache the freshest events in Redis, rather than hitting the log for every request. This choice beats a pure batch approach (stale recommendations) and a pure real-time approach (high read load on the log).
What a Complete Junior Answer Covers
- Lists functional and non-functional requirements.
- Draws a simple request flow: client → API → recommendation service → data store → response.
- Picks DynamoDB for user profiles and S3 for static item data.
- Mentions a cache (ElastiCache) for recent events.
- Shows a basic diagram with one hop per component.
- Touches on scalability by noting DynamoDB’s auto-scaling and S3’s unlimited storage.
What a Senior Answer Adds
- Latency budgeting: Breaks down 200 ms budget (e.g., 20 ms for API, 30 ms for DynamoDB read, 40 ms for Redis, 80 ms for scoring, 30 ms for network).
- Cache invalidation: Describes a TTL-based eviction strategy for Redis entries, refreshed every few minutes.
- Batch vs. real-time pipelines: Introduces a nightly Spark job on EMR that recomputes the candidate pool and updates S3.
- Message durability: Uses MSK Kafka to guarantee event ordering and replayability.
- Observability: Adds CloudWatch metrics for QPS, latency, and error rates, plus X-Ray tracing across hops.
- Fail-over: Shows a secondary DynamoDB table in another region with global tables for read-only fallback.
What a Staff Answer Adds
- Failure modes: Analyzes what happens if Redis goes down (fallback to DynamoDB only, with higher latency) and if the scoring service crashes (serve a cached default list from CloudFront). Provides a circuit-breaker pattern.
- Operational tooling: Describes automated canary deployments for the scoring microservice, Lambda versioning, and DynamoDB backup schedules.
- Cost estimation: Roughly calculates read/write capacity units for DynamoDB (e.g., 100 K reads/s of items up to 4 KB → ~100 K RCUs, or about half that with eventually consistent reads) and storage cost for S3 (10 TB × $0.023/GB ≈ $230 / month). Shows how Redis memory usage (e.g., 200 GB) translates to $0.15 / GB-hour.
- Trade-off defense: Argues for a hybrid model (batch candidate pool + real-time scoring) over a pure deep-learning online model because it keeps latency predictable and cost manageable.
- Capacity planning: Shows how to scale the scoring service horizontally based on request rate, using ECS task auto-scaling triggered by CloudWatch alarms.
Mistakes That Sink This Question
| Mistake | Why It Fails |
|---|---|
| Ignoring latency budget | You may propose a heavy ML model that exceeds 200 ms, showing no plan to prune or cache results. |
| Over-engineering the data pipeline | Adding unnecessary components (e.g., a separate OLAP warehouse) makes the design noisy and distracts from core concerns. |
| Forgetting data freshness | Relying only on nightly batch jobs yields stale recommendations, which interviewers flag as unrealistic. |
| No observability | Without metrics or tracing, you cannot demonstrate how you would monitor and debug the system at scale. |
How to Practice This
- Study the worked design for a recommendation system in the System Design AI library. It shows the step-by-step diagram and notes for each component.
- On the practice canvas, draw your own recommendation flow. Start with the API layer and add each hop, labeling the AWS service you would use.
- Use the rubric to check that you covered requirements, latency budgeting, failure modes, and cost trade-offs. Iterate until the senior-level checklist is satisfied.
Questions people ask
What are the key scalability concerns for a recommendation system?
You need to handle high QPS, keep latency under 200 ms, and store billions of interaction events. Use DynamoDB for fast key-value reads, S3 for cheap bulk storage, and a streaming platform like MSK to ingest events without bottlenecks.
Why combine batch candidate generation with real-time scoring?
Batch jobs can compute expensive similarity scores offline, reducing the candidate set to a manageable size. Real-time scoring then personalizes the final ranking using the freshest user activity, keeping latency low while preserving relevance.
How do you ensure recommendations stay fresh without overwhelming the system?
Cache recent user actions in ElastiCache with a short TTL (a few minutes). Update the user profile in DynamoDB periodically, and fall back to the DynamoDB profile alone when the cache is unavailable.
What monitoring should be in place for a production recommendation service?
Track request latency, error rates, and cache hit ratios in CloudWatch. Use X-Ray to trace request paths across services, and set alarms to trigger auto-scaling or circuit-breaker actions when thresholds are breached.