Design a Distributed Cache: The Junior, Senior and Staff Answer
When an interviewer asks you to design a distributed cache, they are probing three things: how you think about scale, how you reason about consistency and latency, and whether you can translate high-level requirements into concrete component choices. The question is open-ended, so you must drive the conversation with a few numbers that set the scope, then walk through the request path one hop at a time.
What the Interviewer Is Testing
- Scalability – Can you handle millions of requests per second and grow horizontally?
- Consistency & Staleness – Do you understand read-through vs write-through vs write-behind patterns?
- Operational Insight – Do you consider monitoring, failure recovery, and cost?
- Communication – Can you explain each piece clearly and justify trade-offs?
Core Requirements and Driving Numbers
Typical distributed-cache interview constraints look like this:
- Throughput: Roughly 100 million reads per day → about 1,200 reads per second.
- Latency: Sub-millisecond read latency for hot keys; <10 ms for cache miss fallback.
- Data Size: 10 GB of hot objects, each up to 1 MB, fitting comfortably in memory.
- Durability: Cache miss should fall back to a persistent store (e.g., DynamoDB) without data loss.
These ball-park figures let you decide how many cache nodes you need, what replication factor to use, and whether you need a write-ahead log.
Request Path – One Hop at a Time
- Client → Edge: The client hits a CloudFront edge location. CloudFront can serve cached static blobs directly, reducing load on the cache layer.
- Edge → ElastiCache (Redis) Node: If the key is hot, the request is routed to the nearest Redis node in an Amazon ElastiCache cluster. The node performs a
GETand returns the value. - Cache Miss → DynamoDB: On a miss, the application server queries DynamoDB for the authoritative value, writes it back to Redis (
SET), and returns the response. - Write Path: For updates, the app writes to DynamoDB first and then publishes an invalidation message to an SQS queue (cache-aside with asynchronous invalidation; write-through would update the cache in the same synchronous step as the store). A background worker consumes the message and evicts or updates the corresponding Redis entry.
Data Model & Key Choice
- Key: Use a composite of
entity_type:id(e.g.,user:12345). It is readable and namespaced; the cache still hashes the whole key to choose a shard, so load spreads evenly instead of piling everyuserkey onto one shard. - Value: Store a JSON blob or binary payload; keep it under the Redis max-string size (512 MB) but aim for <1 MB for latency.
- Why Not Plain Hashing? Plain hashing spreads load evenly but loses the ability to co-locate related keys for batch invalidation. The composite key lets you target groups of keys (all
userobjects) with a single SQS message.
What a Complete Junior Answer Covers
| Aspect | Junior | Senior | Staff |
|---|---|---|---|
| Scope | Lists basic requirements (latency, throughput). | Refines requirements with numbers, adds durability. | Adds cost, ops, and risk analysis. |
| Architecture | Single-region ElastiCache cluster, fallback to DynamoDB. | Multi-AZ replication, sharding, read-through/write-through patterns. | Full mesh of edge, cache, persistent store, async invalidation, monitoring. |
| Key Design | Simple key, mentions hash. | Composite key, explains grouping. | Discusses hash-based partitioning, consistent hashing, and hot-spot mitigation. |
| Failure Modes | Mentions node crash. | Adds network partition, cache stampede, and fallback logic. | Covers cascading failures, data loss, cost of over-provisioning, and operational alerts. |
| Operations | Basic health checks. | Metrics (hit ratio, latency) and scaling policy. | Detailed SLO/SLA, cost modeling, capacity planning, and incident response runbooks. |
| Trade-offs | Chooses Redis for speed. | Defends write-through vs write-behind. | Defends replication factor vs cost, and when to use a CDN layer. |
Junior Answer Checklist
- State read latency target (<1 ms) and throughput (~1,200 ops/s).
- Sketch a single-region ElastiCache cluster with DynamoDB fallback.
- Explain a simple
GET/SETflow. - Mention cache miss handling and basic health checks.
What a Senior Answer Adds
- Sharding: Use Redis Cluster with 12 shards across three AZs. Each shard holds a subset of the key space, reducing contention.
- Replication: Primary-replica setup for high availability; failover handled by ElastiCache automatic failover.
- Cache Stampede Mitigation: Add a short-lived lock (e.g.,
SETNX) to prevent many requests from hammering DynamoDB on a miss. - Write-Behind Option: Discuss pros/cons of buffering writes before persisting, useful when write bursts are high.
- Metrics: Hit ratio, eviction rate, and latency broken out by region.
- Scaling Policy: Auto-scale the number of nodes based on CPU and memory thresholds.
- Security: VPC isolation, IAM role for DynamoDB access, and encryption-at-rest.
What a Staff Answer Adds
- Failure Mode Deep Dive:
- Network Partition: Use DynamoDB’s global tables to keep a read replica in another region; cache nodes fall back to the remote replica.
- Cache Stampede: Implement “request coalescing” with a small TTL and probabilistic early expiration.
- Cold Start: Warm the cache on deployment using a background job that pre-loads the most popular keys.
- Operations & Cost:
- Estimate node count: 6 Redis nodes (2 per AZ) each with 32 GB memory → roughly 192 GB total, enough for 10 GB hot data with headroom.
- DynamoDB read capacity: 1,200 reads/s → on-demand mode or provisioned 1,500 RCUs.
- Monthly cost: Rough ball-park of a few thousand dollars; compare to a pure DynamoDB solution.
- Trade-off Defense:
- Latency vs Cost: Adding more cache nodes reduces latency but raises cost; a 99.9% hit ratio often justifies the expense.
- Consistency: Choose write-through when the cache must stay in step with the store; write-behind if write latency is critical.
- Observability: CloudWatch dashboards for cache hit ratio, eviction count, and replication lag. Set alarms on >5% miss rate.
- Automation: Use CloudFormation or CDK to provision the whole stack, ensuring reproducibility.
Mistakes That Sink This Question
- Skipping Numbers: Interviewers love concrete numbers. Vague “high throughput” invites follow-up that can trip you.
- Ignoring Cache Invalidation: Forgetting how stale data is evicted is a classic failure.
- Over-Engineering: Pulling in Kafka or complex multi-region replication when the problem only needs a single region.
- No Ops View: Failing to discuss monitoring, scaling, and cost makes the design feel academic.
- Choosing the Wrong Store: Using S3 as a cache layer without a fast lookup layer defeats the purpose.
How to Practice This
- Study the worked design in the System Design AI library. See how a senior-level answer is broken down step by step.
- Draw your own canvas on the practice board. Start with the request path, then add sharding, replication, and failure handling layers.
- Run through the rubric: check that you covered requirements, numbers, key design, failure modes, and trade-offs. Revise until each section is crisp and defensible.
Questions people ask
What is the typical size of a Redis node for a distributed cache?
A Redis node often runs with 32 GB of memory, giving enough headroom for hot data and eviction buffers. The exact size depends on your hit-ratio target and data footprint.
When should I use write-behind instead of write-through?
Write-behind buffers writes to reduce latency, useful when write bursts are high and occasional stale reads are acceptable. Write-through guarantees immediate consistency but adds write latency.
How do I avoid a cache stampede on a miss?
Use a short-lived lock (SETNX) to let only one request populate the cache, while others wait or fallback to the backing store. Probabilistic early expiration can also smooth traffic spikes.
Do I need a CDN like CloudFront for a distributed cache?
CloudFront helps serve static objects directly at the edge, reducing load on the cache layer. For dynamic data that changes frequently, a CDN complements but does not replace the cache.