System Design AI

Design a Distributed Cache: The Junior, Senior and Staff Answer

When an interviewer asks you to design a distributed cache, they are probing three things: how you think about scale, how you reason about consistency and latency, and whether you can translate high-level requirements into concrete component choices. The question is open-ended, so you must drive the conversation with a few numbers that set the scope, then walk through the request path one hop at a time.

What the Interviewer Is Testing

  • Scalability – Can you handle millions of requests per second and grow horizontally?
  • Consistency & Staleness – Do you understand read-through vs write-through vs write-behind patterns?
  • Operational Insight – Do you consider monitoring, failure recovery, and cost?
  • Communication – Can you explain each piece clearly and justify trade-offs?

Core Requirements and Driving Numbers

Typical distributed-cache interview constraints look like this:

  • Throughput: Roughly 100 million reads per day → about 1,200 reads per second.
  • Latency: Sub-millisecond read latency for hot keys; <10 ms for cache miss fallback.
  • Data Size: 10 GB of hot objects, each up to 1 MB, fitting comfortably in memory.
  • Durability: Cache miss should fall back to a persistent store (e.g., DynamoDB) without data loss.

These ball-park figures let you decide how many cache nodes you need, what replication factor to use, and whether you need a write-ahead log.

Request Path – One Hop at a Time

  1. Client → Edge: The client hits a CloudFront edge location. CloudFront can serve cached static blobs directly, reducing load on the cache layer.
  2. Edge → ElastiCache (Redis) Node: If the key is hot, the request is routed to the nearest Redis node in an Amazon ElastiCache cluster. The node performs a GET and returns the value.
  3. Cache Miss → DynamoDB: On a miss, the application server queries DynamoDB for the authoritative value, writes it back to Redis (SET), and returns the response.
  4. Write Path: For updates, the app writes to DynamoDB first and then publishes an invalidation message to an SQS queue (cache-aside with asynchronous invalidation; write-through would update the cache in the same synchronous step as the store). A background worker consumes the message and evicts or updates the corresponding Redis entry.

Data Model & Key Choice

  • Key: Use a composite of entity_type:id (e.g., user:12345). It is readable and namespaced; the cache still hashes the whole key to choose a shard, so load spreads evenly instead of piling every user key onto one shard.
  • Value: Store a JSON blob or binary payload; keep it under the Redis max-string size (512 MB) but aim for <1 MB for latency.
  • Why Not Plain Hashing? Plain hashing spreads load evenly but loses the ability to co-locate related keys for batch invalidation. The composite key lets you target groups of keys (all user objects) with a single SQS message.

What a Complete Junior Answer Covers

AspectJuniorSeniorStaff
ScopeLists basic requirements (latency, throughput).Refines requirements with numbers, adds durability.Adds cost, ops, and risk analysis.
ArchitectureSingle-region ElastiCache cluster, fallback to DynamoDB.Multi-AZ replication, sharding, read-through/write-through patterns.Full mesh of edge, cache, persistent store, async invalidation, monitoring.
Key DesignSimple key, mentions hash.Composite key, explains grouping.Discusses hash-based partitioning, consistent hashing, and hot-spot mitigation.
Failure ModesMentions node crash.Adds network partition, cache stampede, and fallback logic.Covers cascading failures, data loss, cost of over-provisioning, and operational alerts.
OperationsBasic health checks.Metrics (hit ratio, latency) and scaling policy.Detailed SLO/SLA, cost modeling, capacity planning, and incident response runbooks.
Trade-offsChooses Redis for speed.Defends write-through vs write-behind.Defends replication factor vs cost, and when to use a CDN layer.

Junior Answer Checklist

  • State read latency target (<1 ms) and throughput (~1,200 ops/s).
  • Sketch a single-region ElastiCache cluster with DynamoDB fallback.
  • Explain a simple GET/SET flow.
  • Mention cache miss handling and basic health checks.

What a Senior Answer Adds

  • Sharding: Use Redis Cluster with 12 shards across three AZs. Each shard holds a subset of the key space, reducing contention.
  • Replication: Primary-replica setup for high availability; failover handled by ElastiCache automatic failover.
  • Cache Stampede Mitigation: Add a short-lived lock (e.g., SETNX) to prevent many requests from hammering DynamoDB on a miss.
  • Write-Behind Option: Discuss pros/cons of buffering writes before persisting, useful when write bursts are high.
  • Metrics: Hit ratio, eviction rate, and latency broken out by region.
  • Scaling Policy: Auto-scale the number of nodes based on CPU and memory thresholds.
  • Security: VPC isolation, IAM role for DynamoDB access, and encryption-at-rest.

What a Staff Answer Adds

  • Failure Mode Deep Dive:
  • Network Partition: Use DynamoDB’s global tables to keep a read replica in another region; cache nodes fall back to the remote replica.
  • Cache Stampede: Implement “request coalescing” with a small TTL and probabilistic early expiration.
  • Cold Start: Warm the cache on deployment using a background job that pre-loads the most popular keys.
  • Operations & Cost:
  • Estimate node count: 6 Redis nodes (2 per AZ) each with 32 GB memory → roughly 192 GB total, enough for 10 GB hot data with headroom.
  • DynamoDB read capacity: 1,200 reads/s → on-demand mode or provisioned 1,500 RCUs.
  • Monthly cost: Rough ball-park of a few thousand dollars; compare to a pure DynamoDB solution.
  • Trade-off Defense:
  • Latency vs Cost: Adding more cache nodes reduces latency but raises cost; a 99.9% hit ratio often justifies the expense.
  • Consistency: Choose write-through when the cache must stay in step with the store; write-behind if write latency is critical.
  • Observability: CloudWatch dashboards for cache hit ratio, eviction count, and replication lag. Set alarms on >5% miss rate.
  • Automation: Use CloudFormation or CDK to provision the whole stack, ensuring reproducibility.

Mistakes That Sink This Question

  • Skipping Numbers: Interviewers love concrete numbers. Vague “high throughput” invites follow-up that can trip you.
  • Ignoring Cache Invalidation: Forgetting how stale data is evicted is a classic failure.
  • Over-Engineering: Pulling in Kafka or complex multi-region replication when the problem only needs a single region.
  • No Ops View: Failing to discuss monitoring, scaling, and cost makes the design feel academic.
  • Choosing the Wrong Store: Using S3 as a cache layer without a fast lookup layer defeats the purpose.

How to Practice This

  1. Study the worked design in the System Design AI library. See how a senior-level answer is broken down step by step.
  2. Draw your own canvas on the practice board. Start with the request path, then add sharding, replication, and failure handling layers.
  3. Run through the rubric: check that you covered requirements, numbers, key design, failure modes, and trade-offs. Revise until each section is crisp and defensible.

Questions people ask

What is the typical size of a Redis node for a distributed cache?

A Redis node often runs with 32 GB of memory, giving enough headroom for hot data and eviction buffers. The exact size depends on your hit-ratio target and data footprint.

When should I use write-behind instead of write-through?

Write-behind buffers writes to reduce latency, useful when write bursts are high and occasional stale reads are acceptable. Write-through guarantees immediate consistency but adds write latency.

How do I avoid a cache stampede on a miss?

Use a short-lived lock (SETNX) to let only one request populate the cache, while others wait or fallback to the backing store. Probabilistic early expiration can also smooth traffic spikes.

Do I need a CDN like CloudFront for a distributed cache?

CloudFront helps serve static objects directly at the edge, reducing load on the cache layer. For dynamic data that changes frequently, a CDN complements but does not replace the cache.