System Design AI

Design a Rate Limiter: Junior, Senior, and Staff Answers

When you’re asked to design a rate limiter, the interviewer is looking for three things: your ability to model a simple but high-traffic control flow, your sense of scalability, and your awareness of operational concerns. A rate limiter sits at the edge of a service, decides whether a request should be allowed, and must do it in a few milliseconds while handling spikes that can be orders of magnitude larger than the steady-state load.

What the interviewer is testing

  • Consistency vs. availability – can you explain why a limiter needs strong consistency for correctness, and where you can relax it for performance?
  • Data modeling – do you know the difference between a fixed-window counter, a sliding-window log, and a token-bucket algorithm?
  • Scalability – how will you keep the limiter cheap when the request rate grows from a few hundred per second to millions?
  • Operations – can you talk about monitoring, alerting, and graceful degradation when the limiter itself fails?

Core requirements and driving numbers

Typical interview constraints are loosely defined, so you’ll need to pick ball-park figures and state the hedge.

  • Throughput – the service must handle roughly 1 million requests per day, which is about 12 requests per second. In a hot-spot scenario you might see 10 × the average, so design for ~120 RPS.
  • Burst tolerance – clients often need to send a burst of up to 100 requests within a second, then settle to the average rate.
  • Latency budget – the limiter decision must add no more than 5 ms to the request path.
  • Durability – occasional loss of a token is acceptable, but you must avoid granting more tokens than allowed.

Core design in prose

Request path, one hop at a time

  1. Client sends request to your API gateway (e.g., a gateway service running in your VPC behind an Application Load Balancer). The gateway reads the user-id or API key from the request header.
  2. Gateway queries a fast store (Amazon ElastiCache for Redis, in the same VPC) for the current token bucket state keyed by the user-id.
  3. Redis runs a Lua script that atomically checks the bucket, refills tokens based on elapsed time, and either decrements a token (allow) or returns a rejection.
  4. Result propagates back to the gateway, which either forwards the request to the backend service or returns HTTP 429.

Data model and key choice

  • Key: rate_limit:{user_id} – a short string that hashes well in Redis and DynamoDB if you need spillover.
  • Value: a hash with fields tokens, last_refill_ts. The Lua script updates these fields in a single command, guaranteeing atomicity.
  • Why Redis over DynamoDB? Redis gives sub-millisecond latency and native atomic scripts, which is essential for the 5 ms budget. DynamoDB can be used for overflow or persistence, but it adds extra round-trips.

What a complete junior answer covers

  • Identify the problem and list the basic functional requirements (limit per user, burst, global limit).
  • Choose a token-bucket algorithm and explain it with a simple diagram.
  • Pick a single store (Redis) and describe the Lua script flow.
  • Mention a fallback (return 429) and a basic monitoring metric (requests allowed vs. rejected).
  • Sketch the request path in prose, without deep dive into sharding or replication.

What a senior answer adds

  • Sharding strategy – split the Redis keyspace by user-id prefix, use Redis Cluster to spread load across nodes.
  • Consistency trade-off – discuss using a weakly consistent read-through cache for non-critical limits, but keeping the token bucket in the primary node for correctness.
  • Overflow handling – when the Redis cluster is saturated, spill to Amazon DynamoDB with a TTL-based fallback, ensuring the limiter never blocks a request because the cache is full.
  • Metrics and alerts – expose CloudWatch metrics for AllowedRequests, RejectedRequests, and CacheMisses; set alarms for sudden spikes.
  • Testing – describe a load test with an HTTP load generator that replays burst traffic against the gateway and verifies the 5 ms latency.

What a staff answer adds

  • Failure modes – enumerate scenarios such as Redis node loss, network partition, and clock skew. Explain how you would use a multi-AZ Redis replication group and fallback to DynamoDB with a conservative limit.
  • Operational cost – estimate the cost of a small Redis cluster (e.g., three cache.t3.micro nodes) and compare it to a DynamoDB table with on-demand capacity. Show that the Redis cost is roughly a few tens of dollars per month, while DynamoDB costs scale with read/write units.
  • Trade-off defense – argue why you prefer a slightly over-provisioned token bucket in Redis for low latency, even though it may waste a few tokens during idle periods. Contrast this with a strict leaky-bucket implementation that would require tighter coordination and higher latency.
  • Security – mention Redis access control lists (ElastiCache role-based access control) that restrict the gateway to the rate-limit key prefix, and encrypt data at rest with AWS KMS.
  • Observability – add distributed tracing (AWS X-Ray) to follow a request from the gateway through the Redis call, enabling root-cause analysis when latency spikes.

Common mistakes that sink this question

MistakeWhy it hurts
Choosing a database without atomic operations (e.g., plain S3)You cannot enforce the limit in a single round-trip, breaking the latency budget.
Over-engineering a global limiterA global counter forces every request through a single hotspot, causing contention and scaling failures.
Ignoring burst handlingWithout a token bucket or leaky bucket, the system will reject legitimate bursts, which interviewers see as a design gap.
No monitoringWithout metrics you cannot demonstrate that the limiter behaves correctly under load, and you miss an opportunity to discuss ops.
Assuming the limiter is the only failure pointForgetting to plan for Redis outages or network partitions shows a lack of operational thinking.

How to practice this

  1. Study the worked design in the System Design AI library. It opens on a walkthrough whose diagram grows from the junior answer to the staff answer, and every component can be clicked for its notes.
  2. Draw your own canvas: start with the junior version, then layer on sharding, fallback storage, and failure handling. Use the provided rubric to check completeness.
  3. Run a mock interview with a peer, or ask the AI coach to review your canvas, and iterate until you can articulate the staff-level trade-offs fluently.

Questions people ask

What algorithm is best for a rate limiter?

Token bucket is the most common because it handles both steady-state rates and bursts with simple arithmetic. Sliding-window logs give exact counts but are heavier; leaky bucket is similar but harder to implement atomically.

Can I use DynamoDB alone for rate limiting?

You can, but DynamoDB lacks sub-millisecond latency and atomic scripts, so you would need conditional writes and retries, which add latency. It works for low-traffic limits or as a spillover store.

How do I handle Redis failures in a rate limiter?

Deploy Redis in a multi-AZ cluster with automatic failover. On a node outage, fallback to DynamoDB with a conservative limit, and log the event for later analysis.

What metrics should I monitor for a rate limiter?

Track allowed vs. rejected requests, cache hit/miss rates, latency of the limiter decision, and error counts from Redis or DynamoDB. Set alerts on sudden spikes or sustained high rejection rates.