Rate limiting in System Design Interviews: When to Use It and What to Say
When an interviewer asks about protecting a service from a flood of traffic, the go-to answer is rate limiting. In one sentence, rate limiting is a mechanism that caps how many requests a client can make to a service within a defined time window.
The problem rate limiting solves
Imagine you are designing an API for a photo-sharing app. Users can upload images, but the storage backend can only handle roughly 5 GB of new data per minute. If a single user—or a bot—sends 10 000 upload requests in a second, the storage tier will be overwhelmed, latency will spike, and legitimate users will experience failures. Rate limiting solves this by throttling the request rate before it reaches the storage layer, ensuring the system stays within its capacity envelope.
How rate limiting works – step by step
A classic algorithm is the token bucket. The steps are:
- Initialize a bucket with a capacity of C tokens. Each token represents one allowed request.
- Refill the bucket at a steady rate R tokens per second. If the bucket is full, extra tokens are discarded.
- When a request arrives, the system checks the bucket:
- If a token is available, it is removed and the request proceeds.
- If the bucket is empty, the request is rejected (or queued) with a
429 Too Many Requestsresponse.
- Optional burst handling: because the bucket can hold multiple tokens, a short burst of traffic up to C requests can pass instantly, then the refill rate limits further traffic.
Small worked example
Suppose you set C = 10 tokens and R = 2 tokens/second. At time 0 the bucket is full (10 tokens). A client sends 12 requests in the first second:
- The first 10 consume the existing tokens and succeed.
- The remaining 2 find the bucket empty and are rejected.
One second later, the bucket has refilled 2 tokens, so the next two requests succeed, and the pattern repeats. This simple math makes it easy to explain the behavior during an interview.
Where rate limiting shows up in classic design questions
| Design Question | Where Rate Limiting Appears | Reason |
|---|---|---|
| API gateway for a SaaS platform | At the gateway layer, before routing to microservices | Prevents a single tenant from exhausting shared resources |
| Chat service (e.g., WhatsApp) | In the message-send endpoint, per-user or per-IP | Stops spam bots from flooding the system |
| Video streaming service | In the content delivery node, per-client video chunk requests | Controls bandwidth usage and protects CDN edge nodes |
In each case the limiter sits at the edge of the system where it can stop excess traffic before it reaches expensive downstream components.
Trade-offs and the alternative interviewers love to ask about
Strictness vs. latency
- Token bucket: Allows a burst of up to C requests and then admits traffic at the refill rate R, so a window of t seconds can pass up to C + R·t requests; anything beyond that is rejected immediately, which can feel harsh to users.
- Leaky bucket: Smooths traffic by processing requests at a constant rate, turning bursts into a queue. Downstream load is predictable and never exceeds the drain rate, but queued requests wait longer and are dropped once the queue is full.
Distributed consistency
If your service runs on many nodes, each node needs a view of the global request count. Options include:
- Centralized store (e.g., Redis with INCR) – simple, but becomes a single point of failure and adds a network hop.
- Local token buckets with a shared quota – each node enforces its own bucket, and a periodic sync ensures the total does not exceed the global limit. This reduces latency but introduces a small risk of overshoot.
Alternative mechanisms
Interviewers often follow up with questions about circuit breakers or back-pressure. A circuit breaker trips when error rates rise, while back-pressure signals upstream services to slow down. Both achieve protection, but rate limiting is the proactive, request-count-based guard you should discuss first.
A 30-second spoken version
"Rate limiting caps how many requests a client can make in a given time window. A common implementation is the token bucket: you start with a bucket of tokens, refill it at a steady rate, and each request consumes a token. If the bucket is empty you return a 429. It protects downstream services from overload, allows short bursts, and can be centralized or distributed. Trade-offs involve latency versus strictness and the complexity of keeping counters consistent across many nodes."
Common mistakes candidates make
- Treating it like a firewall rule – Saying “just block IPs that exceed a threshold” ignores the nuanced behavior of bursts and refill rates.
- Omitting the algorithm – Interviewers expect you to name token bucket or leaky bucket; vague statements about “throttling” are insufficient.
- Ignoring distributed state – Assuming a single node can enforce the limit in a multi-region deployment shows a lack of scalability thinking.
- Not discussing error handling – Forgetting to mention the
429response, retry-after headers, or how clients should back off leaves the design incomplete.
How to practice this
- Study the worked design of a token bucket rate limiter in the System Design AI library, noting the diagram and component notes.
- Draw your own canvas for a new service—maybe a chat endpoint—and decide where the limiter lives, which algorithm you choose, and how you handle distributed counters.
- Run a mock interview where you explain the limiter in 30 seconds, then answer follow-up questions about trade-offs and alternatives.
Questions people ask
When should I mention rate limiting in a design interview?
Bring it up any time the problem involves shared downstream resources, potential abuse, or unpredictable traffic spikes. It signals you think about protection early.
What’s the difference between token bucket and leaky bucket?
Token bucket allows bursts up to the bucket size and then refills at a steady rate; leaky bucket processes requests at a constant rate, smoothing bursts into a queue.
How do I handle rate limiting across multiple data centers?
You can use a centralized store like Redis with sharding, or maintain local buckets with periodic quota sync. Each approach balances latency against strict global enforcement.
Can rate limiting replace authentication?
No. Authentication verifies identity; rate limiting controls usage. They complement each other but serve different purposes.