System Design AI

Load balancing in System Design Interviews: When to Use It and What to Say

Load balancing is the practice of distributing incoming client requests across a pool of servers so that no single machine becomes a bottleneck. In an interview you can frame it as the answer to the classic "how do we handle more traffic than a single server can sustain?" problem.

The problem it solves

Imagine you are designing a URL-shortening service that must serve billions of redirects per day. A single web server can only handle a few thousand requests per second before CPU or network limits kick in. If traffic spikes, that server will queue requests, increase latency, and eventually fail. The bottleneck is the front-end web tier.

How a load balancer works – step by step

  1. Client sends a request – The DNS entry for the service points to the public IP of a load balancer (e.g., AWS Elastic Load Balancing). The client never talks to the backend directly.
  2. LB receives the request – It terminates the TCP (or TLS) connection, extracts the destination path, and consults its routing table.
  3. Choose a target – The LB runs an algorithm:
  • Round-robin: pick the next healthy instance in a circular list.
  • Least connections: pick the instance with the fewest active streams.
  • Consistent hashing: map the request key (e.g., URL hash) to a specific server, preserving cache locality.
  1. Health check – The LB probes every instance’s health endpoint on a fixed interval in the background, so the algorithm in step 3 only picks from instances currently marked healthy. An instance that fails several checks in a row is taken out of rotation until it passes again.
  2. Forward the request – The LB opens a new connection to the selected backend, sends the request, and streams the response back to the client.
  3. Response to client – The client sees a single IP address and experiences the latency of the fastest healthy server.

Small worked example

Suppose you have three web servers (A, B, C) behind an AWS Application Load Balancer. Traffic arrives at 600 requests per second. Each server can comfortably handle 250 rps.

  • Round-robin picks A for the first request, B for the second, C for the third, then repeats.
  • Each server receives ~200 rps, leaving headroom below its 250 rps limit.
  • If server B fails a health check, the LB removes B from the pool. The round-robin now alternates between A and C, each handling ~300 rps, which is over the 250 rps capacity, so latency climbs and some requests fail with 503 until capacity is added.

Where load balancing shows up in classic interview questions

Design QuestionWhere the LB appearsReason for inclusion
URL Shortener (e.g., bit.ly)At the entry point before the API layerTo spread read-heavy redirect traffic across many stateless web nodes
Video Streaming Service (e.g., Twitch)In front of the media servers that serve chunked video filesTo handle bursty viewer spikes and keep latency low
Real-time Chat System (e.g., Slack)Before the WebSocket gateway that maintains persistent connectionsTo balance long-lived connections and enable horizontal scaling

Trade-offs and the alternative interviewers love to probe

AspectLoad balancer (LB)Direct client-to-backend (no LB)
ScalabilityEasy to add/remove instances; traffic automatically spreadsScaling requires client-side sharding or DNS round-robin, which is coarse-grained
LatencyExtra hop adds a few milliseconds; health checks add negligible overheadOne hop, but risk of hitting an overloaded server
CostYou pay for the LB service and extra network hopsNo LB cost, but you may need larger, more expensive instances
Failure handlingAutomatic health checks, failover, and graceful drainingManual failure detection; higher chance of client errors
ComplexityConfiguration (listener rules, target groups) adds operational workSimpler architecture, but harder to evolve

Interviewers often follow up with questions like:

  • What if the LB itself becomes a bottleneck? – Discuss scaling the LB (multi-AZ deployment, DNS-level load balancing) or moving to a peer-to-peer routing layer.
  • How do you handle sticky sessions? – Explain session affinity using cookies or consistent hashing, and why it matters for stateful services.
  • What if you need layer-7 routing based on content type? – Mention using an application-level LB that can inspect HTTP headers.

30-second spoken version

"Load balancing distributes incoming traffic across multiple identical servers to avoid overloading any single node. A client hits a public IP, the LB terminates the connection, runs an algorithm like round-robin or consistent hashing, checks health, then forwards the request to a healthy backend. This gives you horizontal scalability, better latency under load, and automatic failover. In interviews you’ll see it at the front of URL shorteners, video streaming services, and real-time chat systems. Be ready to discuss trade-offs: extra hop latency, cost of the LB, and the need for health checks versus a simpler direct-to-backend approach."

Common mistakes candidates make

  • Skipping health checks – Claiming the LB magically distributes traffic without mentioning how it avoids dead servers.
  • Over-engineering the algorithm – Jumping to advanced techniques like weighted consistent hashing before establishing the basic round-robin.
  • Treating the LB as a free scalability lever – Forgetting to discuss its own capacity limits, multi-AZ deployment, and cost.
  • Ignoring session affinity – When the service stores state locally, not addressing how sticky sessions are maintained.
  • Listing the LB as a vague "AWS ELB" without tying it to the request flow – Interviewers want to see the step-by-step path, not just a component name.

How to practice this

  1. Study the worked design – In the System Design AI library you can open the worked design for a load balancer like AWS Elastic Load Balancing. Observe the diagram, notes on each component, and the interview board format.
  2. Draw your own diagram – Use the canvas to recreate the request path for a video streaming service. Mark where the load balancer sits, the health-check flow, and any routing rules.
  3. Run a mock interview – Explain the LB in 30 seconds, then answer follow-up questions about scaling the LB, sticky sessions, and cost. Record yourself to spot filler or missing steps.

Questions people ask

When should I mention a load balancer in a design interview?

Bring it up as soon as the problem involves more traffic than a single server can handle, or when you need high availability across zones.

What’s the difference between round-robin and consistent hashing?

Round-robin cycles through servers evenly, while consistent hashing maps a request key to a specific server, preserving cache locality and reducing reshuffling when servers are added or removed.

Can a DNS-level load balancer replace an application load balancer?

DNS can distribute traffic across regions, but it lacks health checks, fine-grained routing, and cannot balance at the request level, so it’s usually paired with a downstream LB.

How do I handle SSL termination with a load balancer?

The LB can terminate TLS, offloading the decryption work from backend servers. You then forward HTTP to the backends or re-encrypt with TLS if end-to-end encryption is required.