System Design AI

Load Balancing: The Follow-Up Questions Interviewers Ask, With Answers

Load balancing is a staple of any scalable architecture, so interviewers use it as a quick way to gauge how deeply you understand distributed systems. The first follow-up question usually checks whether you can move beyond the buzzword and talk about real-world constraints.

When you answer those probes well, you demonstrate that you can design for traffic spikes, failure domains, and cost limits—exactly the skills interviewers look for in senior and staff engineers.

How do you choose between DNS-based and proxy-based load balancing?

I start by comparing the control plane and data plane. DNS-based balancing (e.g., Route 53) is cheap and works at the edge, but it only offers coarse-grained routing and a TTL-driven cache, so a sudden 5× traffic surge may take several seconds to propagate. Proxy-based balancers (like AWS ELB) let me steer traffic per request, support health checks, and enforce sticky sessions. In a typical web service that sees 200 req/s on average with occasional bursts to 2 k req/s, I’d use a proxy balancer for the burst handling and fall back to DNS for global failover.

What health-check strategy would you implement?

I configure active health checks that ping a lightweight /healthz endpoint every 10 seconds. The endpoint returns 200 only if the service can connect to its downstream database and process a dummy request within 100 ms. If three consecutive checks fail, the instance is marked unhealthy and removed from the pool. For critical paths, I also add a passive check that monitors error rates; if the 5-minute error rate exceeds 1 %, the balancer triggers a soft-drain of the affected node.

How do you handle uneven load due to “hot” keys or sessions?

I combine consistent hashing with a secondary round-robin fallback. Requests that contain a user-ID are hashed to a specific backend, ensuring session affinity. If a particular shard receives more than 30 % of total traffic for five minutes, I spin up an extra replica for that shard and re-hash the keys. In practice, on a service with 10 k req/s, this approach keeps the max per-node load under 1.2 k req/s, well within typical CPU limits.

What is the role of connection draining, and how long should it be?

Connection draining lets in-flight requests finish before an instance is terminated. I set a drain timeout of 30 seconds, which matches the typical maximum request latency for our API. If a request exceeds that, I return a 503 so the client can retry. This window balances graceful shutdown with the need to recycle unhealthy nodes quickly; in a 100-node fleet, a 30-second drain adds less than 0.5 % extra capacity overhead.

How would you design load balancing for a multi-region deployment?

First, I use DNS-based geo routing to send users to the nearest region. Within each region, a regional proxy balancer (e.g., ALB) distributes traffic across availability zones. For cross-region failover, I configure health checks on the primary region’s endpoint; I keep the DNS TTL at about 30 seconds from the start, and if the health check fails for 2 minutes, DNS stops answering with the primary region and traffic is rerouted to the secondary region. This layered approach gives sub-second latency for normal traffic while still providing a robust disaster-recovery path.

How do you mitigate the “thundering herd” problem when many instances become healthy at once?

I stagger instance registration using exponential back-off. When a new instance passes its health check, it signals the balancer but remains in a warm-up state for 60 seconds, during which the balancer sends only a fraction (e.g., 10 %) of traffic. The fraction doubles every 15 seconds until the instance reaches full weight. This prevents a sudden surge of traffic that could overload downstream services.

What cost considerations affect your load-balancer choice?

Proxy balancers charge per hour and per GB of data processed, while DNS-based services charge per query. For a service that processes 5 TB of data monthly with an average of 300 req/s, the proxy cost is roughly $0.008 per GB, totaling about $40 per month, plus $18 for the balancer instances. If the traffic is highly variable, the DNS cost (around $0.40 per million queries) is negligible, but the lack of fine-grained routing may increase downstream scaling costs. I usually pick the cheaper option that still meets latency SLAs.

How would you secure traffic between the client and the load balancer?

I terminate TLS at the load balancer using certificates from a managed PKI service. The balancer then re-encrypts traffic to the backend using mutual TLS, enforcing client authentication and protecting data in-flight. For a public API, I also enable WAF rules to block common attack vectors. This layered security adds roughly 0.5 ms of latency per hop, which is acceptable for most web workloads.

How do you future-proof your load-balancing design for unknown traffic patterns?

I build in elasticity by decoupling the balancer from the compute pool. The balancer references a target group that can auto-scale based on CPU, request latency, or queue depth. I also expose metrics (e.g., request per second, error rate) to an observability platform, so I can trigger new scaling policies without code changes. In a recent design, this allowed us to handle a 4× traffic increase during a product launch with only a single configuration tweak.

What separates a senior answer from a staff answer? A senior response explains the "what" and "why"—the choice of algorithm, health-check intervals, and basic cost trade-offs. A staff answer adds the "how" at scale: it discusses operational automation, long-term cost modeling, security layers, and how the design evolves with changing traffic patterns. Staff engineers also anticipate edge cases, such as cross-region latency spikes, and propose monitoring and mitigation strategies that go beyond the immediate design.

How to practice this

  1. Study the worked design on load balancing in the System Design AI library. It shows a complete end-to-end diagram with notes on each component.
  2. Write your own answer to each of the nine questions on a blank canvas, focusing on concrete numbers and trade-offs.
  3. Use the AI coach to review your canvas and iterate until the explanations flow naturally.

Questions people ask

What is the difference between layer-4 and layer-7 load balancing?

Layer-4 balancers route based on IP and port only, giving low overhead but no visibility into HTTP details. Layer-7 balancers inspect HTTP headers, paths, and cookies, allowing routing decisions like canary releases or session affinity, at the cost of extra processing.

Can DNS load balancing replace a traditional proxy balancer?

DNS can provide coarse global routing and simple failover, but it cannot perform per-request health checks or handle rapid traffic spikes. For most interactive services you still need a proxy balancer for fine-grained control.

How often should health checks be performed?

A typical pattern is a health ping every 10 seconds, marking a node unhealthy after three consecutive failures. Adjust the interval for latency-critical services (shorter) or low-traffic backends (longer).

What is a good rule of thumb for connection drain time?

Set the drain timeout slightly longer than your maximum request latency—30 seconds for most APIs, up to 60 seconds for longer-running operations—to let in-flight requests finish gracefully.