10. Design an API Gateway

The brief

Design an API gateway for a multi-tenant product with independently deployed backend services. Follow a request through identity checks, route selection and failure handling. Explain how route and policy changes become effective while a gateway continues serving during configuration-service downtime.

• Route 200,000 requests/second at peak across 200 backend services in two regions.
• Tenants have different quotas, and some requests stream responses for ten minutes.
• Operators update routes and policies hourly; a bad change must be reversible without restarting the fleet.

Constraints

Gateway latency budget≤ 20 milliseconds

Declare p95 overhead excluding backend processing and client network time.

Trusted identity boundary

Do not trust caller-supplied identity headers; propagate authenticated context without leaking credentials.

Configuration outage

Define last-known-good behavior and safe policy limits when updates cannot be fetched.

What to cover

1. 01

   Request lifecycle

   Show authentication, quotas, routing, timeouts, streaming and response handling.

2. 02

   Configuration lifecycle

   Describe versioned route changes, validation, canaries and rollback.

3. 03

   Capacity and overload

   Estimate gateway and connection capacity; isolate a slow or failing backend.

4. 04

   Failure and retry policy

   Trace a bad route and a timed-out write; explain which retries are safe.

Worked designs

Explore the architecture and decisions, then build on an example with Coach.

Official design

Design an API Gateway

One front door for hundreds of services: routing, auth, limits, retries and canaries, with a control plane that ships config safely to thousands of machines: one-hour boards for junior, senior and staff, with the theory behind them.

Explore the design

Review rubric

AI feedback uses these criteria. Scores are practice feedback.