Design an API Gateway: Junior, Senior, and Staff Answers
When you hear "Design an API gateway" in a system-design interview, the interviewer is looking for three things: how you think about traffic flow, how you secure and monitor a public entry point, and how you reason about operational concerns at scale. The gateway sits at the edge of a micro-service ecosystem, so you need to balance performance, reliability, and cost while keeping the design understandable.
What the interviewer is testing
- Scalability – can you size the gateway for millions of requests per day and reason about burst traffic?
- Security – do you know where to place authentication, rate limiting, and DDoS protection?
- Observability & Ops – will you instrument metrics, handle failures, and plan for upgrades?
- Trade-off awareness – are you comfortable choosing a managed service versus a custom solution?
Core requirements and driving numbers
| Requirement | Typical ballpark (hedged) |
|---|---|
| Peak request rate | Roughly 10,000 RPS (≈864 M requests per day) |
| 99th-percentile latency | ≤ 100 ms end-to-end for a simple GET |
| Payload size | Up to 5 MB per request, average 200 KB |
| Availability | 99.99 % uptime (four nines) |
These numbers guide the choice of load balancers, caching layers, and data stores. For example, 10,000 RPS translates to about 10 R per millisecond, which means any single component must handle at least that throughput with headroom.
Core design – request path, one hop at a time
- Client → CloudFront – CloudFront provides global edge caching and DDoS protection (AWS Shield). Static responses (e.g., health checks, docs) are served directly.
- CloudFront → API Gateway (AWS API Gateway) – The managed gateway terminates TLS, performs request validation, and routes to the appropriate backend based on stage and path.
- API Gateway → Lambda authorizer – A lightweight Lambda checks JWTs or OAuth tokens. Successful auth returns a policy that the gateway uses to allow or deny the request.
- API Gateway → VPC Link → NLB – For high-throughput services, the gateway forwards traffic over a VPC Link to a Network Load Balancer (NLB) that distributes to a fleet of ECS/Fargate services.
- NLB → Service – The service processes the request, optionally reads/writes to DynamoDB (for low-latency key-value) or Aurora (for relational needs). Responses travel back the same path.
- Logging & metrics – CloudWatch logs capture request IDs, latency, and error codes. A Kinesis Data Firehose streams logs to S3 for long-term analysis.
Data model – The gateway itself stores no business data. It keeps a small configuration table in DynamoDB: route → target service, auth scheme, rate-limit thresholds. This table is read at cold start and cached in the Lambda authorizer.
Key choice – Using AWS API Gateway over a self-hosted NGINX+Envoy stack. The managed service gives you built-in throttling, request validation, and stage deployments, which a custom stack would require building from scratch.
What a complete junior answer covers
- Basic flow as above, naming CloudFront, API Gateway, Lambda authorizer, NLB, and a backend service.
- Functional requirements – routing, auth, throttling, health checks.
- Non-functional – rough latency target, mention of 99.99 % SLA.
- Simple diagram – one-hop request path.
- Scalability note – API Gateway scales automatically; NLB can handle millions of connections.
- Missing pieces – no deep discussion of failure handling, cost, or operational monitoring.
What a senior answer adds
- Rate limiting strategy – per-API key token bucket stored in ElastiCache (Redis) for low-latency counters.
- Caching – use API Gateway’s built-in response caching for GET endpoints, with CloudFront edge caches in front as a separate layer.
- Versioning & blue-green deployments – separate stages (dev, test, prod) and can shift traffic with weighted aliases.
- Observability – X-Ray tracing across Lambda, NLB, and services; CloudWatch dashboards for RPS, error rates, and latency percentiles.
- Security depth – WAF rules for IP blacklisting, OWASP top-10 protection, and mutual TLS for internal services.
- Cost awareness – rough estimate of API Gateway request cost (≈$3 per million requests) and NLB data processing charges, showing awareness of trade-offs.
What a staff answer adds
- Failure modes – detailed analysis of each hop: CloudFront edge failure, API Gateway throttling, Lambda cold start latency, NLB target health check failures, backend timeouts. Propose circuit-breaker patterns in the service layer (a client library or an Envoy sidecar) or a custom Lambda that returns fallback responses.
- Operational playbook – automated canary deployments via CodeDeploy, health-check alarms, and auto-scaling policies for the ECS service based on CPU and request latency.
- Cost optimization – evaluate reserved capacity for NLB vs. on-demand; use DynamoDB on-demand vs. provisioned based on traffic patterns; enable API Gateway caching to reduce backend calls.
- Trade-off defense – argue why a managed API Gateway is preferred over a self-managed Envoy mesh: lower operational burden, built-in security, and predictable scaling, even though it adds per-request cost.
- Governance – central schema registry for request/response contracts, CI checks for breaking changes, and audit logs stored in S3 with lifecycle policies.
Common mistakes that sink this question
| Mistake | Why it hurts |
|---|---|
| Ignoring security | Leaves the gateway open to injection, auth bypass, and DDoS attacks. |
| Over-engineering the data plane | Adding a database for request routing when a static config or DynamoDB cache would suffice wastes time. |
| Forgetting latency budgets | Without a clear latency target you cannot justify caching or edge placement. |
| No failure handling | Interviewer expects you to discuss what happens if CloudFront times out or the Lambda authorizer fails. |
| Skipping cost discussion | The gateway is a paid service; not addressing cost shows tunnel vision. |
Comparison of the three answer levels
| Aspect | Junior | Senior | Staff |
|---|---|---|---|
| Scope | Basic request flow, auth, health checks | Adds caching, rate limiting, observability, deployment stages | Adds failure analysis, ops playbook, cost trade-offs, governance |
| Depth | Lists services, mentions scaling | Explains why each service is chosen, shows metrics | Provides quantitative failure budgets, cost models, and defense of architecture |
| Risks addressed | None beyond basic SLA | Rate-limit abuse, missing logs | Full failure modes, operational readiness |
| Typical time to convey | 5-7 min | 10-12 min | 15-20 min |
How to practice this
- Study the worked design in the System Design AI library. Read the step-by-step walkthrough and note how each component is justified.
- Sketch your own canvas: draw the request path, label each AWS service, and write a one-sentence rationale for each choice.
- Run a mock interview: explain the junior, senior, and staff versions aloud, then ask a peer to quiz you on failure modes and cost trade-offs.
Questions people ask
What is the main difference between API Gateway and a load balancer?
API Gateway focuses on request-level features like authentication, throttling, and request validation, while a load balancer (NLB/ALB) simply distributes traffic without those API-specific controls.
How do you handle rate limiting for an API gateway?
A common pattern is to store per-key counters in a fast store such as ElastiCache (Redis) and enforce a token-bucket algorithm in a Lambda authorizer or edge function.
Can API Gateway cache responses?
Yes. API Gateway offers built-in response caching for a stage, and CloudFront edge caches in front of it add a second layer, reducing backend load for GET requests.
What monitoring should be in place for an API gateway?
Collect request counts, latency percentiles, and error codes in CloudWatch, enable X-Ray tracing for end-to-end visibility, and set alarms on 99th-percentile latency and error spikes.