System Design AI

Design an API Gateway: Junior, Senior, and Staff Answers

When you hear "Design an API gateway" in a system-design interview, the interviewer is looking for three things: how you think about traffic flow, how you secure and monitor a public entry point, and how you reason about operational concerns at scale. The gateway sits at the edge of a micro-service ecosystem, so you need to balance performance, reliability, and cost while keeping the design understandable.

What the interviewer is testing

  • Scalability – can you size the gateway for millions of requests per day and reason about burst traffic?
  • Security – do you know where to place authentication, rate limiting, and DDoS protection?
  • Observability & Ops – will you instrument metrics, handle failures, and plan for upgrades?
  • Trade-off awareness – are you comfortable choosing a managed service versus a custom solution?

Core requirements and driving numbers

RequirementTypical ballpark (hedged)
Peak request rateRoughly 10,000 RPS (≈864 M requests per day)
99th-percentile latency≤ 100 ms end-to-end for a simple GET
Payload sizeUp to 5 MB per request, average 200 KB
Availability99.99 % uptime (four nines)

These numbers guide the choice of load balancers, caching layers, and data stores. For example, 10,000 RPS translates to about 10 R per millisecond, which means any single component must handle at least that throughput with headroom.

Core design – request path, one hop at a time

  1. Client → CloudFront – CloudFront provides global edge caching and DDoS protection (AWS Shield). Static responses (e.g., health checks, docs) are served directly.
  2. CloudFront → API Gateway (AWS API Gateway) – The managed gateway terminates TLS, performs request validation, and routes to the appropriate backend based on stage and path.
  3. API Gateway → Lambda authorizer – A lightweight Lambda checks JWTs or OAuth tokens. Successful auth returns a policy that the gateway uses to allow or deny the request.
  4. API Gateway → VPC Link → NLB – For high-throughput services, the gateway forwards traffic over a VPC Link to a Network Load Balancer (NLB) that distributes to a fleet of ECS/Fargate services.
  5. NLB → Service – The service processes the request, optionally reads/writes to DynamoDB (for low-latency key-value) or Aurora (for relational needs). Responses travel back the same path.
  6. Logging & metrics – CloudWatch logs capture request IDs, latency, and error codes. A Kinesis Data Firehose streams logs to S3 for long-term analysis.

Data model – The gateway itself stores no business data. It keeps a small configuration table in DynamoDB: route → target service, auth scheme, rate-limit thresholds. This table is read at cold start and cached in the Lambda authorizer.

Key choice – Using AWS API Gateway over a self-hosted NGINX+Envoy stack. The managed service gives you built-in throttling, request validation, and stage deployments, which a custom stack would require building from scratch.

What a complete junior answer covers

  • Basic flow as above, naming CloudFront, API Gateway, Lambda authorizer, NLB, and a backend service.
  • Functional requirements – routing, auth, throttling, health checks.
  • Non-functional – rough latency target, mention of 99.99 % SLA.
  • Simple diagram – one-hop request path.
  • Scalability note – API Gateway scales automatically; NLB can handle millions of connections.
  • Missing pieces – no deep discussion of failure handling, cost, or operational monitoring.

What a senior answer adds

  • Rate limiting strategy – per-API key token bucket stored in ElastiCache (Redis) for low-latency counters.
  • Caching – use API Gateway’s built-in response caching for GET endpoints, with CloudFront edge caches in front as a separate layer.
  • Versioning & blue-green deployments – separate stages (dev, test, prod) and can shift traffic with weighted aliases.
  • Observability – X-Ray tracing across Lambda, NLB, and services; CloudWatch dashboards for RPS, error rates, and latency percentiles.
  • Security depth – WAF rules for IP blacklisting, OWASP top-10 protection, and mutual TLS for internal services.
  • Cost awareness – rough estimate of API Gateway request cost (≈$3 per million requests) and NLB data processing charges, showing awareness of trade-offs.

What a staff answer adds

  • Failure modes – detailed analysis of each hop: CloudFront edge failure, API Gateway throttling, Lambda cold start latency, NLB target health check failures, backend timeouts. Propose circuit-breaker patterns in the service layer (a client library or an Envoy sidecar) or a custom Lambda that returns fallback responses.
  • Operational playbook – automated canary deployments via CodeDeploy, health-check alarms, and auto-scaling policies for the ECS service based on CPU and request latency.
  • Cost optimization – evaluate reserved capacity for NLB vs. on-demand; use DynamoDB on-demand vs. provisioned based on traffic patterns; enable API Gateway caching to reduce backend calls.
  • Trade-off defense – argue why a managed API Gateway is preferred over a self-managed Envoy mesh: lower operational burden, built-in security, and predictable scaling, even though it adds per-request cost.
  • Governance – central schema registry for request/response contracts, CI checks for breaking changes, and audit logs stored in S3 with lifecycle policies.

Common mistakes that sink this question

MistakeWhy it hurts
Ignoring securityLeaves the gateway open to injection, auth bypass, and DDoS attacks.
Over-engineering the data planeAdding a database for request routing when a static config or DynamoDB cache would suffice wastes time.
Forgetting latency budgetsWithout a clear latency target you cannot justify caching or edge placement.
No failure handlingInterviewer expects you to discuss what happens if CloudFront times out or the Lambda authorizer fails.
Skipping cost discussionThe gateway is a paid service; not addressing cost shows tunnel vision.

Comparison of the three answer levels

AspectJuniorSeniorStaff
ScopeBasic request flow, auth, health checksAdds caching, rate limiting, observability, deployment stagesAdds failure analysis, ops playbook, cost trade-offs, governance
DepthLists services, mentions scalingExplains why each service is chosen, shows metricsProvides quantitative failure budgets, cost models, and defense of architecture
Risks addressedNone beyond basic SLARate-limit abuse, missing logsFull failure modes, operational readiness
Typical time to convey5-7 min10-12 min15-20 min

How to practice this

  1. Study the worked design in the System Design AI library. Read the step-by-step walkthrough and note how each component is justified.
  2. Sketch your own canvas: draw the request path, label each AWS service, and write a one-sentence rationale for each choice.
  3. Run a mock interview: explain the junior, senior, and staff versions aloud, then ask a peer to quiz you on failure modes and cost trade-offs.

Questions people ask

What is the main difference between API Gateway and a load balancer?

API Gateway focuses on request-level features like authentication, throttling, and request validation, while a load balancer (NLB/ALB) simply distributes traffic without those API-specific controls.

How do you handle rate limiting for an API gateway?

A common pattern is to store per-key counters in a fast store such as ElastiCache (Redis) and enforce a token-bucket algorithm in a Lambda authorizer or edge function.

Can API Gateway cache responses?

Yes. API Gateway offers built-in response caching for a stage, and CloudFront edge caches in front of it add a second layer, reducing backend load for GET requests.

What monitoring should be in place for an API gateway?

Collect request counts, latency percentiles, and error codes in CloudWatch, enable X-Ray tracing for end-to-end visibility, and set alarms on 99th-percentile latency and error spikes.