System Design AI

Design a Multi-tenant SaaS Backend: Junior, Senior, and Staff Answers

When an interviewer asks you to design a multi-tenant SaaS backend, they are looking for three things: how you reason about scale, how you isolate customers, and how you think about operations. They want to see whether you can turn vague business goals into concrete architecture choices and whether you understand the trade-offs between simplicity, cost, and reliability.

Core Requirements and Driving Numbers

RequirementTypical Ballpark (hedged)
Number of tenants10,000 – 100,000 (order of 10⁴-10⁵)
Requests per second1,200 – 12,000 RPS (≈100 M – 1 B/day)
Data per tenant10 GB – 100 GB total, growing over time

These three figures drive storage sizing, request routing capacity, and isolation strategy. If you assume 100 M daily requests, that is about 1,200 RPS (100 M ÷ 86,400 ≈ 1,158). A tenant count of 50 k means average per-tenant request rate of roughly 0.02 RPS, but hot tenants can spike to several hundred RPS, so you need a design that can handle both the aggregate load and per-tenant spikes.

Request-Path Walkthrough

  1. Client → API Gateway – The entry point is an Amazon API Gateway that terminates TLS, does basic throttling, and routes to a Lambda or ECS service based on the HTTP path.
  2. Auth → Cognito – The gateway forwards the Authorization header to Amazon Cognito for token validation. The token contains the tenant ID (or a user-to-tenant mapping is looked up in a small DynamoDB table).
  3. Routing → Service Layer – A thin Lambda (or a Fargate container) extracts the tenant ID and forwards the request to the appropriate service. For read-heavy paths, it checks an Amazon ElastiCache (Redis) entry keyed by tenantId:resourceId.
  4. Cache Miss → Data Store – If the cache misses, the service reads from Amazon Aurora (PostgreSQL) for relational data or DynamoDB for high-throughput key-value access. The query includes the tenant ID as a partition key, guaranteeing logical isolation.
  5. Write Path → Transaction Log – Writes go through the service layer, which persists to Aurora (using multi-tenant schemas) and publishes an event to an Amazon SQS queue. A background worker consumes the queue, updates secondary indexes, and pushes change events to Kafka on MSK for downstream analytics.
  6. Response → CDN – For static assets (reports, PDFs) the service writes to an S3 bucket. CloudFront distributes the objects globally, with cache-control headers that respect tenant-specific TTLs.

Data Model and Key Choice

  • Tenant Isolation – Use a single-tenant identifier as the partition key in DynamoDB (PK = tenantId). In Aurora, keep a tenant_id column on every table and enforce row-level security via PostgreSQL policies. This choice is preferred over separate databases per tenant because it simplifies provisioning and reduces operational overhead while still offering strong logical isolation.
  • Why Not Separate Schemas for Every Tenant? – Separate schemas become unwieldy beyond a few hundred tenants and make schema migrations painful. The single-table approach scales to millions of tenants with minimal admin effort.

What a Complete Junior Answer Looks Like

  • Scope – Lists functional requirements (CRUD, authentication) and non-functional ones (availability ≈ 99.9%, latency < 200 ms).
  • Diagram – Sketches a single API Gateway → Lambda → Aurora flow.
  • Data Model – Shows a simple users table with tenant_id foreign key.
  • Isolation – Mentions using a tenant ID column but does not discuss trade-offs.
  • Caching – May note a cache but leaves it as “optional”.
  • Failure Handling – Briefly says “retry on error”.
  • Operations – Touches on monitoring with CloudWatch but no deeper metrics.

What a Senior Answer Adds

  • Tenant Isolation Strategy – Explains why a shared Aurora cluster with row-level security and a DynamoDB table keyed by tenant ID is chosen.
  • Cache Layer – Adds ElastiCache with per-tenant eviction policies, reducing read latency.
  • Async Processing – Introduces SQS for write-behind indexing and Kafka on MSK for event-driven analytics.
  • Scalability – Discusses horizontal scaling of the Lambda layer via provisioned concurrency and autoscaling of Aurora read replicas.
  • Security – Details IAM roles per service, VPC isolation, and encryption at rest (KMS) and in transit (TLS).
  • Operational Metrics – Proposes dashboards for request latency, cache hit ratio, and queue depth.
  • Cost Awareness – Names the main cost buckets: Aurora instance or ACU hours, DynamoDB read and write capacity, and ElastiCache node hours, and says which of them grow with tenant count.

What a Staff Answer Looks Like

  • Failure Domains – Breaks the system into zones: API Gateway + Cognito (frontend), service layer (compute), data stores (Aurora/DynamoDB), and async pipelines (SQS/MSK). Shows how each zone can fail independently and how the design recovers (e.g., Lambda cold-start fallback, Aurora multi-AZ, DynamoDB global tables).
  • Operational Playbooks – Defines runbooks for cache eviction storms, database replica lag, and Kafka broker failures. Mentions using AWS CloudWatch Alarms, AWS X-Ray for tracing, and automated canary deployments via CodeDeploy.
  • Cost Optimization – Suggests Aurora Serverless v2 for unpredictable load, DynamoDB on-demand capacity (or provisioned capacity with auto-scaling for steady load), and spot instances for background workers.
  • Trade-off Defense – Defends the choice of a shared Aurora cluster over per-tenant databases by citing operational simplicity, backup consistency, and the ability to use a single read replica pool to serve hot tenants.
  • Security Depth – Adds tenant-level encryption keys stored in KMS, audit logging to CloudTrail, and a zero-trust network with VPC endpoints.
  • Monitoring & Alerting – Presents a table of key SLOs (99.9% availability, 95th-percentile latency < 150 ms) and corresponding metrics.
  • Future-Proofing – Discusses sharding the DynamoDB table by tenant ID hash prefix if tenant count grows beyond 100 k, and migrating hot tenants to dedicated Aurora instances.

Common Mistakes that Sink the Question

MistakeWhy It Fails
Ignoring tenant isolationLeads to data leakage and makes compliance impossible.
Over-engineering with a database per tenantAdds operational burden and hurts scalability.
No caching strategyIncreases read latency, especially under hot-tenant spikes.
Forgetting async pipelinesSynchronous writes cannot sustain high write throughput.
Skipping cost discussionShows lack of real-world awareness; interviewers expect trade-off analysis.
No failure handlingSystem appears brittle; interviewers look for graceful degradation.

Comparison of Junior, Senior, and Staff Answers

AspectJuniorSeniorStaff
IsolationTenant ID column notedRow-level security, DynamoDB partition keyDetailed security policies, KMS per-tenant keys
CachingOptional mentionElastiCache with per-tenant TTLCache eviction monitoring, fallback strategy
Async ProcessingNone or vagueSQS + Kafka for eventsFull runbooks, canary deploys, spot workers
ScalingAutoscale LambdaAurora read replicas, DynamoDB auto-scaleSharding strategy, Aurora Serverless v2
Ops & MonitoringBasic CloudWatch metricsDashboards, alerts, tracingRunbooks, X-Ray, automated remediation
CostNot addressedRough cost bucketsDetailed optimization, spot usage
Failure ModesSimple retryZone-level redundancyEnd-to-end failure domain analysis

How to practice this

  1. Sketch the request path on a blank canvas. Start with API Gateway and add each hop, labeling where the tenant ID flows.
  2. Write a brief rubric: list the core requirements, the three driving numbers, and the extra senior/staff items you must cover.
  3. Use System Design AI to draw your own answer and get a review of missing pieces or weak trade-off arguments.

Questions people ask

What is the simplest way to achieve tenant isolation in a relational database?

Use a tenant_id column on every table and enforce row-level security policies in PostgreSQL. This keeps all tenants in one cluster while preventing cross-tenant reads.

When should I choose DynamoDB over Aurora for a SaaS backend?

DynamoDB shines for high-throughput key-value access and predictable latency, especially for session data or feature flags. Aurora is better for complex queries, joins, and transactional consistency.

How do I handle a sudden spike from a hot tenant without affecting others?

Place a per-tenant cache in ElastiCache, use read replicas for Aurora, and route writes through a dedicated SQS queue that can be scaled independently. Autoscaling policies should be tenant-aware.

What operational metrics matter most for a multi-tenant SaaS service?

Track request latency (p95), cache hit ratio, queue depth, replica lag, and error rates per tenant. Set alerts on thresholds that would breach your SLOs.