Design a Payment System: Junior, Senior, and Staff Answers
When you’re asked to design a payment system, the interviewer is looking for three things: your ability to break a complex flow into simple steps, your awareness of the consistency-vs-latency trade-off, and how you think about operational risk. A payment system must move money reliably, protect against fraud, and stay online even when a component fails. Below we walk through the core request path, the data model, and the key architectural choices, then layer on what each interview level expects.
Core Request Path – One Hop at a Time
- API Gateway receives the payment request – The client (mobile app or web) posts a JSON payload containing
payerId,payeeId,amount,currency, and apaymentMethodToken. The gateway validates the schema and forwards the request to a stateless service. - Payment Service (microservice) – This service performs authentication, looks up the payer’s account, and checks the payment method. It writes a pre-authorization record to DynamoDB (strongly consistent read/write) and publishes an event to an SQS queue for downstream processing.
- Risk Service – A consumer of the SQS queue runs fraud checks (velocity, black-list, device fingerprint). If the check passes, it updates the pre-auth record with a
riskStatus = OK. If it fails, the record is markedrejectedand the flow ends. - Settlement Service – Another consumer reads the approved events, debits the payer’s balance in Aurora (transactional SQL), credits the payee, and writes a final settlement entry to DynamoDB. The service also pushes a notification to an SNS topic that downstream services (email, push) subscribe to.
- Cache Invalidation – After the database writes, the service invalidates any stale entries in ElastiCache (Redis) so that balance queries see the new amount immediately.
- Response – The payment service returns a success/failure code to the API Gateway, which forwards it to the client.
Each hop is isolated: the API layer is cheap and scales automatically; the payment service is stateless and can be replicated behind an Auto Scaling group; the queues decouple risk and settlement, allowing each to scale independently.
Data Model Overview
| Entity | Primary Store | Key Fields | Reason for Choice |
|---|---|---|---|
| Account | Aurora (PostgreSQL) | accountId (PK), balance, currency | Strong ACID guarantees for debits/credits. |
| Pre-Auth | DynamoDB | transactionId (PK), payerId, payeeId, amount, status | Fast look-ups, flexible schema for risk metadata. |
| Settlement Log | S3 (append only) | date/partition, transactionId, details | Cheap, immutable audit trail. |
| Cache | ElastiCache (Redis) | accountId → balance | Low-latency reads for balance displays. |
The key decision is to use Aurora for the source of truth for balances, because financial correctness demands serializable isolation. DynamoDB handles the high-velocity, semi-structured parts of the flow (pre-auth, risk flags) where eventual consistency is acceptable after the final settlement.
Numbers That Drive the Design
- Peak transaction rate: Roughly 10 k TPS (≈864 million per day, about 10 k per second). This drives the need for horizontally scalable queues and stateless services.
- Latency target: End-to-end response <200 ms for most consumer-facing payments. This pushes the cache layer and the use of DynamoDB’s fast reads.
- Durability: Financial records must survive the loss of an Availability Zone, and ideally of a whole region. Aurora's multi-AZ replication covers the first; a regional outage needs a cross-region replica (Aurora Global Database) and S3 Cross-Region Replication for the settlement log.
These ball-park figures guide the sizing of Auto Scaling groups, queue throughput, and the choice of provisioned vs. on-demand capacity.
What a Complete Junior Answer Covers
- High-level diagram showing API Gateway → Payment Service → Database.
- Basic data model with accounts and transactions stored in a single relational table.
- Simple flow: request validation, balance check, debit, credit, response.
- Mention of idempotency (client supplies a unique request ID stored in the DB to avoid double charges).
- No deep discussion of failure handling – a junior can note “retry on failure” without detailing back-off or dead-letter queues.
What a Senior Answer Adds
- Decoupling via SQS to isolate risk checks and settlement, allowing each to scale.
- Cache layer (ElastiCache) for low-latency balance reads.
- Event sourcing: using DynamoDB streams or Kafka on MSK to replay events for audit or recovery.
- Monitoring & alerts: CloudWatch metrics for queue depth, latency, and error rates; dashboards.
- Security: IAM roles, KMS-encrypted payment tokens, and VPC isolation for the database.
- Consistency choices: Explain why Aurora is used for balances (serializable) while DynamoDB can be eventually consistent after settlement.
What a Staff Answer Adds
- Failure modes: network partition between API Gateway and payment service, queue backlog overflow, Aurora replica lag, and cache stampede. Show how circuit breakers, bulkheads, and exponential back-off mitigate each.
- Operational playbook: runbooks for “stuck transaction” (search DynamoDB for
status = pending), “queue drain” (scale SQS consumers), and “aurora failover” (promote replica, verify read-only lag). - Cost analysis: rough cost breakdown – Aurora on-demand instances vs. provisioned capacity, DynamoDB read/write units for 10 k TPS, SQS standard queue pricing, and ElastiCache node count. Show the order of magnitude honestly: a sustained 10 k TPS is roughly 26 billion transactions a month, so the bill runs to thousands or tens of thousands of dollars per month, and the levers are provisioned capacity and how many writes each payment really needs.
- Trade-off defense: argue why strong consistency on balances outweighs the added latency, and why eventual consistency on risk metadata is acceptable. Defend the choice of SQS over Kafka when the team prefers managed durability and simpler ops.
- Scalability beyond the baseline: discuss sharding accounts by region, using Global Tables for multi-region reads, and adding a CDN (CloudFront) for static assets like payment UI.
Common Mistakes That Sink This Question
| Mistake | Why It Fails |
|---|---|
| Single monolithic DB for everything | Limits throughput, makes scaling risk checks impossible, and couples latency to balance updates. |
| No idempotency | Leads to double charges when retries happen; interviewers expect a requestId stored early. |
| Ignoring fraud checks | Shows lack of real-world awareness; risk service should be a separate, async component. |
| Over-engineering with micro-services | At junior level, adding unnecessary RPC layers adds latency and complexity without benefit. |
| Missing operational details | Staff interviewers look for runbooks, alerting, and cost awareness. Skipping them signals a design that cannot be run in production. |
How to Practice This
- Study the worked design in the System Design AI library. Observe how each component is annotated with what to say and what can go wrong.
- Draw your own canvas on the practice board. Replicate the request path hop by hop, then add the data model and a few failure scenarios.
- Run through the rubric: check that your junior, senior, and staff layers each contain the required depth – from basic flow to operational playbooks.
Questions people ask
What latency should a payment system aim for?
A typical consumer-facing payment flow targets under 200 ms end-to-end. This keeps the user experience smooth while still allowing backend consistency checks.
Why use DynamoDB for pre-authorization instead of Aurora?
Pre-auth records are high-velocity, semi-structured, and need fast writes. DynamoDB provides low-latency writes at scale, and the eventual consistency after settlement is acceptable because the final balance lives in Aurora.
How do you ensure idempotency in a payment API?
Require the client to send a unique requestId. Store this ID with the transaction status in DynamoDB or Aurora. Subsequent retries check the stored status and return the original result without re-executing the debit.
What is the role of SQS in this design?
SQS decouples the risk and settlement stages, smoothing spikes in traffic. It also provides built-in dead-letter queues for messages that repeatedly fail, giving a clear path for manual investigation.