Disaster Recovery in System Design Interviews: When to Use It and What to Say
When an interview asks you to design a high-traffic service, the next logical question is: What happens if the primary data center goes down? Disaster recovery (DR) is the set of mechanisms you put in place so that, after a catastrophic event, the system can be brought back to a usable state within an acceptable time window. In one sentence, disaster recovery is the strategy for restoring a service and its data after a total loss of the primary site.
Why Disaster Recovery Matters
Even the most robust architecture can be knocked offline by a power outage, a network partition, a region-wide cloud incident, or an operator error. Without a DR plan, the outage can turn into a data loss event, eroding user trust and revenue. In a concrete example – imagine a social media feed service that stores user posts in a primary MySQL cluster in us-east-1. If that region suffers a prolonged outage, users in all other regions will see a blank timeline, and any new posts written during the outage could be lost. A DR design lets you keep a copy of the data elsewhere and switch traffic over quickly, limiting the impact.
Step-by-Step DR Walkthrough
Below is a minimal but complete DR flow for a key-value store that backs a shopping cart service.
- Primary data store – A MySQL cluster in region A stores cart rows.
- Write path – The application writes to the primary and also publishes the change to a change-data-capture (CDC) stream.
- Replication – A replica MySQL instance in region B consumes the CDC stream and applies the same changes in near-real time.
- Health check – A watchdog monitors latency and error rates of the primary. If latency spikes or the primary stops responding for more than a configurable threshold (e.g., 30 seconds), the watchdog marks the primary as failed.
- Failover – The DNS record for the service’s endpoint is switched to point at the replica in region B. The application logic detects the DNS change and starts reading/writing to the replica.
- Recovery – Once the primary region is restored, data is back-filled from the replica to the primary, and DNS is switched back.
Two key metrics define the quality of this DR plan:
- RPO (Recovery Point Objective) – How much data loss is acceptable. In the example, the CDC stream latency of a few seconds gives an RPO of roughly a few seconds.
- RTO (Recovery Time Objective) – How quickly the service must be back online. DNS propagation plus application warm-up usually yields an RTO of a few minutes.
Where DR Shows Up in Classic Interview Prompts
| Interview Question | Typical DR Touchpoint |
|---|---|
| Design a Twitter timeline service | Replicating tweet storage across regions; failover of the read-through cache (e.g., Redis) when a region goes down |
| Build a payment processing system | Synchronous write to a primary ledger DB with asynchronous replication to a disaster-recovery ledger; switch to the DR ledger if the primary ledger is unavailable |
| Create a video streaming platform | Storing video metadata in a primary object store (e.g., S3) and replicating to a secondary bucket in another region; edge CDN fallback when the origin bucket fails |
In each case, the DR component appears at the point where the system’s state lives – the database, the ledger, or the object store. The interviewer expects you to mention how you keep that state safe and how you route traffic during a failure.
Trade-offs and the Alternative Interviewer May Probe
| Dimension | Hot standby (active-active) | Warm standby (active-passive) |
|---|---|---|
| Cost | High – you pay for duplicate compute and storage in both regions. | Lower – secondary resources are idle until failover. |
| RTO | Seconds to minutes (traffic switches instantly). | Minutes to tens of minutes (needs provisioning and sync). |
| RPO | Near-zero – writes are replicated synchronously. | Seconds to minutes – replication is asynchronous. |
| Complexity | High – you must handle split-brain and data consistency across regions. | Moderate – simpler failover logic, but you must manage data catch-up. |
Interviewers often ask you to compare these two approaches. The alternative they may probe is multi-AZ redundancy versus true cross-region DR. Multi-AZ protects against the loss of a data center (one availability zone) and fails over automatically, usually within a minute or two for a database, but it does not survive a region-wide outage. If you claim cross-region DR without explaining the added latency, cost, or data-sync challenges, the interviewer will push back.
A 30-Second Pitch You Can Say Out Loud
"Disaster recovery is the plan that lets us restore service after a whole data center or region disappears. I would keep a hot replica of the primary store in another region, stream changes in near real time, and use a health-check watchdog to trigger a DNS-based failover. The RPO would be a few seconds and the RTO a couple of minutes. The trade-off is higher cost and added complexity compared to a warm standby, but it guarantees minimal data loss and quick recovery."
Common Mistakes Candidates Make
- Ignoring RPO/RTO – Simply saying "we’ll have a backup" without quantifying how much data could be lost or how long the outage will last shows a lack of depth.
- Over-promising synchronous replication – Synchronous cross-region writes add latency that can break SLAs; interviewers expect you to discuss the latency-vs-durability trade-off.
- Forgetting the application layer – DR is not just about databases; caches, message queues, and DNS need explicit failover strategies.
- Skipping cost discussion – Hot standby doubles infrastructure spend. Not mentioning cost signals that you haven’t considered real-world constraints.
- Treating DR as a one-time step – DR is an ongoing process: you need health monitoring, automated failover, and periodic disaster drills. Treating it as a static diagram loses credibility.
How to Practice This
- Pick a classic interview prompt (e.g., design a Twitter timeline) and sketch the full architecture on a whiteboard.
- Add a disaster-recovery layer: identify the primary state store, choose hot or warm standby, and write out the failover steps.
- Use System Design AI to draw the answer on a canvas and get AI feedback on missing components or unclear RPO/RTO numbers.
Questions people ask
What is the difference between disaster recovery and high availability?
Disaster recovery is about restoring a service after a whole site or region is lost, with defined RPO and RTO. High availability keeps the service running despite component failures and usually operates within a single region using multiple zones.
Do I need to replicate every component for DR?
Only the components that hold durable state—databases, object stores, and sometimes message queues—need replication. Stateless services and caches can be rebuilt from code and configuration when you fail over.
How often should I test my DR plan?
A realistic practice is a quarterly disaster-drill that forces a failover, verifies data freshness, checks DNS propagation, and confirms the application can run from the standby without manual steps.
Can I use a single region with multiple availability zones instead of cross-region DR?
Multi-AZ gives automatic failover, usually within a minute or two for a database, when one data center (an availability zone) is lost but does not protect against a region-wide outage. If the interview scenario includes regional failures, you need a cross-region standby.