System Design AI

Write-ahead logs in System Design Interviews: When to Use It and What to Say

A write-ahead log (WAL) is a sequential log that records every state-changing operation before the operation is applied to the main data store. The log is persisted first, then the system updates its in-memory or on-disk structures. By doing so, the system can recover to a consistent point after a crash by replaying the log.

The problem WAL solves

Imagine you are building a key-value store that must survive power loss. Without a WAL, the store might write a value to an in-memory hash map, then flush the map to disk later. If the machine crashes after the in-memory write but before the disk flush, the update is lost. The system would have no record of the intent, and consistency is broken.

A concrete example is a simple order-processing service. An order arrives, the service deducts inventory, and writes a confirmation to a database. If the service crashes after the inventory is deducted but before the confirmation is persisted, the inventory count becomes wrong and the customer never gets a receipt. A WAL placed in front of the inventory database guarantees that the deduction intent is durable, so the service can finish the confirmation after a restart.

How a write-ahead log works – step by step

  1. Append entry – The client sends a request (e.g., UPDATE inventory SET qty = qty - 1 WHERE id=42). The front-end creates a log entry containing the operation and a unique sequence number, then appends it to the end of the log file.
  2. Flush to durable storage – The log entry is forced to storage (fsync or equivalent). Only after the flush succeeds does the system consider the operation durable.
  3. Apply to state – The service reads the freshly flushed entry and applies the mutation to the primary data store (in-memory cache, B-tree, etc.).
  4. Acknowledge client – Once the mutation is applied, the service returns success to the client.
  5. Checkpoint / compaction – Periodically the system takes a snapshot of the current state and truncates the log up to the checkpoint, reducing log size.

Small worked example

StepActionLog contentData store state
1Client requests balance = balance + 10seq=101, op=+10balance = 100
2Flush log entry to SSDpersisted seq=101balance = 100
3Apply operation–balance = 110
4Ack client–balance = 110
5Checkpoint after 5 opstruncate log up to seq=105snapshot saved

If the node crashes after step 2, recovery reads seq=101 from the log and re-applies the +10, restoring the balance to 110.

Classic interview questions where WAL appears

Design questionWhere WAL is relevant
Primary-replica relational DBIn the primary’s commit path: log entry → flush → apply → replicate to replicas
Distributed message queue (e.g., Kafka-style)Each broker writes incoming messages to a WAL before making them available to consumers
Change Data Capture / Event sourcingThe event store itself is a WAL; every state change is an immutable event that can be replayed

In each of these designs, the interviewer expects you to mention the log as the durability backbone. You can say, "We write the request to a WAL on the primary, flush it, then apply the mutation and replicate the log entry to followers."

Trade-offs and alternative mechanisms

AspectWrite-ahead logAlternative (e.g., shadow paging)
Write latencyExtra fsync adds ~1-5 ms per write (depends on storage)Shadow copy can batch writes, reducing per-write latency but increases memory pressure
Storage overheadLog grows linearly with write volume; needs periodic compactionShadow paging keeps old and new versions of changed pages until commit, which costs space and scatters data on disk
ComplexitySimple append-only logic; recovery is deterministicShadow paging needs copy-on-write page management and an atomic root-pointer switch, and is hard to combine with many concurrent transactions
Recovery speedReplay log up to last checkpoint; fast if checkpoint frequentRe-building state from snapshots may be slower if snapshots are stale

Interviewers often ask you to compare WAL to these alternatives. Emphasize that WAL trades a little extra write latency for deterministic crash recovery and simpler coordination.

A 30-second pitch you can say out loud

"A write-ahead log is an append-only, durable record of every state-changing operation. You write the intent to the log, flush it, then apply the change to the data store. If the node crashes, you replay the log to recover. It gives you strong durability with modest latency overhead and is the backbone of databases, message queues, and event-sourced systems. The main trade-off is extra write latency and log storage, which you mitigate with periodic checkpoints."

Common mistakes candidates make

  • Skipping the flush step – Saying the log is written but not persisted gives a false sense of durability. Interviewers look for the explicit fsync or equivalent.
  • Confusing WAL with replication – Replication ships changes, often the log records themselves, to other nodes for availability; the WAL makes a change durable on one node before it is applied. Treating one as a substitute for the other shows a gap in understanding the durability chain.
  • Omitting checkpoint / compaction – Without a strategy to truncate the log, the system will run out of disk space. Mentioning periodic snapshots demonstrates completeness.
  • Over-optimizing latency – Claiming WAL adds no latency ignores the cost of durable writes. A realistic answer acknowledges the trade-off and possible mitigations (e.g., group-commit).

How to practice this

  1. Study the worked design in the System Design AI library that walks through a primary-replica database using a WAL. Observe how each component is annotated.
  2. Draw your own diagram on the canvas: start with a client request, add the WAL component, show the flush, apply, and replication steps.
  3. Explain the flow aloud using the 30-second pitch, then answer follow-up questions about checkpoint frequency and alternative durability mechanisms.

Questions people ask

When is a write-ahead log necessary?

When the system must guarantee that no committed operation is lost after a crash, and the cost of a small extra write latency is acceptable.

Can I replace a WAL with just a database transaction?

Transactions often use a WAL internally, but if you design the system yourself you still need a durable log to survive crashes before the transaction commits.

How often should checkpoints be taken?

It depends on write volume and recovery time goals; a common heuristic is to checkpoint every few minutes or after a certain number of log entries, balancing log size against replay cost.

What does the log contain besides the operation?

Typically a sequence number, timestamp, and enough metadata (e.g., transaction ID) to order and idempotently replay the entry during recovery.