System Design AI

Distributed Locks: Follow-Up Questions Interviewers Ask, With Answers

Interviewers bring up distributed locks because they sit at the intersection of consistency, availability, and performance. A candidate who can name a lock algorithm often hasn’t explored the edge cases that matter when the system is under load or when nodes fail. Probing further lets the interviewer see whether you understand the trade-offs and can reason about operational risk.

The follow-up questions also reveal how you think about alternatives. If you can discuss why a lock may be overkill, or how you would fall back to an optimistic approach, it shows you can choose the right tool for the job, not just the most popular one.

How does a distributed lock guarantee mutual exclusion?

I would say that a lock guarantees mutual exclusion by ensuring that only one client can hold the lock token at any time. In practice this is achieved by a consensus protocol such as Paxos or Raft, or by using a primitive like a compare-and-set (CAS) on a central store (e.g., a Redis SETNX). The store atomically records the lock owner and a TTL. If two clients attempt to acquire the lock simultaneously, the CAS operation lets only one succeed, while the other receives a failure response and must retry.

What latency penalty does acquiring a lock add?

Acquiring a lock typically adds a round-trip to the lock service plus any quorum latency if the service uses replication. For a three-node quorum, the worst-case latency is roughly the sum of two network RTTs. In a datacenter with 0.5 ms RTT, that’s about 1 ms per acquire. If you also need to write a lease renewal every few seconds, the steady-state overhead is on the order of a few hundred microseconds per request, which can become noticeable at high QPS.

How do you handle lock expiration and “dead client” scenarios?

I set a short TTL—often 5–10 seconds—when I acquire the lock, and I require the client to renew the lease before it expires. If the client crashes, the lock automatically releases when the TTL lapses, preventing a permanent deadlock. To avoid premature expiration during a long critical section, the client can extend the lease in a separate thread, but the extension must be idempotent and guarded by the same lock token to prevent a split-brain situation.

What consistency model does the lock service need to provide?

The lock service must provide linearizable writes for the lock key. Linearizability ensures that once a client sees the lock as free and writes its token, no other client can later claim the same token without first seeing the lock as held. This is why systems like etcd or Consul, which implement Raft, are preferred over eventually consistent stores for lock semantics.

How would you avoid the “thundering herd” problem when many clients retry a lock?

I would add exponential back-off with jitter to the retry loop. For example, after a failed acquire, the client waits a random time between 10 ms and 20 ms, then doubles the range on each subsequent failure up to a maximum (say 1 second). This spreads retries across time, reducing contention on the lock service and preventing a spike of requests that could overwhelm it.

Can you implement a lock without a dedicated lock service?

Yes. One common pattern is to use a database row with a unique constraint. A client attempts to insert a row with a unique lock identifier; if the insert succeeds, it owns the lock. The row includes a timestamp column that the client updates as a heartbeat. If the timestamp becomes stale—say older than 8 seconds—the lock is considered abandoned and can be reclaimed. This approach leverages the database’s built-in transactional guarantees.

How do you decide between a pessimistic lock and an optimistic concurrency control?

I compare the expected contention rate. If the critical section is short and conflicts are rare, optimistic control—using version numbers or CAS—usually yields higher throughput because it avoids the extra network hop for lock acquisition. However, if the operation touches many shared resources or the contention probability exceeds roughly 10 %, a pessimistic lock reduces retries and latency spikes. In practice I run a small load test: if the retry rate is under 5 % at target QPS, I stay optimistic; otherwise I switch to a lock.

What are the trade-offs of using a lock service that runs on a cloud provider’s managed offering (e.g., AWS DynamoDB with conditional writes)?

Managed services remove the operational burden of running a quorum protocol, but you lose some control over latency tails and failure modes. DynamoDB’s conditional write latency is typically a few ms, but a hot lock key can be throttled and calls can fail during an outage, and your client must handle both. Also, the cost model is request-based, so a high lock-acquire rate can become expensive. On the other hand, you gain durability and automatic scaling, which simplifies capacity planning.

How would you evolve the design for a system that must handle 100 k lock acquisitions per second?

At that scale I would shard the lock namespace across multiple lock servers to avoid a single hotspot. Each shard would own a disjoint subset of lock keys, and clients would hash the lock name to pick the shard. I’d also move the lock acquisition path into the data plane: for example, let a holder take a lease once and reuse it for many operations, and attach a fencing token (a number that grows with every grant) that the downstream store checks on every write, so a stale holder is rejected. This reduces the per-request round-trip to the lock service from ~2 ms to sub-millisecond while still preserving mutual exclusion.

What separates a senior answer from a staff answer on this topic?

A senior answer focuses on correctness: the algorithm, failure handling, and basic performance numbers. A staff answer goes further—talking about operational complexity, scaling strategies, and how the lock design fits into the larger system evolution. A staff candidate will also discuss monitoring (e.g., lock wait time histograms), alerting on abnormal lease expirations, and how to migrate from a simple lock service to a sharded, in-process design without breaking existing clients.

How to practice this

  1. Study the worked design on contention in the System Design AI library, where thousands of requests race for the same item. Observe how the diagram grows from the junior answer to the staff answer.
  2. Write your own answer on a canvas: list the lock algorithm, failure handling, and scaling plan in bullet form.
  3. Use the AI coach to review your canvas. Incorporate feedback on latency estimates and operational concerns, then iterate.

Questions people ask

When should I prefer a lease-based lock over a simple mutex?

Lease-based locks are better when clients may crash or become partitioned. The TTL automatically frees the lock, preventing permanent deadlocks, whereas a simple mutex requires explicit release.

Can I use Redis SETNX for a production-grade distributed lock?

It works for low-risk scenarios, but you must add a TTL, handle clock drift, and implement a safe renewal protocol. For strong consistency, a Raft-based store is safer.

How do I measure lock contention in a live system?

Instrument the lock service to emit metrics such as lock acquisition latency, number of retries, and lock hold time. Plotting these as histograms helps spot contention spikes.

What is the impact of network partition on a distributed lock?

If the lock service loses quorum, it must reject new lock requests. Existing holders continue until their lease expires, after which the lock becomes free. Clients should treat a partition as a failure and back off.