Kafka in System Design Interviews: When to Use It and What to Say
Kafka is a distributed, partitioned commit log that stores streams of records in immutable order. In an interview you can say it "provides a durable, ordered, replayable event stream that decouples producers from consumers".
The Problem Kafka Solves
Many modern services need to ingest a high volume of events, guarantee they are not lost, and allow multiple downstream systems to process the same data at their own pace. A classic example is an e-commerce site that records every order, inventory change, and user click. Without a reliable backbone, the order service might block the inventory service, and a downstream analytics pipeline could miss events if it is temporarily down.
How Kafka Works: Step-by-Step
- Producers write to a topic – Each topic is split into partitions. The producer picks a partition based on a key (e.g., order ID) to preserve ordering for that key.
- Brokers append the record – The chosen broker writes the record to the end of the partition log and replicates it to follower brokers for durability.
- Offsets are assigned – Each record gets a sequential offset within its partition. Consumers use offsets to track progress.
- Consumers read from the log – A consumer group consists of one or more instances. Kafka assigns each partition to only one member of the group, ensuring each event is processed once per group.
- Retention policy – The log is retained based on time or size. Older records are eventually discarded, but they can be replayed as long as they are within the retention window.
Small Worked Example
Imagine a "click-stream" pipeline:
- Producer: Front-end JavaScript sends click events to a Kafka producer service.
- Topic:
clickswith 3 partitions, keyed by user ID. - Broker: Each partition lives on three brokers (replication factor 3). The leader broker for a partition writes the record and replicates to the two followers.
- Consumer Group A: Real-time analytics service with 2 instances. Partition 0 → instance A1, partition 1 → A2, partition 2 → A1 (round-robin).
- Consumer Group B: Offline batch job that reprocesses the entire day’s clicks for training a recommendation model. It starts from offset 0 each day, independent of Group A.
The diagram would show a single request path: click → producer → partition → brokers → consumer A1/A2 → analytics → consumer B → batch job.
Where Kafka Shows Up in Classic Design Questions
| Design Question | Where Kafka Appears | Reason for Using Kafka |
|---|---|---|
| Real-time analytics pipeline (e.g., page-view stats) | Between the event collector and the analytics service | Provides ordered, durable stream and fan-out to multiple analytics workers |
| Order processing system with inventory sync | Between order service and inventory service | Guarantees that inventory updates are not lost even if the order service crashes |
| Log aggregation / monitoring platform | Ingest logs from many servers into a central processing cluster | Handles high write throughput and allows multiple consumers (alerting, storage, UI) |
In each case, Kafka sits at the boundary where you need to decouple producers from multiple downstream consumers while preserving order for a given key.
Trade-offs and the Alternative Interviewer May Probe
| Aspect | Kafka | Typical Alternative (e.g., HTTP queue, RabbitMQ) |
|---|---|---|
| Durability | Records are replicated; loss is unlikely unless a majority of replicas fail. | Depends on implementation; often in-memory or single-node persistence. |
| Ordering | Guarantees order per partition/key. | May provide per-queue ordering but not across multiple queues. |
| Scalability | Horizontal scaling by adding partitions and brokers; can handle millions of messages per second. | Scaling may require sharding queues manually; throughput limits are lower. |
| Latency | Typically a few milliseconds to tens of ms; can increase with higher replication factor or large batch sizes. | Lower latency for single-hop HTTP, but lacks durability and replay. |
| Complexity | Requires managing brokers, topics, partitions, and consumer offsets. | Simpler to set up; fewer moving parts. |
Interviewers often ask: "What if we need sub-second latency for critical alerts?" You can answer that a direct HTTP push may be faster, but you can still use Kafka with a low batch size and a small replication factor, accepting a slightly higher risk of loss.
30-Second Pitch
"Kafka is a distributed log that lets producers write events to a durable, ordered stream, and lets multiple consumer groups read those events at their own pace. It solves the problem of reliable, replayable event ingestion for high-throughput systems like real-time analytics or order processing. The key ideas are partitions for ordering, replication for durability, and consumer offsets for independent progress. Trade-offs include higher latency than a direct HTTP call and operational complexity, but it scales to millions of messages per second and decouples services cleanly."
Common Mistakes Candidates Make
- Treating Kafka as a database – Kafka stores streams, not arbitrary queryable data. Mention that you would still need a downstream store for ad-hoc queries.
- Ignoring consumer groups – Claiming “Kafka guarantees exactly-once processing” without clarifying that exactly-once semantics require idempotent downstream writes and proper transaction configuration.
- Over-partitioning – Adding too many partitions can increase overhead and cause small-batch latency spikes. Explain that the number of partitions should match the expected parallelism.
- Skipping retention considerations – Assuming records are kept forever. Note that retention is time- or size-based, and you must size it for replay needs.
- Forgetting about ordering scope – Saying “Kafka preserves global order” – it only guarantees order within a partition.
How to Practice This
- Study the worked design in the System Design AI library. It is the design of a distributed log like Kafka itself, built up one request path at a time.
- Draw your own canvas for a new problem—e.g., a video-upload service that needs transcoding and notification. Place Kafka between the upload service and the transcoding workers.
- Use the AI coach to review your canvas. Ask it to critique your placement of Kafka, the number of partitions you chose, and how you handle consumer offsets.
Questions people ask
When should I bring up Kafka in a design interview?
Mention Kafka when the problem needs high-volume event ingestion, durable replay, or fan-out to multiple downstream services. It signals you understand decoupling and stream processing.
What is the difference between a Kafka topic and a queue?
A topic is an immutable log retained for a configurable period, allowing many consumer groups to read the same data independently. A queue usually holds a single copy of each message and deletes it after a consumer acknowledges it.
How does Kafka achieve durability?
Each partition is replicated across several brokers. The leader broker writes the record and forwards it to follower brokers. The write is considered committed once the required replication factor acknowledges it.
Can Kafka guarantee exactly-once processing?
Kafka provides at-least-once delivery by default. Exactly-once semantics are possible but require idempotent downstream writes and the use of Kafka transactions on the producer side.