System Design AI

Data Pipelines in System Design Interviews: When to Use Them

In a system design interview you often need to move data from where it is generated to where it is analyzed or acted upon. A data pipeline is the set of components that reliably transport, transform, and store that data. Think of it as a conveyor belt: raw items are dropped onto the belt, the belt moves them through workstations, and the finished product lands in a bin ready for the next step.

One-sentence definition

A data pipeline is a series of connected services that ingest, buffer, transform, and store data so that downstream consumers can query or act on it.

The problem it solves

Imagine you run a video-sharing platform. Every time a user watches a video you generate an event: user-id, video-id, timestamp, playback quality. You need these events for three purposes:

  1. Real-time recommendation updates (seconds-level latency).
  2. Daily business dashboards (hourly-level latency).
  3. Long-term analytics for product decisions (days-to-weeks latency).

If you tried to write each service directly to a database, you would face race conditions, scaling limits, and tangled code. A pipeline decouples producers from consumers, smooths spikes, and lets you apply different processing logic at each stage.

How it works – step by step

Below is a minimal pipeline that could satisfy the three needs above.

  1. Ingest – Clients push events to a streaming service (e.g., Amazon Kinesis Data Streams). The service shards the stream to handle parallel writes.
  2. Buffer – The stream retains records for a configurable retention window (default 24 h). This gives downstream workers time to pull data without losing bursts.
  3. Transform (real-time) – A serverless function (AWS Lambda) reads from the stream, enriches the event (adds user profile data), and writes the result to a fast key-value store (Amazon DynamoDB) that powers recommendations.
  4. Transform (batch) – A separate consumer reads the same stream, aggregates events into 5-minute windows, and writes aggregated rows to a data warehouse (Amazon Redshift) for dashboards.
  5. Store for long-term – An S3 bucket receives a copy of every raw event via Kinesis Firehose. S3 provides cheap, durable storage for analytics jobs that run later.
  6. Serve – Downstream services query DynamoDB for the latest recommendation, run SQL queries on Redshift for daily reports, or run Spark jobs on the S3 data for deeper analysis.

Each stage has clear responsibilities, can be scaled independently, and can be swapped out without breaking the whole system.

Classic interview questions where pipelines appear

Interview questionWhere the pipeline fits
Log aggregation platform (e.g., “Design a system like Splunk”)Ingest → Buffer → Transform → Store. The pipeline is the core path that moves log lines from agents to searchable storage.
Real-time analytics for a social feed (e.g., “Design Twitter’s timeline”)Ingest events → Buffer → Real-time transform → Serve from a fast store. The pipeline powers the live ranking engine.
Metrics collection for a SaaS product (e.g., “Design a usage-metrics system”)Ingest → Buffer → Batch transform → Store in a warehouse. The pipeline feeds dashboards and alerts.

In each case the pipeline appears right after the data-source layer and before the storage/serving layer.

Trade-offs and the alternative interviewers love to ask about

AspectPipeline advantagePossible downside
ScalabilityHorizontal scaling of each stage (shards, Lambda concurrency).More components mean more operational overhead.
LatencyReal-time stage can deliver sub-second results.Batch stages add delay; you must choose where to accept latency.
ReliabilityBuilt-in replay (stream retention) and idempotent processing.Mis-configured checkpoints can cause duplicate processing or data loss.
ComplexityDecouples producers and consumers, allowing independent evolution.Over-engineering for simple use-cases; a single DB write may be sufficient.

Interviewers often follow up with questions like:

  • What if the stream spikes to 10× the expected rate? Discuss auto-scaling shards and back-pressure handling.
  • Can you guarantee exactly-once processing? Explain idempotent writes and deduplication strategies.
  • What if you need both real-time and batch results from the same data? Show how two consumers can read the same stream in parallel.

The classic alternative is a direct write architecture: producers write straight to a database or a message queue that consumers poll. This reduces component count but sacrifices flexibility, makes scaling harder, and couples producers to downstream schema changes.

30-second spoken version

"A data pipeline moves data from its source through a series of stages—ingest, buffer, transform, store—so you can handle high volume, apply real-time or batch processing, and keep producers and consumers decoupled. In an interview, describe a concrete pipeline using services like Kinesis, Lambda, DynamoDB, and S3, then discuss latency vs consistency trade-offs and what you’d do if traffic spikes."

Common mistakes candidates make

  • Skipping the buffer layer – Jumping from stream to transform assumes the consumer can keep up, which is rarely true under bursty traffic.
  • Treating every stage as a black box – Interviewers expect you to name concrete components and explain why they fit (e.g., Lambda for stateless, event-driven processing).
  • Ignoring back-pressure – Not mentioning how you would throttle producers or increase shard count when the pipeline lags.
  • Over-loading a single stage – Packing ingestion, transformation, and storage into one monolithic service defeats the purpose of a pipeline.
  • Forgetting durability – Not persisting raw events for replay leads to data loss if a downstream consumer fails.

How to practice this

  1. Open the practice brief on aggregating events that arrive late or twice in the System Design AI library, and read its rubric before you draw anything.
  2. Pick a familiar product (e.g., a video-streaming app) and draft your own pipeline on the practice canvas, labeling ingest, buffer, transform, and store stages.
  3. Record a 30-second pitch of your design and compare it to the rubric notes in the library. Adjust for clarity and coverage of trade-offs.

Questions people ask

What is the difference between a stream and a queue?

A stream retains ordered records for a configurable window and lets multiple consumers read at their own pace; a queue delivers each message to a single consumer and removes it after receipt.

When should I use batch processing instead of real-time?

Use batch when the business can tolerate minutes-to-hours latency, the workload is heavy, or you need to join large datasets that are costly to compute in real time.

How do I guarantee exactly-once processing in a pipeline?

Combine idempotent writes, deduplication keys, and checkpointing. Include a unique event ID and have the consumer check a store like DynamoDB before writing.

Can a pipeline handle both structured and unstructured data?

Yes. Ingest can accept raw blobs; downstream transform stages can parse and enrich the data into structured tables while preserving the original blob for later analysis.