System Design AI

Design a Logging Pipeline: Junior, Senior, and Staff Answers

When an interviewer asks you to design a logging pipeline, they are probing several core competencies. They want to see that you can think about high-throughput data ingestion, durability, queryability, and operational observability. They also watch how you break a big problem into small, testable steps and whether you can discuss trade-offs without getting lost in vendor hype.

What the Interview Tests

  • Scalability – Can you handle a stream that grows to billions of log events per day?
  • Reliability – How do you guarantee that no log is lost, even if a component crashes?
  • Latency vs. Cost – Do you understand the difference between near-real-time dashboards and archival storage?
  • Operational Insight – Do you know which metrics to expose and how to alert on them?
  • Design Communication – Are you able to walk through a request path one hop at a time?

Core Requirements and Driving Numbers

RequirementTypical BallparkWhy it matters
Ingest rate200 k–500 k events/second (≈ 17–43 B per day)Sets the throughput for the front-end and the queue layer.
Retention30 days of searchable logs, 90 days of raw archivesDetermines storage tiering and cost.
Query latency< 5 s for recent logs, < 30 s for older logsDrives choice of indexing vs. batch processing.
Availability99.9 % for ingestion, 99.99 % for retrievalInfluences replication and failover strategy.

These numbers are not exact; they give you a sense of order-of-magnitude. If the ingest rate were 1 M events/second, you would double the number of partitions and likely add a streaming service like Kafka on MSK.

Core Design Walkthrough

Below is a step-by-step request path, each hop described as you would sketch on a whiteboard.

  1. Application → API Gateway – The client sends a JSON log line to a public endpoint. API Gateway validates the schema and throttles bursts.
  2. API Gateway → Kinesis Data Streams – Validated logs are put onto a sharded stream. Sharding is based on a hash of the source identifier, ensuring even distribution.
  3. Kinesis → Lambda (or Kinesis Data Firehose) – A consumer reads the stream in micro-batches (e.g., 5 s windows). The function does light enrichment (timestamp normalization, adding a UUID) and writes two outputs:
  • Hot store – Writes to DynamoDB for recent-log lookup (indexed by source and timestamp).
  • Cold store – Writes the raw batch to S3 with a partition layout s3://logs/yyyy/mm/dd/.
  1. DynamoDB → Elasticsearch (or OpenSearch) – An asynchronous Lambda streams new items from DynamoDB to an OpenSearch domain for full-text search and dashboarding.
  2. S3 → Glacier (optional) – After 30 days, a lifecycle rule moves objects to Glacier for cheap long-term retention.

Data Model

  • LogRecord: { id: UUID, source: string, timestamp: ISO8601, level: string, message: string, metadata: map }
  • In DynamoDB the primary key is (source, timestamp). A GSI on level enables quick filtering for recent alerts.
  • In S3 the object key mirrors the same partitioning, making replay easy.

Key Choice

We chose Kinesis over SQS because Kinesis provides ordered shards and built-in scaling for high write rates. SQS would require additional workers to preserve ordering, adding latency.

What a Complete Junior Answer Covers

AspectExpected Junior Coverage
High-level diagramSimple three-box flow: App → Queue → Storage
Ingest rateRough estimate (e.g., 100 k/s) and justification
DurabilityMention replication (e.g., S3 multi-AZ)
Basic data modelOne table with primary key, no secondary indexes
MonitoringBasic CloudWatch metrics for queue depth
Mistakes to avoidForgetting back-pressure, ignoring burst traffic

A junior answer should be coherent, show the main components, and explain why each piece is needed. It does not need to dive into sharding strategies or cost breakdowns.

What a Senior Answer Adds

AspectSenior Enhancements
ShardingExplain Kinesis shard calculation: required_shards = ceil(ingest_rate / 1 k per shard)
Back-pressureUse API Gateway throttling and Kinesis put-record limits to protect downstream Lambda
IndexingAdd a GSI on level in DynamoDB, and a separate OpenSearch domain for full-text search
LatencyShow how micro-batching (5 s) balances cost and near-real-time needs
ObservabilityDetail CloudWatch dashboards: iterator age, Lambda error rate, DynamoDB throttles
Cost awarenessRough cost estimate: Kinesis shards (~$0.015 per shard-hour), DynamoDB reads/writes, S3 storage

A senior candidate also discusses trade-offs: more shards reduce hot-spot risk but increase cost; using Lambda vs. a dedicated consumer fleet affects cold start latency.

What a Staff Answer Adds

AspectStaff-level Depth
Failure modesDiscuss shard loss, Lambda timeout, DynamoDB throttling, and late or partial S3 batch writes. Provide fallback paths (e.g., dead-letter queue, replay from Kinesis).
OperationsExplain automated alerts, runbooks for “iterator age > 30 s”, and periodic health checks.
Cost optimizationShow tiered storage: hot logs in DynamoDB for 7 days, then S3 Standard-IA, then Glacier. Include a simple cost model: cost ≈ (shards * hours * $0.015) + (storage_gb * $0.023).
Trade-off defenseArgue why DynamoDB is chosen for low-latency lookups instead of a single Elasticsearch index, citing write amplification and hot-spot avoidance.
Scaling beyond 1 M/sPropose moving ingestion to Kafka on MSK, using multiple Kinesis streams, or a hybrid approach with SQS for burst buffering.
SecurityMention IAM role segregation, KMS-encrypted S3 buckets, and VPC endpoints for private traffic.

Staff candidates also anticipate operational concerns: data skew, schema evolution, and compliance (e.g., GDPR retention). They can sketch a runbook for a “log pipeline outage” scenario.

Common Mistakes That Sink This Question

  • Ignoring burst traffic – Assuming a flat 200 k/s rate and forgetting that spikes can be 5× higher.
  • Over-engineering storage – Jumping straight to a data warehouse for raw logs, which adds unnecessary latency.
  • No back-pressure – Forgetting to throttle at the API layer, leading to Lambda throttling and dropped records.
  • Missing durability guarantees – Not mentioning multi-AZ replication or dead-letter queues.
  • Skipping cost discussion – Interviewers expect you to be aware that Kinesis shards and DynamoDB reads add up quickly.

Comparison Table: Junior vs. Senior vs. Staff

LevelCore PathData ModelScaling DetailOps & Cost
JuniorApp → Queue → S3Single table, primary key onlyRough shard count, no explicit back-pressureBasic CloudWatch metrics
SeniorApp → API GW → Kinesis → Lambda → DynamoDB + S3 → OpenSearchPrimary key + GSI on levelCalculated shards, micro-batching, throttlingCost estimate, dashboards, alerting
StaffAll senior components + dead-letter SQS, replay, Glacier lifecycle, IAM/KMS, runbooksSame model + versioningMulti-stream or Kafka for >1 M/s, hot-spot mitigationFull cost model, tiered storage, failure-mode analysis

How to Practice This

  1. Study the worked design in the System Design AI library. Observe how each hop is annotated with notes on what to say and what can go wrong.
  2. Draw your own pipeline on the canvas. Start with the high-level flow, then add sharding, back-pressure, and monitoring details.
  3. Run a mock interview: explain the design aloud, then ask the AI coach to review your canvas and give feedback on missing failure modes or cost considerations.

Questions people ask

What ingest rate should I assume for a logging pipeline interview?

A typical large-scale service generates on the order of a few hundred thousand events per second, which is tens of billions of events per day. Use that as a baseline and explain how you would add shards if the rate grew.

Why not store logs directly in Elasticsearch?

Direct writes can overwhelm the index and increase latency. A two-tier approach—DynamoDB for recent queries and S3 for bulk storage—keeps write latency low and reduces cost.

How do I handle spikes that exceed my Kinesis shard capacity?

Enable API Gateway throttling, add a burst buffer such as SQS, and configure Lambda retries from a dead-letter queue. This provides back-pressure while preserving all logs.

What is a good retention strategy?

Keep the most recent 7 days in DynamoDB for fast lookups, move raw logs to S3 Standard-IA for the next 23 days, then transition older objects to Glacier for cheap long-term storage.