Design a Logging Pipeline: Junior, Senior, and Staff Answers
When an interviewer asks you to design a logging pipeline, they are probing several core competencies. They want to see that you can think about high-throughput data ingestion, durability, queryability, and operational observability. They also watch how you break a big problem into small, testable steps and whether you can discuss trade-offs without getting lost in vendor hype.
What the Interview Tests
- Scalability – Can you handle a stream that grows to billions of log events per day?
- Reliability – How do you guarantee that no log is lost, even if a component crashes?
- Latency vs. Cost – Do you understand the difference between near-real-time dashboards and archival storage?
- Operational Insight – Do you know which metrics to expose and how to alert on them?
- Design Communication – Are you able to walk through a request path one hop at a time?
Core Requirements and Driving Numbers
| Requirement | Typical Ballpark | Why it matters |
|---|---|---|
| Ingest rate | 200 k–500 k events/second (≈ 17–43 B per day) | Sets the throughput for the front-end and the queue layer. |
| Retention | 30 days of searchable logs, 90 days of raw archives | Determines storage tiering and cost. |
| Query latency | < 5 s for recent logs, < 30 s for older logs | Drives choice of indexing vs. batch processing. |
| Availability | 99.9 % for ingestion, 99.99 % for retrieval | Influences replication and failover strategy. |
These numbers are not exact; they give you a sense of order-of-magnitude. If the ingest rate were 1 M events/second, you would double the number of partitions and likely add a streaming service like Kafka on MSK.
Core Design Walkthrough
Below is a step-by-step request path, each hop described as you would sketch on a whiteboard.
- Application → API Gateway – The client sends a JSON log line to a public endpoint. API Gateway validates the schema and throttles bursts.
- API Gateway → Kinesis Data Streams – Validated logs are put onto a sharded stream. Sharding is based on a hash of the source identifier, ensuring even distribution.
- Kinesis → Lambda (or Kinesis Data Firehose) – A consumer reads the stream in micro-batches (e.g., 5 s windows). The function does light enrichment (timestamp normalization, adding a UUID) and writes two outputs:
- Hot store – Writes to DynamoDB for recent-log lookup (indexed by source and timestamp).
- Cold store – Writes the raw batch to S3 with a partition layout
s3://logs/yyyy/mm/dd/.
- DynamoDB → Elasticsearch (or OpenSearch) – An asynchronous Lambda streams new items from DynamoDB to an OpenSearch domain for full-text search and dashboarding.
- S3 → Glacier (optional) – After 30 days, a lifecycle rule moves objects to Glacier for cheap long-term retention.
Data Model
- LogRecord:
{ id: UUID, source: string, timestamp: ISO8601, level: string, message: string, metadata: map } - In DynamoDB the primary key is
(source, timestamp). A GSI onlevelenables quick filtering for recent alerts. - In S3 the object key mirrors the same partitioning, making replay easy.
Key Choice
We chose Kinesis over SQS because Kinesis provides ordered shards and built-in scaling for high write rates. SQS would require additional workers to preserve ordering, adding latency.
What a Complete Junior Answer Covers
| Aspect | Expected Junior Coverage |
|---|---|
| High-level diagram | Simple three-box flow: App → Queue → Storage |
| Ingest rate | Rough estimate (e.g., 100 k/s) and justification |
| Durability | Mention replication (e.g., S3 multi-AZ) |
| Basic data model | One table with primary key, no secondary indexes |
| Monitoring | Basic CloudWatch metrics for queue depth |
| Mistakes to avoid | Forgetting back-pressure, ignoring burst traffic |
A junior answer should be coherent, show the main components, and explain why each piece is needed. It does not need to dive into sharding strategies or cost breakdowns.
What a Senior Answer Adds
| Aspect | Senior Enhancements |
|---|---|
| Sharding | Explain Kinesis shard calculation: required_shards = ceil(ingest_rate / 1 k per shard) |
| Back-pressure | Use API Gateway throttling and Kinesis put-record limits to protect downstream Lambda |
| Indexing | Add a GSI on level in DynamoDB, and a separate OpenSearch domain for full-text search |
| Latency | Show how micro-batching (5 s) balances cost and near-real-time needs |
| Observability | Detail CloudWatch dashboards: iterator age, Lambda error rate, DynamoDB throttles |
| Cost awareness | Rough cost estimate: Kinesis shards (~$0.015 per shard-hour), DynamoDB reads/writes, S3 storage |
A senior candidate also discusses trade-offs: more shards reduce hot-spot risk but increase cost; using Lambda vs. a dedicated consumer fleet affects cold start latency.
What a Staff Answer Adds
| Aspect | Staff-level Depth |
|---|---|
| Failure modes | Discuss shard loss, Lambda timeout, DynamoDB throttling, and late or partial S3 batch writes. Provide fallback paths (e.g., dead-letter queue, replay from Kinesis). |
| Operations | Explain automated alerts, runbooks for “iterator age > 30 s”, and periodic health checks. |
| Cost optimization | Show tiered storage: hot logs in DynamoDB for 7 days, then S3 Standard-IA, then Glacier. Include a simple cost model: cost ≈ (shards * hours * $0.015) + (storage_gb * $0.023). |
| Trade-off defense | Argue why DynamoDB is chosen for low-latency lookups instead of a single Elasticsearch index, citing write amplification and hot-spot avoidance. |
| Scaling beyond 1 M/s | Propose moving ingestion to Kafka on MSK, using multiple Kinesis streams, or a hybrid approach with SQS for burst buffering. |
| Security | Mention IAM role segregation, KMS-encrypted S3 buckets, and VPC endpoints for private traffic. |
Staff candidates also anticipate operational concerns: data skew, schema evolution, and compliance (e.g., GDPR retention). They can sketch a runbook for a “log pipeline outage” scenario.
Common Mistakes That Sink This Question
- Ignoring burst traffic – Assuming a flat 200 k/s rate and forgetting that spikes can be 5× higher.
- Over-engineering storage – Jumping straight to a data warehouse for raw logs, which adds unnecessary latency.
- No back-pressure – Forgetting to throttle at the API layer, leading to Lambda throttling and dropped records.
- Missing durability guarantees – Not mentioning multi-AZ replication or dead-letter queues.
- Skipping cost discussion – Interviewers expect you to be aware that Kinesis shards and DynamoDB reads add up quickly.
Comparison Table: Junior vs. Senior vs. Staff
| Level | Core Path | Data Model | Scaling Detail | Ops & Cost |
|---|---|---|---|---|
| Junior | App → Queue → S3 | Single table, primary key only | Rough shard count, no explicit back-pressure | Basic CloudWatch metrics |
| Senior | App → API GW → Kinesis → Lambda → DynamoDB + S3 → OpenSearch | Primary key + GSI on level | Calculated shards, micro-batching, throttling | Cost estimate, dashboards, alerting |
| Staff | All senior components + dead-letter SQS, replay, Glacier lifecycle, IAM/KMS, runbooks | Same model + versioning | Multi-stream or Kafka for >1 M/s, hot-spot mitigation | Full cost model, tiered storage, failure-mode analysis |
How to Practice This
- Study the worked design in the System Design AI library. Observe how each hop is annotated with notes on what to say and what can go wrong.
- Draw your own pipeline on the canvas. Start with the high-level flow, then add sharding, back-pressure, and monitoring details.
- Run a mock interview: explain the design aloud, then ask the AI coach to review your canvas and give feedback on missing failure modes or cost considerations.
Questions people ask
What ingest rate should I assume for a logging pipeline interview?
A typical large-scale service generates on the order of a few hundred thousand events per second, which is tens of billions of events per day. Use that as a baseline and explain how you would add shards if the rate grew.
Why not store logs directly in Elasticsearch?
Direct writes can overwhelm the index and increase latency. A two-tier approach—DynamoDB for recent queries and S3 for bulk storage—keeps write latency low and reduces cost.
How do I handle spikes that exceed my Kinesis shard capacity?
Enable API Gateway throttling, add a burst buffer such as SQS, and configure Lambda retries from a dead-letter queue. This provides back-pressure while preserving all logs.
What is a good retention strategy?
Keep the most recent 7 days in DynamoDB for fast lookups, move raw logs to S3 Standard-IA for the next 23 days, then transition older objects to Glacier for cheap long-term storage.