System Design AI

Design a Metrics and Monitoring System: The Junior, Senior and Staff Answer

When you’re asked to design a metrics and monitoring system, the interviewer is looking for three things: your grasp of high-throughput data pipelines, your ability to choose the right storage primitives for time-series data, and how you think about observability of the system you are building. The question is a proxy for real-world services like CloudWatch or Datadog, so they expect you to talk about ingestion, storage, aggregation, and alerting.

Requirements and Driving Numbers

Functional

  • Collect arbitrary numeric metrics from thousands of services.
  • Support real-time dashboards that query recent data (last few minutes) and historical queries (days to months).
  • Provide alerting based on thresholds or anomaly detection.
  • Expose a simple API for push-based ingestion and a pull-based query endpoint.

Non-functional

  • Ingestion rate: about 100 million metric points per day, which is roughly 1 200 writes per second.
  • Query rate: around 10 k dashboard queries per second, with most queries hitting the last few minutes of data.
  • Latency: ingestion to query visibility under a few seconds for recent data, minutes for historical.
  • Durability: keep raw data for at least 30 days, aggregated data for a year.

These numbers drive the choice of services. A write rate of 1 200 /sec is well within DynamoDB’s provisioned capacity, but you need a buffering layer to smooth spikes. Query traffic of 10 k /sec pushes you toward a cache for recent data and a read-optimized store for aggregates.

Core Design Walkthrough

1. Ingestion Path (one hop at a time)

  1. Client SDK – services push metric points via HTTPS POST to an API Gateway endpoint.
  2. API Gateway validates the payload and forwards it to an AWS Lambda function.
  3. Lambda writes the raw point to an Amazon Kinesis Data Stream (or MSK Kafka) for durability and fan-out.
  4. Kinesis feeds two parallel consumers:
  • S3 Writer – batches points and stores them in S3 as immutable objects (e.g., Parquet). This satisfies the raw-data durability requirement.
  • Time-Series Processor – aggregates points into minute-level buckets and writes them to DynamoDB (partition key = metric name, sort key = timestamp bucket).

2. Query Path

  1. API Gateway receives a GET request for a metric range.
  2. Lambda first checks Amazon ElastiCache (Redis) for a recent aggregation. If a cache miss, it queries DynamoDB for the required buckets.
  3. Results are merged, formatted, and returned to the client.

3. Alerting

  • A CloudWatch Event triggers a Lambda that scans recent aggregates in DynamoDB.
  • When a threshold is crossed, the Lambda publishes to an SNS topic, which fans out to email, Slack, or PagerDuty.

4. Data Model

  • Raw storage: S3 objects keyed by date/hour for cheap, long-term retention.
  • Aggregated storage: DynamoDB items with attributes metric_name, bucket_start, count, sum, min, max.
  • Cache key: metric_name:bucket_start → serialized aggregate.

5. Key Choice

Choosing DynamoDB over Aurora for the aggregated store is driven by the need for predictable low-latency reads at scale and the ability to scale write throughput linearly. Aurora could handle the volume but would require more complex sharding and would be costlier for the same read latency.

What a Complete Junior Answer Covers

  • Lists the components (API Gateway, Lambda, Kinesis, S3, DynamoDB, ElastiCache, SNS).
  • Shows the request path from client to storage and back.
  • Mentions basic durability (S3) and latency goals.
  • Provides a simple diagram of the pipeline.
  • Touches on alerting but does not dive into threshold evaluation.

What a Senior Answer Adds

  • Details the DynamoDB table schema, explains the choice of partition and sort keys.
  • Discusses write-capacity planning: 1 200 writes/sec → provision ~2 000 WCUs to handle spikes.
  • Explains the use of Kinesis as a buffer to decouple ingestion from downstream processing.
  • Adds a second consumer that writes to Amazon OpenSearch Service for ad-hoc analytics.
  • Describes cache eviction policy (TTL of a few minutes) and why recent data is cached.
  • Mentions eventual consistency of DynamoDB reads and how the system tolerates it.
  • Provides a brief failure-mode for a single component (e.g., Lambda throttling) and a fallback (direct S3 write).

What a Staff Answer Adds

  • Failure Modes: enumerates cascading failures (Kinesis backlog, DynamoDB hot partitions, ElastiCache eviction storms) and mitigation strategies (sharding metric names, auto-scaling Lambda, using SQS as a dead-letter buffer).
  • Operations: describes monitoring of the pipeline itself (Kinesis PutRecord latency, Lambda error rates, DynamoDB throttles) using CloudWatch dashboards.
  • Cost Trade-offs: compares DynamoDB on-demand vs. provisioned, S3 storage tiering, and the impact of cache size on EC2 vs. managed Redis pricing.
  • Defensible Trade-off: argues for DynamoDB over Aurora because the write pattern is write-heavy with small items, and DynamoDB’s predictable latency outweighs Aurora’s richer query language.
  • Security: notes IAM role isolation, VPC endpoints for private connectivity, and encryption at rest for S3 and DynamoDB.
  • Scalability: projects growth to 1 billion points per day (≈12 000 writes/sec) and shows how to add more Kinesis shards and increase DynamoDB WCUs accordingly.

Common Mistakes That Sink This Question

LevelTypical Mistake
JuniorLists components without showing the flow; forgets durability or latency goals.
SeniorOver-engineers the storage layer (e.g., builds a custom time-series DB) instead of using managed services; ignores cache for recent queries.
StaffFails to discuss failure isolation; treats cost as an after-thought; does not address operational monitoring of the monitoring system itself.

How to Practice This

  1. Study the worked design for a metrics and monitoring system in the System Design AI library. Observe how the diagram builds the request path step by step.
  2. Write your own answer on a canvas, starting with the ingestion path, then the query path, and finally alerting. Use the rubric to check if you covered junior, senior, and staff expectations.
  3. Explain the design aloud to a peer, or ask the AI coach to review your canvas, then look for missing failure modes or cost considerations and iterate on your diagram.

Questions people ask

What core AWS services are best for a high-throughput metrics pipeline?

API Gateway for the entry point, Lambda for lightweight processing, Kinesis (or MSK) as a buffer, S3 for raw immutable storage, DynamoDB for aggregated time-series data, and ElastiCache for low-latency recent queries.

How do you model time-series data in DynamoDB?

Use the metric name as the partition key and a timestamp bucket (e.g., minute granularity) as the sort key. Store aggregates like count, sum, min, and max in the item attributes.

Why choose DynamoDB over Aurora for storing aggregates?

DynamoDB offers predictable single-digit millisecond latency, linear write scaling, and simple key-value access, which matches the write-heavy, small-item pattern of metric aggregates better than Aurora’s relational model.

What are typical failure modes to watch for in this design?

Backlog in Kinesis, hot partitions in DynamoDB, cache eviction storms, and Lambda throttling. Mitigations include sharding metric names, auto-scaling, and using SQS as a dead-letter buffer.