Metrics & Monitoring System

A metrics and monitoring system collects numeric measurements from thousands of services and hosts, stores them as time series, powers dashboards, and fires alerts. The defining challenge is a firehose of tiny writes plus queries that scan large time ranges, which calls for specialized time-series storage.

Clarify requirements

Functional

• Ingest metrics (counters, gauges, histograms) with labels, e.g. http_requests_total{service="orders", status="500"}.
• Query by metric name, label filters, and time range with aggregations (rate, sum, percentiles), for dashboards.
• Alerting rules evaluated continuously; notify via pager, chat, email.
• Retention: raw 10 s resolution for 15 days, downsampled 1 min for 90 days, 1 h for 2 years.
• Out of scope: logs and traces (mention integration).

Non-functional

• 10M active time series; 10 s scrape interval.
• Ingestion must not drop data under bursts; dashboards p99 < 1 s for common queries.
• Alerting must keep working even if parts of the system degrade (monitoring must be more reliable than what it monitors).