DevOps
Prometheus Cardinality: Recording Rules That Save Your TSDB
July 202610 min read
Photo by kevinmarsh on flickr
Cardinality is the number of unique time series a single metric name produces once every combination of label values is counted. High cardinality matters because Prometheus keeps every active series resident in memory, so a metric with unbounded labels like user IDs or raw request paths can silently grow into millions of series and exhaust a server's RAM.
Recording rules run a PromQL expression on a fixed schedule and store the aggregated result as a new, much smaller set of series. Dashboards and alerts then query that cheap pre-aggregated series instead of recomputing an expensive, high-cardinality query from raw data on every request, which lowers both query cost and effective series count for downstream consumers.
No, they solve different problems. Recording rules run on data already scraped and stored, so the raw high-cardinality series still costs ingestion, WAL, and head-block memory before any rollup happens. Relabeling with metric_relabel_configs runs at scrape time, before samples are written, so it is the right tool for stopping truly unbounded labels from ever reaching storage.
Open the built-in tsdb-status page on your Prometheus server, which ranks metric names and label pairs by how many series they are responsible for. Combine that with the prometheus_rule_evaluation_duration_seconds metric to see whether your existing recording rules are starting to lag behind their evaluation interval, an early sign that cardinality has outgrown your current aggregation strategy.
The official convention is level colon metric colon operations, where level lists the remaining aggregation labels, metric keeps the original metric name unchanged, and operations lists applied functions with the newest first. This keeps rule names self-documenting so anyone reading the rule immediately knows which raw metric it derives from and what was done to it.
Photo by kevinmarsh on flickr
Every Prometheus operator eventually hits the same wall: dashboards that used to render instantly start timing out, the server's memory graph climbs in a straight line, and then one night the process gets OOM-killed. The root cause is almost always cardinality, not raw traffic volume. A single metric name with a handful of labels can silently explode into millions of distinct time series, and Prometheus has to keep every one of those series resident in memory while it is active.
This post is a practical walkthrough of how cardinality actually costs you memory, how to measure it before you start guessing, and the two complementary tools that keep it under control in production: recording rules that pre-aggregate expensive queries, and relabeling that stops unbounded labels from ever reaching storage. By the end you will have a checklist you can run against your own Prometheus deployment this week.
Cardinality is the number of unique time series a metric name produces once you account for every combination of label values. A counter called http_requests_total with a method label and a status label might only have a dozen series. Add a raw request path or a customer identifier as a label, though, and that same counter can produce one new series per unique path or per unique customer, forever. Each active series sits in the in-memory head block, and industry write-ups on Prometheus internals put the cost at roughly one to three kibibytes of RAM per series, so a server holding a few million active series needs double-digit gigabytes just to stay alive. Cross a few million series and query latency, rule evaluation time, and WAL replay time on restart all degrade together.
A metric does not need to look dangerous to be dangerous. Two innocuous labels with 1,000 values each combine multiplicatively into a million series the moment they appear on the same metric. Cardinality risk is a product of label value counts, not a sum.
Guessing which metrics are the problem wastes time and often leads to trimming labels that were never the bottleneck. Prometheus ships with a built-in diagnostic page at the slash tsdb-status endpoint on your server that ranks metric names and label pairs by how many series they are responsible for. Pair that with the prometheus_rule_evaluation_duration_seconds metric, which tells you when your own recording rules are starting to fall behind their evaluation interval, a strong early signal that cardinality growth has outpaced your aggregation strategy.
| Diagnostic signal | Where to find it | What it tells you |
|---|---|---|
| Total active series count | The Head Stats section of the tsdb-status page | Overall memory pressure across the whole server, and how close you are to the danger zone |
| Label pairs ranked by series count | The label cardinality table on the same tsdb-status page | Exactly which label on which metric is responsible for the bulk of your series, so you know what to fix first |
| prometheus_rule_evaluation_duration_seconds | Prometheus's own slash metrics endpoint, scraped like any other target | Whether your recording rules are keeping up with their configured evaluation interval or silently lagging behind |
A recording rule runs a PromQL expression on a fixed schedule and stores the result as a brand-new time series. The trick that makes them effective against cardinality is aggregation: instead of exposing per-pod, per-path, per-status detail on every query, you compute a rolled-up series once per evaluation interval and let every dashboard and alert read that cheap, pre-aggregated series instead of recomputing the expensive query from raw data every time. The official naming convention is level colon metric colon operations, where level describes the remaining aggregation labels, metric keeps the original metric name unchanged, and operations lists the applied functions with the newest one first. Keeping the metric name untouched is deliberate: anyone reading job_path colon http_request_duration_seconds colon p99_5m immediately knows it derives from http_request_duration_seconds.
# groups/http-slo.rules.yml
groups:
- name: http_slo_aggregations
interval: 30s
rules:
# level:metric:operations naming convention
- record: instance_path:http_requests:rate5m
expr: |
sum without (pod, replica) (
rate(http_requests_total[5m])
)
- record: job_path:http_request_duration_seconds:p99_5m
expr: |
histogram_quantile(
0.99,
sum without (pod, replica) (
rate(http_request_duration_seconds_bucket[5m])
)
)
- record: job:http_errors:ratio_rate5m
expr: |
sum without (pod, replica) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum without (pod, replica) (rate(http_requests_total[5m]))
Always pair aggregation with an explicit without clause listing the labels you are dropping, rather than listing the labels you keep. That way new labels added to the source metric later are preserved automatically. And never average a ratio or average an average across instances, that is not statistically valid; aggregate the numerator and denominator separately first, then divide.
Recording rules run on data Prometheus has already scraped and stored, which means the raw high-cardinality series still costs you ingestion, WAL, and head-block memory before the rollup ever happens. If a label is truly unbounded and you never need the raw detail, the better fix is to stop it at scrape time using metric_relabel_configs. This runs after a target is scraped but before samples are written to the TSDB, so you can drop entire metrics you no longer need, or rewrite a label's value into a small, bounded set of buckets.
scrape_configs:
- job_name: checkout-service
metric_relabel_configs:
# drop the raw histogram once the recording rule covers it
- source_labels: [__name__]
regex: "http_request_duration_seconds_bucket"
action: drop
# collapse unbounded path label into a bounded set
- source_labels: [path]
regex: "/orders/[0-9]+"
target_label: path
replacement: "/orders/:id"
# cap total series this target can expose per scrape
sample_limit: 10000
Recording rules are not a silver bullet, and it is worth being honest about their limits before you lean on them as your only defense. Because they operate on data that has already been scraped and written to the TSDB, they reduce the cost of querying and dashboarding, but they do nothing for the ingestion-time cost of the raw series. Streaming aggregation at ingest, in tools built for that purpose, avoids this by never letting the high-cardinality raw series land in long-term storage at all.
Get the combination right and the payoff is dramatic: dashboards that once took seconds to render from raw histograms return in milliseconds from pre-aggregated series, and a healthy mid-sized production Prometheus typically settles somewhere between 100,000 and 2,000,000 total active series, comfortably below the range where operators start reporting instability.
Start by pulling up the tsdb-status page on your own server and writing down the top five label pairs by series count, that list is your actual priority order, not a guess. For each offender, decide whether the detail is something a human will query at full resolution during an incident, in which case a recording rule that aggregates it down is the right tool, or whether it is genuinely unbounded and never needed at full resolution, in which case relabeling it away at scrape time is the right tool. Most production fixes end up using both: relabel the truly unbounded labels to stop the bleeding, then add recording rules for the aggregations your dashboards and alerts actually query most often.
Before shipping a new recording rule group to production, validate it with promtool's rule test command against a small fixture file. It catches syntax errors and obviously wrong aggregations in seconds, long before a broken rule silently feeds a bad number into your on-call alerting.
Cardinality problems are rarely dramatic until the moment they are: a slow memory creep for weeks, then an OOM kill at the worst possible time. Measuring first with the tsdb-status page, then applying relabeling to stop unbounded labels at the door and recording rules to make the queries you actually run cheap, turns that slow creep into a manageable, monitored budget instead of a recurring incident.