# Prometheus Cardinality: Recording Rules That Save Your TSDB

> High-cardinality labels quietly eat your Prometheus server's RAM until it OOMs. Here is how recording rules, label hygiene, and relabeling keep the TSDB small and PromQL fast.

URL: https://www.matthewswong.com/en/blog/prometheus-recording-rules-cardinality/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-07-06
- Updated: 2026-07-06
- Category: DevOps

Every Prometheus operator eventually hits the same wall: dashboards that used to render instantly start timing out, the server's memory graph climbs in a straight line, and then one night the process gets OOM-killed. The root cause is almost always cardinality, not raw traffic volume. A single metric name with a handful of labels can silently explode into millions of distinct time series, and Prometheus has to keep every one of those series resident in memory while it is active.

This post is a practical walkthrough of how cardinality actually costs you memory, how to measure it before you start guessing, and the two complementary tools that keep it under control in production: recording rules that pre-aggregate expensive queries, and relabeling that stops unbounded labels from ever reaching storage. By the end you will have a checklist you can run against your own Prometheus deployment this week.

## Why Cardinality Is the Metric That Silently Kills Your Server

Cardinality is the number of unique time series a metric name produces once you account for every combination of label values. A counter called http_requests_total with a method label and a status label might only have a dozen series. Add a raw request path or a customer identifier as a label, though, and that same counter can produce one new series per unique path or per unique customer, forever. Each active series sits in the in-memory head block, and industry write-ups on Prometheus internals put the cost at roughly one to three kibibytes of RAM per series, so a server holding a few million active series needs double-digit gigabytes just to stay alive. Cross a few million series and query latency, rule evaluation time, and WAL replay time on restart all degrade together.

- Free-form identifiers as labels: user_id, request_id, trace_id, session token, or any value generated per request rather than per resource type.
- Raw HTTP paths with path parameters left in place, such as slash orders slash 48213, instead of a templated route like slash orders slash colon id.
- Ephemeral infrastructure names: pod name, container instance hash, or short-lived hostname labels that churn every time Kubernetes reschedules a workload.

> A metric does not need to look dangerous to be dangerous. Two innocuous labels with 1,000 values each combine multiplicatively into a million series the moment they appear on the same metric. Cardinality risk is a product of label value counts, not a sum.

## Measure Before You Optimize

Guessing which metrics are the problem wastes time and often leads to trimming labels that were never the bottleneck. Prometheus ships with a built-in diagnostic page at the slash tsdb-status endpoint on your server that ranks metric names and label pairs by how many series they are responsible for. Pair that with the prometheus_rule_evaluation_duration_seconds metric, which tells you when your own recording rules are starting to fall behind their evaluation interval, a strong early signal that cardinality growth has outpaced your aggregation strategy.

| Diagnostic signal | Where to find it | What it tells you |
| --- | --- | --- |
| Total active series count | The Head Stats section of the tsdb-status page | Overall memory pressure across the whole server, and how close you are to the danger zone |
| Label pairs ranked by series count | The label cardinality table on the same tsdb-status page | Exactly which label on which metric is responsible for the bulk of your series, so you know what to fix first |
| prometheus_rule_evaluation_duration_seconds | Prometheus's own slash metrics endpoint, scraped like any other target | Whether your recording rules are keeping up with their configured evaluation interval or silently lagging behind |

## Recording Rules: Pay the Cost Once, Query the Result Forever

A recording rule runs a PromQL expression on a fixed schedule and stores the result as a brand-new time series. The trick that makes them effective against cardinality is aggregation: instead of exposing per-pod, per-path, per-status detail on every query, you compute a rolled-up series once per evaluation interval and let every dashboard and alert read that cheap, pre-aggregated series instead of recomputing the expensive query from raw data every time. The official naming convention is level colon metric colon operations, where level describes the remaining aggregation labels, metric keeps the original metric name unchanged, and operations lists the applied functions with the newest one first. Keeping the metric name untouched is deliberate: anyone reading job_path colon http_request_duration_seconds colon p99_5m immediately knows it derives from http_request_duration_seconds.

> Always pair aggregation with an explicit without clause listing the labels you are dropping, rather than listing the labels you keep. That way new labels added to the source metric later are preserved automatically. And never average a ratio or average an average across instances, that is not statistically valid; aggregate the numerator and denominator separately first, then divide.

## Relabeling: Stop Cardinality Before It Ever Lands in Storage

Recording rules run on data Prometheus has already scraped and stored, which means the raw high-cardinality series still costs you ingestion, WAL, and head-block memory before the rollup ever happens. If a label is truly unbounded and you never need the raw detail, the better fix is to stop it at scrape time using metric_relabel_configs. This runs after a target is scraped but before samples are written to the TSDB, so you can drop entire metrics you no longer need, or rewrite a label's value into a small, bounded set of buckets.

1. Find the offending metric and label using the tsdb-status page or a cardinality-focused exporter, and confirm the label truly has no analytical value at full resolution.
2. Add a metric_relabel_configs rule under the relevant scrape_configs job: use action drop for metrics you no longer need, or a regex replacement to collapse a path parameter into a templated route.
3. Set a per-target sample_limit as a hard ceiling. It will not fix the root cause, but it stops a single misbehaving target from taking the whole server down while you roll out the real fix.
4. Reload Prometheus, wait one full scrape interval, and re-check the tsdb-status page to confirm the series count for that metric actually dropped before moving on to the next offender.

## Where Recording Rules Fall Short

Recording rules are not a silver bullet, and it is worth being honest about their limits before you lean on them as your only defense. Because they operate on data that has already been scraped and written to the TSDB, they reduce the cost of querying and dashboarding, but they do nothing for the ingestion-time cost of the raw series. Streaming aggregation at ingest, in tools built for that purpose, avoids this by never letting the high-cardinality raw series land in long-term storage at all.

- Ingestion cost is unchanged: the raw series still needs to be scraped, appended to the WAL, and held in the head block before your rule can aggregate it away.
- Stale rules accumulate: a recording rule referencing a label that a source metric no longer exposes will silently produce no data instead of erroring loudly, so periodic review matters.
- Evaluation lag is a real failure mode: if prometheus_rule_evaluation_duration_seconds creeps past your configured evaluation interval, your rolled-up series start lagging reality, which quietly breaks alerting thresholds.

Get the combination right and the payoff is dramatic: dashboards that once took seconds to render from raw histograms return in milliseconds from pre-aggregated series, and a healthy mid-sized production Prometheus typically settles somewhere between 100,000 and 2,000,000 total active series, comfortably below the range where operators start reporting instability.

## A Rollout Checklist You Can Ship This Week

Start by pulling up the tsdb-status page on your own server and writing down the top five label pairs by series count, that list is your actual priority order, not a guess. For each offender, decide whether the detail is something a human will query at full resolution during an incident, in which case a recording rule that aggregates it down is the right tool, or whether it is genuinely unbounded and never needed at full resolution, in which case relabeling it away at scrape time is the right tool. Most production fixes end up using both: relabel the truly unbounded labels to stop the bleeding, then add recording rules for the aggregations your dashboards and alerts actually query most often.

> Before shipping a new recording rule group to production, validate it with promtool's rule test command against a small fixture file. It catches syntax errors and obviously wrong aggregations in seconds, long before a broken rule silently feeds a bad number into your on-call alerting.

Cardinality problems are rarely dramatic until the moment they are: a slow memory creep for weeks, then an OOM kill at the worst possible time. Measuring first with the tsdb-status page, then applying relabeling to stop unbounded labels at the door and recording rules to make the queries you actually run cheap, turns that slow creep into a manageable, monitored budget instead of a recurring incident.

## Sources and further reading

- Prometheus Storage documentation — head block, WAL, and chunk internals
- Prometheus recording rules best practices — official naming conventions
- Prometheus configuration reference — relabel_configs and metric_relabel_configs

## Source links

- https://prometheus.io/docs/prometheus/latest/storage/
- https://prometheus.io/docs/practices/rules/
- https://prometheus.io/docs/prometheus/latest/configuration/configuration/

## FAQ

### What is metric cardinality in Prometheus and why does it matter?

Cardinality is the number of unique time series a single metric name produces once every combination of label values is counted. High cardinality matters because Prometheus keeps every active series resident in memory, so a metric with unbounded labels like user IDs or raw request paths can silently grow into millions of series and exhaust a server's RAM.

### How do recording rules help reduce Prometheus cardinality?

Recording rules run a PromQL expression on a fixed schedule and store the aggregated result as a new, much smaller set of series. Dashboards and alerts then query that cheap pre-aggregated series instead of recomputing an expensive, high-cardinality query from raw data on every request, which lowers both query cost and effective series count for downstream consumers.

### Can recording rules replace relabeling for cardinality control?

No, they solve different problems. Recording rules run on data already scraped and stored, so the raw high-cardinality series still costs ingestion, WAL, and head-block memory before any rollup happens. Relabeling with metric_relabel_configs runs at scrape time, before samples are written, so it is the right tool for stopping truly unbounded labels from ever reaching storage.

### How can I measure which metrics are causing high cardinality?

Open the built-in tsdb-status page on your Prometheus server, which ranks metric names and label pairs by how many series they are responsible for. Combine that with the prometheus_rule_evaluation_duration_seconds metric to see whether your existing recording rules are starting to lag behind their evaluation interval, an early sign that cardinality has outgrown your current aggregation strategy.

### What naming convention should I use for Prometheus recording rules?

The official convention is level colon metric colon operations, where level lists the remaining aggregation labels, metric keeps the original metric name unchanged, and operations lists applied functions with the newest first. This keeps rule names self-documenting so anyone reading the rule immediately knows which raw metric it derives from and what was done to it.

---

- HTML version: https://www.matthewswong.com/en/blog/prometheus-recording-rules-cardinality/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/prometheus-recording-rules-cardinality/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json