Every Prometheus operator eventually hits the same wall: dashboards that used to render instantly start timing out, the server's memory graph climbs in a straight line, and then one night the process gets OOM-killed. The root cause is almost always cardinality, not raw traffic volume. A single metric name with a handful of labels can silently explode into millions of distinct time series, and Prometheus has to keep every one of those series resident in memory while it is active.
This post is a practical walkthrough of how cardinality actually costs you memory, how to measure it before you start guessing, and the two complementary tools that keep it under control in production: recording rules that pre-aggregate expensive queries, and relabeling that stops unbounded labels from ever reaching storage. By the end you will have a checklist you can run against your own Prometheus deployment this week.
Why Cardinality Is the Metric That Silently Kills Your Server
Cardinality is the number of unique time series a metric name produces once you account for every combination of label values. A counter called http_requests_total with a method label and a status label might only have a dozen series. Add a raw request path or a customer identifier as a label, though, and that same counter can produce one new series per unique path or per unique customer, forever. Each active series sits in the in-memory head block, and industry write-ups on Prometheus internals put the cost at roughly one to three kibibytes of RAM per series, so a server holding a few million active series needs double-digit gigabytes just to stay alive. Cross a few million series and query latency, rule evaluation time, and WAL replay time on restart all degrade together.
- Free-form identifiers as labels: user_id, request_id, trace_id, session token, or any value generated per request rather than per resource type.
- Raw HTTP paths with path parameters left in place, such as slash orders slash 48213, instead of a templated route like slash orders slash colon id.
- Ephemeral infrastructure names: pod name, container instance hash, or short-lived hostname labels that churn every time Kubernetes reschedules a workload.
A metric does not need to look dangerous to be dangerous. Two innocuous labels with 1,000 values each combine multiplicatively into a million series the moment they appear on the same metric. Cardinality risk is a product of label value counts, not a sum.
Measure Before You Optimize
Guessing which metrics are the problem wastes time and often leads to trimming labels that were never the bottleneck. Prometheus ships with a built-in diagnostic page at the slash tsdb-status endpoint on your server that ranks metric names and label pairs by how many series they are responsible for. Pair that with the prometheus_rule_evaluation_duration_seconds metric, which tells you when your own recording rules are starting to fall behind their evaluation interval, a strong early signal that cardinality growth has outpaced your aggregation strategy.
| Diagnostic signal | Where to find it | What it tells you |
|---|---|---|
| Total active series count | The Head Stats section of the tsdb-status page | Overall memory pressure across the whole server, and how close you are to the danger zone |
| Label pairs ranked by series count | The label cardinality table on the same tsdb-status page | Exactly which label on which metric is responsible for the bulk of your series, so you know what to fix first |
| prometheus_rule_evaluation_duration_seconds | Prometheus's own slash metrics endpoint, scraped like any other target | Whether your recording rules are keeping up with their configured evaluation interval or silently lagging behind |
Recording Rules: Pay the Cost Once, Query the Result Forever
A recording rule runs a PromQL expression on a fixed schedule and stores the result as a brand-new time series. The trick that makes them effective against cardinality is aggregation: instead of exposing per-pod, per-path, per-status detail on every query, you compute a rolled-up series once per evaluation interval and let every dashboard and alert read that cheap, pre-aggregated series instead of recomputing the expensive query from raw data every time. The official naming convention is level colon metric colon operations, where level describes the remaining aggregation labels, metric keeps the original metric name unchanged, and operations lists the applied functions with the newest one first. Keeping the metric name untouched is deliberate: anyone reading job_path colon http_request_duration_seconds colon p99_5m immediately knows it derives from http_request_duration_seconds.
# groups/http-slo.rules.yml
groups:
- name: http_slo_aggregations
interval: 30s
rules:
# level:metric:operations naming convention
- record: instance_path:http_requests:rate5m
expr: |
sum without (pod, replica) (
rate(http_requests_total[5m])
)
- record: job_path:http_request_duration_seconds:p99_5m
expr: |
histogram_quantile(
0.99,
sum without (pod, replica) (
rate(http_request_duration_seconds_bucket[5m])
)
)
- record: job:http_errors:ratio_rate5m
expr: |
sum without (pod, replica) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum without (pod, replica) (rate(http_requests_total[5m]))
Always pair aggregation with an explicit without clause listing the labels you are dropping, rather than listing the labels you keep. That way new labels added to the source metric later are preserved automatically. And never average a ratio or average an average across instances, that is not statistically valid; aggregate the numerator and denominator separately first, then divide.
Relabeling: Stop Cardinality Before It Ever Lands in Storage
Recording rules run on data Prometheus has already scraped and stored, which means the raw high-cardinality series still costs you ingestion, WAL, and head-block memory before the rollup ever happens. If a label is truly unbounded and you never need the raw detail, the better fix is to stop it at scrape time using metric_relabel_configs. This runs after a target is scraped but before samples are written to the TSDB, so you can drop entire metrics you no longer need, or rewrite a label's value into a small, bounded set of buckets.
scrape_configs:
- job_name: checkout-service
metric_relabel_configs:
# drop the raw histogram once the recording rule covers it
- source_labels: [__name__]
regex: "http_request_duration_seconds_bucket"
action: drop
# collapse unbounded path label into a bounded set
- source_labels: [path]
regex: "/orders/[0-9]+"
target_label: path
replacement: "/orders/:id"
# cap total series this target can expose per scrape
sample_limit: 10000
- Find the offending metric and label using the tsdb-status page or a cardinality-focused exporter, and confirm the label truly has no analytical value at full resolution.
- Add a metric_relabel_configs rule under the relevant scrape_configs job: use action drop for metrics you no longer need, or a regex replacement to collapse a path parameter into a templated route.
- Set a per-target sample_limit as a hard ceiling. It will not fix the root cause, but it stops a single misbehaving target from taking the whole server down while you roll out the real fix.
- Reload Prometheus, wait one full scrape interval, and re-check the tsdb-status page to confirm the series count for that metric actually dropped before moving on to the next offender.
Where Recording Rules Fall Short
Recording rules are not a silver bullet, and it is worth being honest about their limits before you lean on them as your only defense. Because they operate on data that has already been scraped and written to the TSDB, they reduce the cost of querying and dashboarding, but they do nothing for the ingestion-time cost of the raw series. Streaming aggregation at ingest, in tools built for that purpose, avoids this by never letting the high-cardinality raw series land in long-term storage at all.
- Ingestion cost is unchanged: the raw series still needs to be scraped, appended to the WAL, and held in the head block before your rule can aggregate it away.
- Stale rules accumulate: a recording rule referencing a label that a source metric no longer exposes will silently produce no data instead of erroring loudly, so periodic review matters.
- Evaluation lag is a real failure mode: if prometheus_rule_evaluation_duration_seconds creeps past your configured evaluation interval, your rolled-up series start lagging reality, which quietly breaks alerting thresholds.
Get the combination right and the payoff is dramatic: dashboards that once took seconds to render from raw histograms return in milliseconds from pre-aggregated series, and a healthy mid-sized production Prometheus typically settles somewhere between 100,000 and 2,000,000 total active series, comfortably below the range where operators start reporting instability.
A Rollout Checklist You Can Ship This Week
Start by pulling up the tsdb-status page on your own server and writing down the top five label pairs by series count, that list is your actual priority order, not a guess. For each offender, decide whether the detail is something a human will query at full resolution during an incident, in which case a recording rule that aggregates it down is the right tool, or whether it is genuinely unbounded and never needed at full resolution, in which case relabeling it away at scrape time is the right tool. Most production fixes end up using both: relabel the truly unbounded labels to stop the bleeding, then add recording rules for the aggregations your dashboards and alerts actually query most often.
Before shipping a new recording rule group to production, validate it with promtool's rule test command against a small fixture file. It catches syntax errors and obviously wrong aggregations in seconds, long before a broken rule silently feeds a bad number into your on-call alerting.
Cardinality problems are rarely dramatic until the moment they are: a slow memory creep for weeks, then an OOM kill at the worst possible time. Measuring first with the tsdb-status page, then applying relabeling to stop unbounded labels at the door and recording rules to make the queries you actually run cheap, turns that slow creep into a manageable, monitored budget instead of a recurring incident.