Prometheus 3.0: OTLP, Native Histograms, UTF-8 & New UI

I have run Prometheus in production since the 2.x days, so when the team announced version 3.0 in November 2024 — the first major release in seven years — I read the notes carefully before touching a single server. The headline is that 3.0 is not a rewrite. It is the same battle-tested TSDB and PromQL engine, with a modernized UI and a batch of features that were experimental for years finally becoming production-ready.

That is also the trap. Several things that used to live behind feature flags are now on by default, and a few behaviors changed in ways that will quietly break dashboards and recording rules if you upgrade without reading. This post walks through what is genuinely new and useful, then the breaking changes I check for before rolling 3.0 into any cluster.

A redesigned UI built on the PromLens work

The most visible change is the web UI. The old expression browser was functional but dated; the new one adopts work from PromLens, including a tree view that breaks a PromQL query into its parsed structure so you can see exactly how each part evaluates. It also renders UTF-8 metric names correctly and generally feels like a modern app. If something in your workflow depends on the old interface, the legacy UI is still reachable behind the old-ui feature flag for now.

UTF-8 metric and label names by default

For years Prometheus restricted metric and label names to a limited ASCII character set, which meant OpenTelemetry names full of dots got those dots rewritten to underscores. In 3.0, all valid UTF-8 characters are allowed in names by default. The cost is a small syntax change: to select a metric whose name is not a legal identifier, you use a new quoted form inside the selector, or spell out the name label yourself.

# In 3.0, a metric name can contain dots, slashes, and other UTF-8.
# PromQL adds a new quoting syntax to select such names:
{"my.custom.metric", job="api"}

# ...which is equivalent to spelling out __name__ explicitly:
{__name__="my.custom.metric", job="api"}

Stable OTLP ingestion for OpenTelemetry

Prometheus 3.0 can act as a native receiver for the OTLP metrics protocol, so an OpenTelemetry Collector can push straight into it with no sidecar exporter. You turn the receiver on with a command-line flag, and it then accepts OTLP over HTTP at the OTLP metrics endpoint. I pair it with the NoUTF8EscapingWithSuffixes translation strategy so OTel names keep their dots, and I widen the out-of-order window because OTel pipelines frequently deliver samples slightly late.

# Start Prometheus with the OTLP receiver turned on.
# It then serves OTLP metrics at /api/v1/otlp/v1/metrics
prometheus --web.enable-otlp-receiver

# prometheus.yml — recommended OTLP settings for OTel data
otlp:
  # Keep UTF-8 names instead of rewriting dots to underscores
  translation_strategy: NoUTF8EscapingWithSuffixes

storage:
  tsdb:
    # OTel pipelines often push slightly out of order
    out_of_order_time_window: 30m

Native histograms — powerful, still experimental

Native histograms are the feature I am most excited about and the one I am slowest to adopt in production. Instead of pre-defining fixed buckets, they use exponential buckets that adapt automatically, giving far higher resolution at a fraction of the storage and cardinality cost of classic histograms. In 3.0 they remain experimental: you opt in with a feature flag, the on-disk and wire formats can still change, and the tooling around them is incomplete. I run them in staging and keep classic buckets scraped in parallel.

# Native histograms are still EXPERIMENTAL in 3.0 — opt in explicitly.
prometheus --enable-feature=native-histograms

# During migration you can keep scraping the classic buckets too.
# (this option was renamed from scrape_classic_histograms in 3.0)
scrape_configs:
  - job_name: api
    always_scrape_classic_histograms: true

Remote Write 2.0

Remote Write 2.0 is a new version of the protocol that carries everything a remote-write target long wanted in one payload: metric metadata, exemplars, created timestamps, and native histograms all travel together. It also uses string interning to deduplicate repeated label strings, which cuts both payload size and CPU on busy senders. It is negotiated automatically, so a 3.0 sender falls back to 1.0 when the receiver does not speak 2.0 yet.

Breaking changes when upgrading from 2.x

None of this is free. Before I move a cluster to 3.0 I walk this checklist, because these are the changes that tend to break silently rather than loudly:

• Upgrade to 2.55 first. The TSDB format moved, so a rollback from 3.0 only works back to 2.55 — not to older 2.x releases. Stage the two-step jump; do not leap straight from an old 2.x build.
• Regex now matches newlines. In PromQL and relabeling, the dot in a regex matches a newline character in 3.0. If a rule relied on the old behavior, replace the dot with an explicit non-newline class to keep it accurate.
• Range selector boundaries changed. Ranges are now left-open and right-closed, which subtly shifts subqueries and aligned queries; re-check any recording rule where a sample landing exactly on a boundary matters.
• Histogram labels are normalized. The le and quantile label values become float form, so an le of 1 is stored as 1.0 — any query or alert matching the integer string will stop matching.

Prometheus 2.x versus 3.0 at a glance

My advice after running the upgrade: treat 3.0 as low-risk on the engine and high-attention on the edges. The storage and query core are the same code you already trust, so the migration itself is fast. The work is in the details — the regex and range-selector semantics, the label normalization, the stricter scraping — so read the official migration guide end to end, upgrade to 2.55 first, and roll it out one cluster at a time.