Scaling Prometheus Storage with Grafana Mimir

A single Prometheus is a wonderful thing until you ask it to remember. Local TSDB retention is measured in days or weeks, scaling means a bigger box, and once you run more than one Prometheus you have no single place to query everything. On a monitoring stack I run, I hit all three walls at once: the disk filled, the queries slowed, and I had three Prometheus servers that could not see each other's data.

Grafana Mimir is the answer I settled on. It is an open-source, horizontally scalable long-term store that speaks the Prometheus remote_write protocol on the write side and PromQL on the read side. Prometheus keeps doing what it is good at (scraping and short-term local queries) and forwards every sample to Mimir, which persists it to object storage. This post is how the pieces fit and the configuration I actually use.

Why remote_write instead of just a bigger Prometheus

Scaling Prometheus vertically hits a hard ceiling: the TSDB is single-node, and there is no built-in replication or clustering. remote_write breaks that ceiling by decoupling ingestion from storage. Prometheus buffers samples in a write-ahead log and streams them asynchronously to a remote endpoint, so it keeps scraping and serving local queries even while forwarding. If the remote endpoint is briefly down, the WAL absorbs the backlog and replays it. That single design choice is what lets you fan many Prometheus servers into one central, replicated store.

The minimal remote_write configuration

On the Prometheus side you add a remote_write block pointing at Mimir's push endpoint. The path is /api/v1/push, and the one thing people miss is the X-Scope-OrgID header — Mimir refuses writes that do not name a tenant. Here is the block I use, with sane queue tuning for a busy scrape config:

# prometheus.yml
remote_write:
  - url: http://mimir:8080/api/v1/push
    headers:
      X-Scope-OrgID: team-platform      # the tenant ID — required by Mimir
    queue_config:
      capacity: 10000                   # samples buffered per shard
      max_shards: 50                    # upper bound on parallel senders
      min_shards: 1
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    metadata_config:
      send: true                        # forward metric metadata (help/type)

Mimir does not authenticate on its own — the docs are explicit that auth is expected to sit in front of it, typically an nginx or the mimir gateway that validates a token and injects the X-Scope-OrgID header. In a trusted internal network you can run with the header set directly, but never expose the raw push endpoint to the internet.

What happens to a sample after it lands

Understanding Mimir's write path is what makes the durability guarantees make sense. Mimir is a set of horizontally scalable microservices, each doing one job, and a sample flows through them in order:

1. The distributor receives the remote_write request, validates it, and splits the series across ingesters using a hash ring.
2. Each series is written to the replication factor (three by default), so the loss of one ingester never loses data.
3. Ingesters append samples to a per-tenant TSDB on local disk and, every two hours, cut a block and upload it to object storage.
4. The compactor merges those small two-hour blocks into larger, deduplicated blocks, shrinking storage and speeding up reads.
5. The store-gateway serves queries against those long-term blocks, using shuffle-sharding and caches to keep object-storage calls low.

Tenants: one cluster, many isolated worlds

The X-Scope-OrgID header is not a formality — it is Mimir's tenancy boundary. Every tenant gets its own set of TSDB blocks in object storage, its own series, and its own configurable limits. This is how I run one Mimir cluster for several projects without their metrics colliding or one noisy team starving the others. Limits are set per tenant and enforced at ingestion:

# mimir runtime overrides — per-tenant limits
overrides:
  team-platform:
    ingestion_rate: 150000            # samples/sec accepted
    ingestion_burst_size: 1500000
    max_global_series_per_user: 1500000
    compactor_blocks_retention_period: 1y
  team-billing:
    ingestion_rate: 25000
    max_global_series_per_user: 200000
    compactor_blocks_retention_period: 90d

Notice retention is per tenant too. The billing team's data expires after ninety days; the platform team keeps a full year. On the query side, Grafana passes the same X-Scope-OrgID so a datasource only ever sees its own tenant. If you need to read across tenants you can enable multi-tenant queries by sending a pipe-separated list of tenant IDs, but I keep that off by default to preserve isolation.

Keeping queries fast at long-range scale

Object storage is cheap and durable but slow per request, so a naive design would make a one-year PromQL range query crawl. Mimir avoids this on several fronts. The query-frontend splits a wide time range into smaller parallel sub-queries and caches results, so re-running yesterday's dashboard is nearly free. The store-gateway shards blocks across instances and caches the block index and chunks, cutting the number of object-storage round trips. And the compactor's whole job is to turn thousands of tiny blocks into a handful of large ones, which is the single biggest lever on read latency.

• Run the compactor. Without it, blocks pile up and both query latency and storage cost climb steadily.
• Give the store-gateway and query-frontend a results cache (memcached or Redis) — it is the highest-leverage tuning for dashboards.
• Watch per-tenant series counts; runaway cardinality, not raw sample volume, is what usually breaks a Mimir cluster.

When Mimir is overkill

Be honest about scale. If you run one Prometheus and need thirty days of history, just bump local retention or bolt on Thanos sidecar — Mimir's operational surface (object storage, hash rings, several stateful components) is not free. I reach for Mimir when there is more than one Prometheus to unify, when retention needs to span quarters or years, or when separate teams need isolated, rate-limited tenancy on shared infrastructure. Below that bar, the simpler tool wins.