OpenCost Inference Cost per Million Tokens for Self-Hosted vLLM

The question that started this was simple: what does a million tokens cost on the vLLM server we run, and is that cheaper than the API? Every answer I could produce was a monthly GPU bill divided by a token count pulled from a dashboard, with a utilisation assumption hidden in the middle.

OpenCost 1.121.0 turns that division into a metric. This post is a close reading of the CNCF announcement, OpenCost's own blog post, the inference-cost-tracking guide in the repository and the two pull requests behind it, checked against vLLM's metrics documentation. I have not run 1.121.0 on a cluster, so every figure below comes from those pages or is my own arithmetic, labelled as such.

What OpenCost 1.121.0 adds: two series per model

The CNCF announcement of 5 August 2026 describes 1.121.0 as connecting GPU infrastructure cost to LLM token throughput, so a platform team can see what each model and each token costs. It was built with llm-d in mind, but the post says the core metrics also work with plain vLLM. The v1.121.0 release page lists the feature as pull request 3845, AI Inference Costs v1, plus a follow-up, pull request 3901, that fixes how the usage basis is calculated.

Token counts come from vLLM's vllm:prompt_tokens_total and vllm:generation_tokens_total counters; the dollar side comes from OpenCost's existing allocation engine. Every series carries model_name, model_version, namespace, cost_basis and workload_type, and workload_type is currently always inference.

Allocation cost is the bill; usage cost is the work

The two cost bases answer different questions, and the rest of this post depends on keeping them apart. The CNCF article frames allocation as what the model costs us to keep available, and usage as what its active inference actually cost, with savings from KV cache hits. The repository guide, which I read on the develop branch and which may be newer than the 1.121.0 tag, is more mechanical.

Utilisation falls out of the ratio: usage cost per million tokens divided by allocation cost per million tokens. The article's worked example is 1.00 dollar against 4.00 dollars, which is 25 per cent. Notice what that ratio really measures. Because the usage basis drops idle and shared cost, a model that is busy every second still sits below 100 per cent whenever shared infrastructure is labelled. It is a cost-efficiency ratio, not a GPU counter.

Turning it on: one flag and one label that must match

Setup is small. Set INFERENCE_COST_ENABLED to true on the OpenCost deployment; it reuses PROMETHEUS_SERVER_ENDPOINT for the vLLM metrics, so there is no second Prometheus to configure. The join between cost and tokens is keyed on model name and namespace, which makes one pod label the whole integration.

# 1. Turn it on. Names and defaults are from docs/inference-cost-tracking.md.
#    OpenCost reuses PROMETHEUS_SERVER_ENDPOINT, so no second Prometheus.
env:
  - name: INFERENCE_COST_ENABLED
    value: "true"                    # default is false
  - name: INFERENCE_MODEL_LABEL      # default: llm-d.ai/model
    value: "llm-d.ai/model"
  - name: INFERENCE_SHARED_INFRA_LABEL        # default: llm-d.ai/inference-shared
    value: "llm-d.ai/inference-shared"
  - name: INFERENCE_SHARED_INFRA_LABEL_VALUE  # default: "true"
    value: "true"

# 2. Label the vLLM pods. The value must equal vLLM's model_name, and the
#    namespace must match exactly. Wrong: label says "acme-chat-8b" while vLLM
#    reports the full path, so tokens show up and every cost field is zero.
#    Right: pin the name on both sides.
#      vllm serve /models/acme-chat-8b --served-model-name acme-chat-8b
metadata:
  namespace: llm-serving
  labels:
    llm-d.ai/model: acme-chat-8b

# 3. Prove the join. These are the guide's own endpoints (examples use :9003).
#    A non-zero totalCost for the model means the labels line up.
#    curl "http://localhost:9003/inferenceCost/total?window=7d&aggregate=model_name"
#    curl "http://localhost:9003/inferenceCost/timeseries?window=24h&accumulate=hour&costBasis=usage&filter=namespace:llm-serving"

The label value must equal vLLM's model_name, which is the --served-model-name if you set it and the full --model path if you did not. The guide says OpenCost reconciles full org/model names with short ones, but the namespace has to match exactly. Pods for the scheduler and gateway get the shared-infra label so their cost is spread across models instead of showing up as unattributed overhead.

Utilisation and break-even as PromQL you can alert on

In what I read, OpenCost does not publish a utilisation series, so I would derive it. Select the blended series, the ones without a phase label, on each basis and divide. Record it once and every dashboard and alert shares the same definition.

# prometheus-rules.yaml. The blended series has no phase label, hence phase="".
groups:
  - name: inference-cost
    rules:
      # utilisation = usage cost per 1M tokens / allocation cost per 1M tokens
      - record: model:llm_utilisation:ratio
        expr: |
          sum by (model_name, namespace) (
            llm_cost_per_million_tokens{cost_basis="usage", phase=""}
          )
          /
          sum by (model_name, namespace) (
            llm_cost_per_million_tokens{cost_basis="allocation", phase=""}
          )

      # Compare ALLOCATION cost with the API price. 2.00 is the article's
      # illustrative price; put your real one here.
      - alert: SelfHostedCostAboveApiPrice
        expr: |
          sum by (model_name, namespace) (
            llm_cost_per_million_tokens{cost_basis="allocation", phase=""}
          ) > 2.00
        for: 6h   # allocation cost swings with traffic; do not page on a quiet hour
        labels:
          severity: ticket

# Daily spend for one model, from the hourly-rate gauge (it is a rate, not a counter):
#   avg_over_time(llm_total_hourly_cost{cost_basis="allocation"}[24h]) * 24

The alert fires on allocation cost, because that is the number the CNCF post says to compare with an API price. It uses 2.00 dollars only because the article's example does; replace it with the price of the hosted model you would actually switch to. The long for duration matters: a lightly used model's allocation cost swings with traffic.

The break-even: usage cost divided by the API price

The CNCF example is 1.00 dollar usage and 4.00 dollars allocation per million tokens, which is 25 per cent utilisation. Against a 2.00 dollar API price the API is cheaper, and the article says self-hosting becomes competitive only above roughly 50 per cent. That 50 is not a rule of thumb; it is 1.00 divided by 2.00. If usage cost per million tokens stays roughly flat as traffic changes, which is my assumption here, allocation cost is usage cost divided by utilisation.

Every row except the 25 per cent one is my own arithmetic on the article's illustrative prices, not a measurement. The last row is there because the article says a low-traffic model might spend 95 per cent of its time warm but idle. If cost scaled one-for-one with busy time, that model would sit near 5 per cent. In practice the usage basis scales GPU cost by average GPU usage rather than by time busy, so treat the bottom rows as a direction, not a forecast.

Two practical reads follow. The threshold moves whenever either price moves, so store the API price as a variable rather than a constant. And the lever you control is utilisation: consolidating traffic onto fewer replicas moves a model up the table, while a cheaper GPU changes the numerator instead.

Reading the four quadrants before you decide

The CNCF post gives a two-by-two for reading allocation against usage. I reproduce it because it separates the cases where the fix is traffic from the cases where the fix is the model.

The top row is where the break-even calculation bites. A high-allocation, low-usage model may lose to the API today and win after consolidation, whereas a high-usage model will not be rescued by traffic. The post says its proof of concept ran on a cluster with 109 GPUs and 30 models; I found no per-model figures from it, so I have none to show you.

Where the numbers lie: failure modes to test first

The quiet one is zero cost. Per the guide, if the pod label is missing or mismatched, tokens still appear but every cost field is zero and allocation_method is empty. Nothing errors; the dashboard simply shows a model that processes tokens for free.

The input and output split can fall back to a guess. OpenCost divides infrastructure cost by prefill time against total time, using vllm:request_prefill_time_seconds_sum and vllm:request_time_per_output_token_seconds_sum. If those are missing or zero, allocation_method shows multiplier and output cost is estimated as 2.5 times input cost. The blended figure is unaffected; the split is not.

The usage basis shipped with a bug. Pull request 3901 says allocation costs were previously used for GPUs, CPUs and RAM even when costBasis was usage. The fix scales GPU by average GPU usage, CPU by core usage over request and RAM by bytes used over requested, and leaves network, persistent volume, load balancer and external cost unscaled. It is in the v1.121.0 release notes alongside the feature, but it means your utilisation ratio rests on a measured GPU usage average, which is not the same quantity as tokens served per second.

Other limits from the pages I read: the feature needs Prometheus, because vLLM metrics are only available there; wasted-GPU measurement, idle-GPU detection, UI integration and team attribution are listed as ongoing work; and the pull request notes that per-pod UID labels from llm-d would allow more accurate per-token cost but were not yet done. Finally, the CNCF post points out that cost is not price. Your API figure is a price, your self-hosted figure is a cost, and neither includes the engineering time to run the stack, which is my observation rather than the article's.

What to do on Monday: a measurement week

Do not migrate anything yet. Spend a week proving the numbers, then decide model by model.

1. Enable INFERENCE_COST_ENABLED on a non-production OpenCost, label one vLLM deployment, and call /inferenceCost/total to confirm the cost is non-zero.
2. Check that allocation_method reports compute_time; if it says multiplier, fix the vLLM timing metrics before you use the input and output split.
3. Record the blended allocation figure, the usage figure and their ratio for each model across a full week, so a quiet weekend is in the sample.
4. Compute each model's break-even as usage cost divided by the API price you would switch to, and compare it with the measured ratio.
5. Act on the quadrant: consolidate or share high-allocation, low-usage models, resize high-usage ones, and leave the rest alone.

# Port-forward OpenCost (the service name varies by install; the guide's
# examples use localhost:9003), then pull a week on both bases.
kubectl -n opencost port-forward svc/<opencost-service> 9003:9003 &

curl -s "http://localhost:9003/inferenceCost/total?window=7d&aggregate=model_name&costBasis=allocation"   | jq '.data.inferenceCosts'
curl -s "http://localhost:9003/inferenceCost/total?window=7d&aggregate=model_name&costBasis=usage"   | jq '.data.inferenceCosts'

# Break-even utilisation = usage cost per 1M tokens / API price per 1M tokens.
# Plug in YOUR measured usage cost; 1.00 and 2.00 are the article's example.
awk 'BEGIN { usage = 1.00; api = 2.00; printf "break-even: %.0f%%
", 100 * usage / api }'

I have not run this on a cluster, so I cannot hand you a measured break-even. What I can say is that the method is two API calls, one recording rule and one division.

Price self-hosted inference by allocation cost per million tokens, read utilisation as usage over allocation, and trust neither until the label join is proven non-zero. The break-even is not a fixed percentage but usage cost divided by the API price, and it moves every time either one changes.