vLLM Semantic Router v0.4: Mixture-of-Models Routing Guide

Most teams pick a model once, hard-code its name into every client, and revisit the choice only when the bill or a complaint forces them to. The awkward part is that a support greeting and a multi-step refactor are sent to the same expensive model, and swapping it means editing every caller.

vLLM Semantic Router v0.4, codenamed Hermes and released on 24 September 2026, is the project's answer: one stable model name in front of a pool. I have not deployed it. This post is a close reading of the release post and the project's own documentation, with every figure and field name taken from those pages and the YAML marked where I assembled it from documented fields rather than copying it.

One model name in front of a pool

The release post describes the core idea plainly: applications use one stable virtual model name, and the Router selects a backend from a qualified pool, returns the answer and measures the result. The repository README gives the public model ID as vllm-sr/auto and describes the project as an open, programmable decision layer for models and compute, licensed Apache-2.0.

Internally a request travels Signal, then Projection, then Decision, then Algorithm, then Model, with route plugins attached to whichever path was chosen. The clients see three endpoints: /v1/messages, /v1/chat/completions and /v1/responses, served through a shared codec for the three protocols in both buffered and streaming paths. The release post says v0.5 targets general availability for those three paths.

The last row is the honest cost of the whole idea. A static choice cannot misroute, because it never routes. Everything this post covers is machinery that buys per-request choice at the price of new ways to be wrong.

Signals only detect

The signal documentation splits signals into two families. Heuristic signals use explicit rules, request shape, identity or lightweight detectors: Authz, Conversation, Context, Event, Input Modality, Keyword, Language, Metadata and Structure. Learned signals use embeddings, classifiers or configured detector models: Classifier, Complexity, Domain, Embedding, Modality, Fact Check, Hallucination, Jailbreak, Safety, PII, Preference, Reask, Knowledge Base and User Feedback.

Signals are declared under routing.signals, one key per type, each entry needing a name. The page's own example declares a keyword signal and two embedding signals, then groups the embeddings in a projection so that two intents cannot both win.

# From the signal overview in the vLLM Semantic Router docs.
routing:
  signals:
    keywords:
      - name: urgent_keywords
        operator: OR
        keywords: ["urgent", "asap"]
    embeddings:
      - name: technical_support
        threshold: 0.75
        candidates: ["installation guide", "troubleshooting steps"]
      - name: account_management
        threshold: 0.72
        candidates: ["billing information", "subscription management"]
  projections:
    partitions:
      - name: support_intents
        semantics: exclusive      # members compete; one wins
        temperature: 0.3
        members: [technical_support, account_management]
        default: technical_support

The rule the docs state is worth copying into your review checklist: signals stay detection-only. Outcomes belong in decisions, model choice in algorithms, behaviour in plugins. The docs also warn that learned signals may send data to a configured remote classifier or embedding provider, so check each family's data-handling notes before pointing one at a hosted endpoint.

Decisions: rules, tiers and what Unknown means

A decision is a named rule tree plus the candidates it routes to. Rules combine leaf conditions with AND, OR and NOT. The decision page's only full example routes a business-domain request to one model. Below it I added a second decision assembled from documented fields only (tier, on_unknown, operator, type, name); the leaf names must match signals you declared, and I have not run this file.

routing:
  strategy: priority            # default; the other value is confidence
  decisions:
    # Verbatim shape from the decisions page.
    - name: business_route
      description: Route business requests to the business model.
      priority: 110
      rules:
        operator: AND
        conditions:
          - type: domain
            name: business
      modelRefs:
        - model: qwen2.5:3b
          use_reasoning: false

    # Assembled by me from documented fields. Not copied, not executed.
    - name: guarded_route
      tier: 1                   # lower tier wins; no tier means tier 0
      priority: 200
      rules:
        operator: AND
        on_unknown: fail_request  # classifier failure returns a 503, not a guess
        conditions:
          - type: jailbreak
            name: prompt_attack
      modelRefs:
        - model: guarded-model

Ranking has four steps. Match evaluates the rules, with on_unknown choosing no_match, match or fail_request when the result is still Unknown; fail_request returns a 503 whose message names the fix. Tier is the hard boundary and a lower tier wins. Rank applies routing.strategy: priority by default, or confidence. Explain is the part I would use daily: Preview and Replay show the strategy, the tier and why the winner beat the runner-up.

Confidence is stricter than it sounds. A pool ranks by confidence only when every non-catch-all match has exactly one evidence leaf and all of them share the same score kind, such as classifier probability or vector similarity. Complexity and jailbreak signals are never comparable. Otherwise the router quietly falls back to priority and warns at startup. The config validator also rejects malformed trees, for example: decision billing, rules.conditions[1]: NOT requires exactly one child condition, got 2.

Algorithms: picking one model, or several

An algorithm runs after a decision matches. Selection algorithms return one candidate from the decision's modelRefs. Looper algorithms make additional model calls through global.integrations.looper.endpoint, which the docs say adds latency and token usage and sends intermediate content to every configured worker. With a single candidate, omit the algorithm entirely.

The documented example uses latency_aware. Candidates are filtered by context, protocol and declared task capabilities before scoring, and a route that requires a minimum pool fails rather than quietly running a smaller panel. Note that latency observations are per process, so two router replicas can disagree about which model is fastest.

routing:
  decisions:
    - name: responsive-route
      description: Prefer the model with the best observed latency.
      priority: 100
      rules:
        operator: AND
        conditions: []          # empty: matches everything, so it acts as a catch-all
      modelRefs:
        - model: small-model
        - model: large-model
      algorithm:
        type: latency_aware
        minimum_candidates: 2   # fail the route if fewer than 2 distinct modelRefs survive
        latency_aware:
          tpot_percentile: 90
          ttft_percentile: 95

Plugins: behaviour that belongs to one route

Plugins live under routing.decisions[].plugins and apply only after that route matches. Shared services are defined once under global, and the plugin enables and tunes them per route. The documentation inventories fifteen plugin types, among them fast_response, system_prompt, header_mutation, tools, tool_selection, context_compression, response_cache, memory, rag, router_replay, shadow_dispatch, hallucination and response_jailbreak.

# From the plugin overview. The shared store is declared once under
# global.stores.response_cache; this route only switches the plugin on.
routing:
  decisions:
    - name: cached-support
      description: Reuse cached responses for support requests.
      priority: 100
      rules:
        operator: AND
        conditions: []
      modelRefs:
        - model: support-model
      plugins:
        - type: response_cache
          configuration:
            enabled: true
            ttl_seconds: 3600

The release post adds the details that decide whether you can trust these in production:

• Response Cache checks compatibility of history, tools, format, model and recipe before reusing an answer, and a negation check or a failed optional verifier turns a hit into a miss.
• Shadow Dispatch sends a bounded, sampled copy of a single-model request to a candidate without touching the live answer; Replay records the outcome, but answer quality needs separate evaluation.
• Context Compression is opt-in, protects instructions and tool-call pairs, and preserves the current user turn unless truncation is explicitly enabled.
• Router Memory extracts facts asynchronously on the response side and reports each as written, failed or dropped.

The models doing the routing

The classifiers are open models, which matters if you cannot send prompts to a third party. Vela 1.0 is a family of 15 open models on the Hugging Face organisation vllm-sr: eleven 307M-parameter encoders for tasks such as domain, feedback, modality, fact-check, guard, safety, hazard, shield, PII, hallucination spans and reranking, plus an embedding encoder, two multimodal Omni embedding models and a masked-language base. Shield was contributed by KR-Labs and brought the family from 14 to 15.

Decision 1.0 is a separate set of six open-weight scoring models that return inspectable distributions: Kai and Lex at 0.6B with a 1,024-token input budget, then Eos 0.8B, Sol 2B, Nox 4B and Lux 9B at 16,384 tokens. On the project's published 54-task evaluation, Lux-9B scores 76.94 and Nox-4B 73.09, against 81.05 for the hosted Jev reference. The post says Lux-9B leads the displayed open-model references. It also says native Decision integration in the Router is the next step, so these models are not yet part of the routing path you would deploy.

Where it breaks

The release post is unusually candid, and its caveats are the failure section of this article. Streaming responses are observed only after completion, so the Router cannot retract bytes already sent. Hallucination checks need supplied grounding context, and without it an answer cannot be verified. Fusion, Workflows and the prompt selector are experimental, and the support matrix lists hallucination policy, jailbreak error handling and response jailbreak among its experimental examples that are not qualified for production.

Fallback has a hard edge: cross-model fallback is opt-in and stops once a stream is committed or a tool side effect may have occurred. A request that already started streaming from the wrong model stays wrong. Classifier errors stay Unknown, and a weighted projection that depends on one withholds its score rather than treating it as zero, which is correct and also means your rules must handle the missing value.

I would add one more: Looper algorithms send intermediate content to every configured worker, so a Vault-style privacy objective and a panel of external models are in direct tension. Check where each worker runs before enabling Fusion on a route that carries sensitive input.

When it beats a static choice, and what to do on Monday

It earns its place when three things hold: your traffic genuinely mixes cheap and hard requests, you have at least two backends that are each qualified for part of that mix, and you can measure the outcome. If any one is missing, keep the static name. The project's own benchmark exists for the third condition. Here is the order I would work in.

1. Write down your traffic classes and the model you would pick for each by hand. That list is your static baseline and your first set of decisions.
2. Install the Router and point a non-critical client at the public model ID vllm-sr/auto. Start with one decision and one candidate, so nothing is selected yet.
3. Declare on_unknown on every decision that uses a classifier, and use Preview to check the choice before using Probe to check that a real backend answers.
4. Enable Shadow Dispatch on a sampled slice so Replay records what a candidate would have done, then grade those answers yourself, because the plugin does not.
5. Evaluate with sr-bench: tune on a quick slice, then claim savings only on the disjoint standard holdout.

# Install (stable channel), from the project README.
curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel stable

# List the benchmark adapters installed. This is the only command the
# sr-bench page states; planning and running are on its linked pages.
vllm-sr benchmark catalog

# Smoke-test a client against the virtual model. ROUTER_URL is yours:
# the docs I read do not name a default host or port.
curl "$ROUTER_URL/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"model": "vllm-sr/auto", "messages": [{"role": "user", "content": "ping"}]}'

On sr-bench: it covers nine benchmarks and a run missing any of them has no complete score. Smoke is 36 tasks, quick 740 and the standard holdout 3,183, with standard disjoint from quick. The page warns that public tasks are not contamination-free and previously seen GPQA labels need a retest disclosure even when a split is called holdout. It names no metrics and publishes no scores, so treat any savings figure as something you must produce yourself.

Semantic routing is a trade, not an upgrade: you swap a hard-coded model name for a policy you have to test, observe and operate. Take it when your traffic mix and a holdout measurement justify it, keep on_unknown explicit, and leave the experimental algorithms off production routes until the support matrix says otherwise.