vllm-metal on Apple Silicon: a Local vLLM Server for Mac

I wanted to know whether the Mac on my desk could serve three coding sessions at once without the second one stalling behind the first. One prompt on one model is easy on Apple Silicon. The trouble starts when a long prompt lands beside short ones and every request waits for the slowest.

I did not benchmark vllm-metal myself. This is a close reading of the vLLM team's announcement of 22 September 2026, the repository's installation guide and model matrix, the v0.29.0 release notes and the open roadmap issue. Every number below is theirs, measured on their hardware, and I say where that matters.

On a Mac the problem is overlap, not raw speed

The announcement opens with the right diagnosis: local inference is straightforward until several requests overlap, and then time to first token (TTFT), memory growth and admission control become serving problems rather than model-execution problems. vllm-metal answers with the parts of vLLM that already solve those on GPUs: the V1 scheduler, paged KV block management, chunked prefill and the OpenAI-compatible frontend. MLX and Metal do the execution.

The split is clean. vLLM supplies the scheduler, the KV blocks, sampling and the frontend with streaming and tool-call parsing. mlx_lm supplies the model implementations, and vllm-metal reuses its weight loading and token-wise layers (RMSNorm, linear, MoE, MLP) unchanged, replacing only attention with a paged varlen Metal kernel ported from vLLM's unified Triton kernel. The consequence for you is practical: most model-specific code sits in one layer, and any client that speaks the OpenAI API needs no change.

Install it and serve a first model

You need Apple Silicon and macOS 15 or later. The installation guide says no compiler is required, because the install brings Python 3.12, vLLM core and prebuilt vllm-metal wheels. The Homebrew tap is the stable route, and the endpoint comes up at http://localhost:8000/v1.

# Apple Silicon and macOS 15 or later. No compiler: it installs prebuilt wheels.
brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

# Start small, with half of unified memory as the serving budget.
vllm serve Qwen/Qwen3.5-0.8B --gpu-memory-utilization 0.5

# In a second terminal. The endpoint is OpenAI-compatible under /v1.
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3.5-0.8B",
       "messages": [{"role": "user", "content": "Say hi"}]}'

# Upgrade and uninstall are Homebrew's job, not pip's.
brew update && brew upgrade vllm-metal
brew uninstall vllm-metal

# The alternative: the install script. It builds a dev release unless you ask
# for --stable, and keeps its own environment at ~/.venv-vllm-metal.
curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm-metal/main/install.sh \
  | bash -s -- --stable
source ~/.venv-vllm-metal/bin/activate   # again in every new shell

# Wrong: the installation guide states this is not supported.
# pip install vllm-metal

Two details in that block are easy to miss. The install script defaults to the latest development build and only installs a stable release if you add --stable. And pip is not an option: the guide states plainly that pip install vllm-metal is not supported, so a tutorial that tells you otherwise is out of date.

Set the memory budget before you set the model

On a Mac the GPU and your apps share one pool of unified memory, so vllm-metal gives you a budget knob, --gpu-memory-utilization. Like upstream vLLM it runs a warmup pass at startup to account for model weights, activations and temporary buffers, and the budget that remains becomes a fixed KV cache. It also caps MLX's reusable buffer cache so memory does not creep up while serving. A request that does not fit in the KV pool waits for pages to free up; it is not dropped.

The announcement's own examples use 0.5 for a 0.8B and a 4B model and 0.7 for the 27B hybrid on a 64 GB Mac, and its reproduction notes list 0.92 as vllm-metal's default. My reading, which is an inference and not a documented rule, is that the number trades two failures against each other: too high leaves macOS and your other apps short, too low shrinks the KV pool so concurrent long prompts queue.

Packed queries and paged KV are the actual difference

In mlx_lm's padded batches the attention queries have shape B, H, T_max, D, so every request is padded to the longest query in the batch, and MLX's scaled_dot_product_attention has no varlen interface. vllm-metal packs every scheduled query token into total_q, H, D with cu_seqlens marking request boundaries, runs the mixed prefill and decode step in one forward, and stores KV in fixed-size pages addressed by per-request block tables. The announcement compares seven engines on those three choices.

Does it matter? The team measured Qwen3.6-35B-A3B in 4-bit on batches of eight requests with roughly 6,000 prompt tokens in total and 20 output tokens each. Batch A has similar prompt lengths. Batch B has one long prompt among short ones. The table gives batch wall time in seconds.

The finding is that only the padded engine suffers: mlx_lm's batch time went from 4.52 s to 10.99 s once one 3,000-token prompt joined seven 430-token ones, a rise of 143%, while vllm-metal went from 3.87 s to 3.64 s. oMLX and llama.cpp were also flat but slower in absolute terms. Read it with the right weight: it is one model, one eight-request batch and 20 output tokens, published by the project that wins the table.

What the agent-load runs show, and what they do not

For realistic traffic the team used the agent split of SiliconBench: 100 multi-turn prompts of roughly 4.6K input tokens each, on an M5 Pro with 64 GB, all at 4-bit, with a fresh server at every concurrency level. The post cites a SiliconBench paper that evaluates nine Apple Silicon engines more broadly. I did not read that paper, so I am not leaning on it.

The results are mixed in a useful way. On Qwen3.8-27B, vllm-metal had the lowest TTFT and end-to-end latency at concurrency 2 and 4, but oMLX with SSD offload led at concurrency 1. On Gemma 4 E4B, swept to concurrency 16, vllm-metal kept TTFT low throughout. On Qwen3.6-35B-A3B at concurrency 4, vllm-metal and oMLX with a RAM cache were close on throughput and latency, with a larger gap in TTFT, and mlx_lm completed only a minority of requests.

The lesson is to read the concurrency 1 point first. If you are one developer with one agent, the announcement's own 27B result says a different engine can be faster. The case for vllm-metal is two or more overlapping sessions, which is exactly the shape of a shared team Mac or several parallel coding agents.

MTP speculative decoding helps one user, barely helps a crowd

Multi-token prediction (MTP) uses an assistant model to draft tokens that the target model verifies inside the continuous batch. It is opt-in through --speculative-config. The announcement compares one draft token per step with no MTP on Gemma 4 E4B, and the effect depends heavily on concurrency.

At concurrency 1 the wall time fell 15% and output throughput rose 20%. By concurrency 8 the gain had gone, and at 16 the average TTFT was 20% worse even though wall time was 8% better. So MTP is a single-user latency feature, and the envelope is narrow: Gemma 4 only, plain greedy sampling with temperature 0, and synchronous scheduling.

# Right: inside the documented envelope. Gemma 4, synchronous scheduling,
# one draft token per step, and a client that sends temperature 0.
vllm serve google/gemma-4-E4B-it \
  --gpu-memory-utilization 0.5 \
  --max-model-len 16384 \
  --no-async-scheduling \
  --speculative-config '{"method":"mtp","model":"mlx-community/gemma-4-E4B-it-assistant-bf16","num_speculative_tokens":1}'

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "google/gemma-4-E4B-it", "temperature": 0,
       "messages": [{"role": "user", "content": "Explain paged attention in two sentences."}]}'

# Wrong: each of these steps outside what the announcement says the Metal
# path supports, so do not build on it.
#   - a model family other than Gemma 4
#   - any sampling other than greedy (temperature above 0)
#   - leaving out --no-async-scheduling

If your clients send a non-zero temperature, MTP does not apply to them as documented. Check what your client actually sends before you plan around the 20% figure.

The trade-offs that decide whether you can rely on it

I did not run into a failure, because I did not run it, so this section lists what the sources themselves say is limited. Each row is stated in the announcement, the model matrix, the v0.29.0 release notes or the roadmap issue.

There is a structural caveat too. The project is described as a community-maintained hardware plugin, and the announcement lists an experimental PyTorch MPS backend as a roadmap item, with the roadmap issue reporting a gap of about 30% against MLX. The MLX path is the one to use today.

What to do on Monday

Do not adopt it on the strength of someone else's benchmark, including this post's sources. Spend an hour measuring your own workload.

1. Check the basics: Apple Silicon, macOS 15 or later, and enough unified memory for the model plus the KV budget. Install with Homebrew and serve one model you already use, at 0.5.
2. Look up your model in the supported matrix before downloading anything. Note whether it is a supported, experimental or unverified row.
3. Run the announcement's benchmark below at concurrency 1 and again at the concurrency you really expect, restarting the server between runs, and keep the successful-request count next to each result.
4. Run the same two points against whatever you use now. If concurrency 1 favours the incumbent and your real traffic is single-user, you have your answer.
5. Only then try MTP, only on Gemma 4, only with temperature 0, and pin the version once it works.

# Terminal 1: the announcement's cross-machine setup. Prefix caching is off so
# every run measures prefill, and the budget is fixed at 0.5.
vllm serve mlx-community/gemma-4-e4b-it-4bit \
  --gpu-memory-utilization 0.5 \
  --max-model-len 2048 \
  --no-enable-prefix-caching \
  --host 127.0.0.1 --port 8000

# Terminal 2: 100 Sonnet prompts, about 1,024 tokens in and 128 out.
curl -fsSL https://raw.githubusercontent.com/vllm-project/vllm/main/benchmarks/sonnet.txt \
  -o sonnet.txt

BENCH_MACHINE=my-mac-32gb
BENCH_CONCURRENCY=1     # rerun at 8, restarting the server each time

vllm bench serve \
  --backend vllm \
  --model mlx-community/gemma-4-e4b-it-4bit \
  --base-url http://127.0.0.1:8000 \
  --dataset-name sonnet --dataset-path sonnet.txt \
  --sonnet-input-len 1024 --sonnet-output-len 128 \
  --num-prompts 100 --num-warmups 3 \
  --request-rate 10 --max-concurrency "$BENCH_CONCURRENCY" \
  --temperature 0 --ignore-eos --seed 0 \
  --save-result --result-dir benchmark-results \
  --result-filename "$BENCH_MACHINE-c$BENCH_CONCURRENCY.json"

# Pointing a coding agent at it follows the vLLM Claude Code page. The model
# name must match --served-model-name and must not contain a slash, so serve
# with --served-model-name my-model rather than the raw Hugging Face id.
# That page documents plain vLLM; I did not check it against vllm-metal.
ANTHROPIC_BASE_URL=http://localhost:8000 \
ANTHROPIC_API_KEY=dummy ANTHROPIC_AUTH_TOKEN=dummy \
ANTHROPIC_DEFAULT_OPUS_MODEL=my-model \
ANTHROPIC_DEFAULT_SONNET_MODEL=my-model \
ANTHROPIC_DEFAULT_HAIKU_MODEL=my-model \
claude

One practical note for coding agents: the vLLM Claude Code page requires the model name in the environment variables to match --served-model-name, and that name must not contain a slash. A raw Hugging Face id such as Qwen/Qwen3.5-0.8B therefore needs an alias. That page documents plain vLLM and I did not verify it against vllm-metal.

A Mac becomes a credible shared inference box when it schedules requests like a server rather than running them like a script. vllm-metal brings that scheduler to Apple Silicon. Adopt it for overlapping sessions, keep MTP and hybrid caching in the experiment column, and let your own concurrency 1 and concurrency N numbers decide.