AI
Run Local LLMs with Ollama: Models, API, and Modelfile
July 20269 min read
Photo by Christian Wiediger on Unsplash
Ollama is an open-source runtime for running open-weight large language models locally on your own hardware. It bundles the model weights, an inference engine, and a local HTTP server behind a single binary, so you can pull and run models like Llama, Gemma, Qwen, and DeepSeek with one command instead of setting up CUDA and Python dependencies by hand.
Run a single command such as ollama run llama3.2. On the first run Ollama downloads the model weights and drops you into an interactive chat prompt; every run after that is instant because the model is cached on disk. Use ollama pull to fetch a model ahead of time and ollama list to see what you already have locally.
Yes. Ollama serves an HTTP API on port 11434, including a built-in endpoint compatible with the OpenAI Chat Completions API at http://localhost:11434/v1. Existing code written against the OpenAI SDK can point at your local server by changing the base URL; the API key is required by the client but unused by Ollama, so any value works.
A Modelfile is the blueprint for a customized model, similar in spirit to a Dockerfile. It starts FROM a base model and layers configuration: PARAMETER sets runtime options like temperature and context length, SYSTEM sets a persistent system prompt, and TEMPLATE controls the prompt format. You build a named model from it with ollama create.
Ollama's official README recommends at least 8 GB of RAM for 7B models, 16 GB for 13B models, and 32 GB for 33B models. The whole model must fit in memory, and quantization makes larger models feasible on consumer hardware by storing weights at lower precision. A GPU with enough VRAM is much faster but not required to start.
Photo by Christian Wiediger on Unsplash
Key Takeaway
Ollama runs open-weight large language models on your own hardware. One command pulls and runs models such as Llama, Gemma, Qwen, and DeepSeek. It exposes a local REST API plus an OpenAI-compatible endpoint, supports structured JSON outputs and tools, and keeps every prompt on your machine at zero per-token cost.
For a long time, calling a language model meant sending my prompt to someone else's server and paying per token. That is fine for a prototype, but it becomes a problem the moment the data is sensitive, the bill scales with traffic, or the network is unreliable. Ollama is the tool I reach for when I want a model running entirely on my own machine, with nothing leaving the box.
Ollama is an open-source runtime for open-weight models. It bundles the model weights, the inference engine, and a local HTTP server behind a single binary, so getting a capable model answering prompts takes one command instead of a weekend of CUDA and Python dependency wrangling. Everything below is drawn from the official docs and the Ollama blog, not from memory.
The core workflow is deliberately boring. You name a model, and Ollama downloads it, loads it, and drops you into a chat prompt. The same models you have read about in papers are available from the model library: Llama, Gemma, Qwen, DeepSeek, Mistral, and many more, each in several parameter sizes.
# Pull and run a model in one command
ollama run llama3.2
# Or just download it now, use it later
ollama pull qwen2.5
# See what you already have on disk
ollama listThe first run downloads the weights; every run after that is instant because the model is cached on disk. Pull without run when you want to fetch a model ahead of time, and list to see what you already have locally.
Model names carry a tag, like llama3.2:1b or llama3.2:3b, that selects a specific parameter size and quantization. Smaller tags load faster and fit in less memory; larger tags answer better. When you omit the tag, Ollama picks a sensible default variant for that model.
A Modelfile is the blueprint for a customized model. It is a small text file, similar in spirit to a Dockerfile, that starts FROM a base model and layers configuration on top: PARAMETER lines set runtime knobs like temperature and context length, SYSTEM sets a persistent system prompt, and TEMPLATE controls the full prompt format. ADAPTER, LICENSE, and MESSAGE instructions are available too.
# Modelfile
FROM llama3.2
PARAMETER temperature 0.6
PARAMETER num_ctx 8192
SYSTEM You are a terse senior backend engineer. Answer in at most three sentences.You build a named model from that file with ollama create, then run it like any other model. Because the Modelfile is plain text, it lives in Git next to your code, so a teammate gets the exact same system prompt and parameters without you explaining anything.
ollama create backend-helper -f ./Modelfile
ollama run backend-helperOnce Ollama is running, it serves an HTTP API on port 11434. There is a native REST API for running and managing models, and, crucially, a built-in OpenAI Chat Completions compatible endpoint. That means existing code written against the OpenAI SDK can point at your local server by changing only the base URL to end in /v1. The API key is required by the client but unused by Ollama, so any string works.
# OpenAI-compatible endpoint — drop-in for existing SDK code.
# The API key is required but unused; any value works.
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer ollama" \
-d '{
"model": "llama3.2",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Explain quantization in one line." }
]
}'This compatibility is the single biggest reason adoption is easy. Tools, agent frameworks, and IDE plugins that already speak the OpenAI protocol tend to work against Ollama with a one-line configuration change, so you can trial a local model in an existing app before committing to it.
Two features move Ollama from a chat toy to something you can build on. Structured outputs, added in December 2024, let you pass a JSON schema in the format field so the model is constrained to return valid JSON in exactly the shape you asked for. That is far more reliable than parsing free text and hoping the braces line up.
# Force the model to answer inside a JSON schema
curl http://localhost:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1",
"stream": false,
"messages": [{ "role": "user", "content": "List two Go web frameworks." }],
"format": {
"type": "object",
"properties": {
"frameworks": { "type": "array", "items": { "type": "string" } }
},
"required": ["frameworks"]
}
}'Tool calling, added in July 2024, lets a model request a function you defined. You pass a list of tools in the request, the model replies with a tool_calls block naming the function and arguments, your code runs it, and you feed the result back with a tool role message. Models such as Llama 3.1 were among the first to support this, and it also works through the OpenAI-compatible endpoint.
Ollama runs on a GPU when it can and falls back to CPU when it cannot, so it works on a plain laptop, just more slowly. The binding constraint is memory: the whole model has to fit, and quantization is what makes that feasible on consumer hardware by storing weights at lower precision. The official README gives clear minimums.
Bigger is not automatically better for your machine. A large model that spills out of GPU memory onto CPU RAM can crawl to a few tokens per second, while a smaller quantized model that fits entirely in VRAM feels instant. Match the model size and quantization to the memory you actually have, then move up only if quality demands it.
The choice is not local-good, hosted-bad. Each wins on different axes, and I mix both depending on the project. This is how the trade-offs line up in practice.
| Concern | Local (Ollama) | Hosted API |
|---|---|---|
| Data privacy | Prompts never leave your machine | Prompts sent to a third-party server |
| Cost model | One-time hardware, no per-token fee | Pay per token, scales with usage |
| Offline use | Works with no internet at all | Requires a network connection |
| Model ceiling | Bounded by your RAM and VRAM | Access to the largest frontier models |
| Latency | No network hop; bound by your hardware | Network round trip plus provider load |
| Operations | You manage updates and hardware | Provider handles scaling and uptime |
My rule of thumb: reach for Ollama when privacy, cost at scale, or offline operation matter, and reach for a hosted API when you need the absolute strongest model and do not want to own hardware. Start small, pull a 3B model tonight, point one existing script at localhost, and see how far a model running on your own machine actually gets you.