# Running Local LLMs with Ollama in Production: What I Learned

> A practical guide to deploying Ollama for production local LLM inference — covering model selection, hardware requirements, API integration, and when local beats cloud.

URL: https://www.matthewswong.com/en/blog/local-llm-ollama-production/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2025-06-10
- Updated: 2026-10-17
- Category: AI

Running LLMs locally used to require a machine learning PhD and a rack of GPUs. Ollama changed that. It is a single binary that manages model downloads, quantization, and a local OpenAI-compatible API endpoint. Local inference is especially attractive for internal tooling where data privacy requirements make cloud LLM APIs complicated — sending employee or client data to OpenAI or Anthropic raises compliance questions that a legal team needs time to evaluate. Running the model locally sidesteps that question. This guide covers a practical setup: model selection, hardware sizing, API integration, and the honest trade-offs versus cloud APIs.

## Why Local LLMs Are Now Practical

The quantization revolution is why local LLMs are viable in 2025. A 70B parameter model like Llama 3.1 70B, quantized to 4-bit (Q4_K_M), requires about 40GB of RAM and can run on a workstation with 64GB unified memory or two consumer GPUs. The 8B models like Llama 3.1 8B run comfortably on 8GB VRAM — a single RTX 3080 or Apple M2 Pro. Quality for most business tasks (summarization, classification, structured extraction, Q&A over documents) is now within 85-90% of GPT-4-class models. For tasks like code completion, formatting ERP data, and generating reports, the gap is even smaller.

### Model Selection Guide

For production local inference, I use: Llama 3.1 8B (Q4_K_M) for fast, latency-sensitive tasks like chat and quick lookups — 2-4 tokens/second on CPU, 20-40 tokens/second on GPU. Llama 3.1 70B (Q4_K_M) for complex reasoning, code generation, and document analysis — runs on our dev server with 128GB RAM. Mistral NeMo 12B for structured extraction tasks — excellent at following JSON schemas. Qwen2.5-Coder 7B for code-specific tasks — beats larger general models on coding benchmarks at a fraction of the compute cost. Always use the -Instruct or -Chat variants, not the base models.

> From my experience running Ollama in production on a DigitalOcean GPU Droplet: preload your most frequently used models at startup to avoid cold-start latency. Ollama keeps models in memory if you set OLLAMA_KEEP_ALIVE=24h in the environment. The first request after loading a model takes 3-8 seconds for the initial load; subsequent requests are nearly instant. For latency-sensitive applications, implement a warm-up call on server startup.

## Production Setup and Configuration

Running Ollama as a production service rather than a developer tool requires several additional steps: systemd service configuration, resource limits, authentication proxy, and monitoring. The default Ollama setup has no authentication — anyone on your network can call the API. For production, always put an authenticated nginx reverse proxy in front.

### Nginx Auth Proxy for Ollama

The configuration I use puts Ollama behind nginx with HTTP basic auth for simple internal deployments, or an API key validation proxy for team deployments. For enterprise use, a more robust solution is an OpenAI-compatible proxy like LiteLLM that handles authentication, rate limiting, model routing, and cost tracking across both local and cloud models from a single endpoint.

> **Hardware Sizing Warning**

> Running out of VRAM during inference causes the model to fall back to CPU computation, which is 10-20x slower. This is not obvious from logs — the API still responds, just extremely slowly. If you are seeing unexpectedly slow inference (under 2 tokens/second for an 8B model), check GPU memory usage with nvidia-smi. The rule of thumb: Q4 quantized model size in GB plus 2GB overhead must fit in VRAM. For a 13B Q4 model (~8GB), you need at least 10GB VRAM. Size your hardware with headroom — running at 95% VRAM utilization causes cache thrashing.

## API Integration: Drop-in OpenAI Replacement

Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1, which means any OpenAI SDK works without modification. Switch base_url and model name, and your existing code runs against local models. This is the fastest path to local inference for existing applications. I use this to A/B test local vs cloud responses for the same prompts, measure quality differences, and gradually migrate cost-sensitive workloads to local models.

## When Local Beats Cloud

Local LLMs win when: data privacy requirements prevent sending data to third parties, you have high volume repetitive tasks (classification, extraction) where API costs add up, latency requirements need sub-100ms responses (possible with 7B models on GPU), or you need guaranteed availability without API rate limits. Cloud LLMs win when: you need the best possible quality (GPT-4o, Claude Opus), tasks require the largest context windows, you need built-in multimodal capabilities, or your team lacks the infrastructure expertise to maintain local deployments.

## My Production Stack

A reference setup: Ollama running on a dedicated server with plenty of RAM and a GPU with ample VRAM. A sensible model split is Llama 3.1 8B Instruct for primary chat, Qwen2.5-Coder 7B for code tasks, and Mistral NeMo 12B for document extraction. Put nginx in front for authentication with API key validation. Monitor with Prometheus and a custom Ollama exporter tracking request latency, token throughput, and model load times. The compute cost is fixed server depreciation with zero per-token costs, so whether it beats a cloud API on price depends on your own token volume.

## Sources & Further Reading

- Ollama — Official Documentation — https://ollama.com/docs
- Meta — Llama 3 Model Card — https://ai.meta.com/blog/meta-llama-3/
- LiteLLM — OpenAI-compatible proxy for 100+ LLMs — https://github.com/BerriAI/litellm

## Source links

- https://ollama.com/docs
- https://ai.meta.com/blog/meta-llama-3/
- https://github.com/BerriAI/litellm

## FAQ

### Why use Ollama for local LLM inference instead of cloud APIs like OpenAI or Anthropic?

The primary driver is usually data privacy — sending employee or client data to cloud providers raises compliance questions that local inference sidesteps. Beyond privacy, local inference eliminates per-token costs, though whether it is cheaper than a cloud API depends on your usage volume.

### What hardware do you need to run Llama 3.1 8B or 70B locally in production?

The 8B model (Q4_K_M quantization) runs on 8GB VRAM — a single RTX 3080 or Apple M2 Pro is sufficient. The 70B model requires about 40GB of RAM and fits on a workstation with 64GB unified memory or two consumer GPUs. The key rule: Q4 model size in GB plus 2GB overhead must fit in VRAM, or inference silently falls back to CPU at 10–20x slower speeds.

### How do you handle authentication since Ollama has no built-in auth?

The default Ollama setup has no authentication, so the recommendation is to always put an authenticated nginx reverse proxy in front for production deployments. For team use, an OpenAI-compatible proxy like LiteLLM adds authentication, rate limiting, model routing, and cost tracking across both local and cloud models from a single endpoint.

### How do you avoid cold-start latency when Ollama loads a model for the first time?

Set OLLAMA_KEEP_ALIVE=24h in the environment so Ollama keeps models in memory between requests. Additionally, preload the most frequently used models at server startup and implement a warm-up call on startup. The first request after loading a model takes 3–8 seconds; subsequent requests are nearly instant.

### When do local LLMs lose to cloud models, and when do they win?

Local LLMs win for data-privacy-sensitive workloads, high-volume repetitive tasks like classification and extraction where API costs accumulate, sub-100ms latency requirements on GPU, and when guaranteed availability without rate limits matters. Cloud models win when you need the absolute highest quality (GPT-4o, Claude Opus), the largest context windows, built-in multimodal capabilities, or when your team lacks the infrastructure expertise to maintain local deployments.

---

- HTML version: https://www.matthewswong.com/en/blog/local-llm-ollama-production/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/local-llm-ollama-production/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json