What Is Ollama? Run LLMs Locally, Explained

Ollama is a free, open-source tool that downloads and runs large language models directly on your own computer. Instead of sending every prompt to a cloud API, you pull a model once and run it locally — privately, offline, and at no per-token cost. One install and one command is the entire on-ramp, which is why Ollama has become the default first step for developers who want AI on their own hardware.

If you want the hands-on version, we have a full step-by-step Ollama guide. This article is the deeper “what is it, how does it actually work, and when should I use it” overview.

The problem Ollama solves

Running an LLM locally used to be genuinely painful. You had to find the model weights, convert them to the right format, figure out quantization (shrinking a model to fit in memory), wrangle GPU drivers, and wire up a server to talk to it. Each model had its own quirks, and a mistake at any step produced cryptic errors instead of a chat prompt.

Ollama collapses all of that into a single command. It packages the model, its configuration, and a runtime into one tool that works the same way across macOS, Linux, and Windows. The comparison people reach for is Docker, and it fits: the way Docker made containers approachable, Ollama made local models approachable — the same pull-and-run mental model, the same registry-of-images feel.

How Ollama works under the hood

Ollama is best understood as a packaging and serving layer on top of llama.cpp, the open-source C/C++ inference engine that made running transformer models on ordinary CPUs and consumer GPUs practical. llama.cpp does the heavy lifting — loading weights, running the math, generating tokens. Ollama wraps it with model management and a clean developer experience. A few pieces do most of the work:

• The GGUF format. Models in Ollama’s library ship as GGUF files: a single-file format that bundles the weights, tokenizer, and metadata a runtime needs. One file, zero assembly — this is why ollama run needs no setup step.
• Quantized weights. Models are distributed in quantized form — weights stored in 4 or 8 bits instead of the 16 they were trained in. That cuts memory use by roughly 2–4x with only a modest quality cost, which is the difference between “needs a data-center GPU” and “runs on a laptop.”
• Layered, content-addressed storage. Like Docker images, models are stored as layers. Two variants that share base weights don’t store them twice, and pulling an update only downloads what changed.
• A local API server. Ollama runs a background service on localhost:11434 and exposes an OpenAI-compatible endpoint. Most code and tools written for the OpenAI API work against your local model by changing only the base URL.
• GPU offload with CPU fallback. Ollama splits a model between GPU memory and system RAM automatically. A model that doesn’t fit entirely in VRAM still runs — just slower — instead of failing outright.

# Pull and chat with a model in your terminal
ollama run llama3.1

# Or hit the local API from any app
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{ "role": "user", "content": "Hello" }]
}'

The model library and Modelfiles

You pull models by name from Ollama’s library — ollama run llama3.1 fetches Meta’s Llama, and the popular open families include Llama, Mistral, Qwen, Gemma, Phi, and DeepSeek, each in several sizes. Names carry tags for size and quantization level (llama3.1:8b, llama3.1:70b), so you choose the variant that fits your machine rather than a one-size build.

Customization happens through a Modelfile — a short text file, similar in spirit to a Dockerfile, that bakes a system prompt and parameters into a reusable variant:

FROM llama3.1
PARAMETER temperature 0.3
SYSTEM You are a concise code-review assistant. Answer in bullet points.

Run ollama create reviewer -f Modelfile, and from then on ollama run reviewer gives you that behavior every time, no prompt-pasting required. Teams use this to standardize local assistants the same way they standardize container images.

What hardware do you need?

Memory is the constraint that matters. As a rough rule, a model needs about its download size in available RAM or GPU VRAM, plus headroom for context. By parameter count, the durable size classes look like this:

Model class	Typical quantized size	Runs comfortably on
1–4B	1–3 GB	Almost any modern laptop
7–14B	4–9 GB	16 GB RAM or an 8 GB+ GPU
30–70B	18–40 GB	24 GB+ VRAM, or Apple Silicon with 32–64 GB unified memory

Two practical notes. First, Apple Silicon Macs punch above their weight because unified memory lets the GPU address all system RAM — a 64 GB MacBook can load models that would otherwise demand a workstation GPU. Second, generation speed tracks memory bandwidth more than raw compute, so if a model is slow or won’t load, step down a size class or pick a more aggressively quantized tag. The small-language-model tier has improved enough that even a 4B model now handles summarization, classification, and drafting credibly.

Ollama vs llama.cpp vs LM Studio

All three run the same GGUF models — llama.cpp is the engine, and the other two are different vehicles built around it.

	Ollama	llama.cpp	LM Studio
Interface	CLI + local API	Library + example binaries	Desktop GUI
Setup	One installer	Build and configure yourself	One installer
Model management	Built-in registry, pull by name	Manual GGUF downloads	Built-in model browser
Open source	Yes	Yes	Free app, proprietary
Best for	Developers, scripts, self-hosted apps	Maximum control, embedding in software	Non-technical users, desktop chat

Choose by interface: API-first automation points to Ollama, a friendly chat window points to LM Studio, and total control (or embedding inference inside your own software) points to raw llama.cpp.

When to use Ollama vs a cloud API

Local wins when the constraint is privacy, cost, or connectivity:

• Privacy and compliance. Prompts and data never leave your machine — essential for sensitive, regulated, or proprietary work.
• Cost. After the hardware you already own, inference is free. No metered billing, no rate limits — which changes how freely you experiment.
• Offline. It works with no internet connection at all.
• Prototyping. Swapping models locally is fast and consequence-free, which makes Ollama a natural local backend for a retrieval-augmented generation pipeline before you commit to hosted inference.

The cloud wins when the constraint is capability or scale: a model running on consumer hardware won’t match a hosted frontier model on the hardest reasoning tasks, and serving many concurrent users is what hosted platforms are built for. The gap has narrowed dramatically as open-weight models have caught up, and plenty of teams land on a hybrid — local models for development, testing, and private data; a hosted API for production traffic.

What Ollama is not

Three common mix-ups worth clearing up:

• Ollama is not a model. It’s the tool that runs models. “Ollama” doesn’t answer your questions; the Llama, Qwen, or Gemma model you pulled does.
• It’s not a training tool. Ollama runs inference. Fine-tuning happens elsewhere — though you can import the result as a GGUF file and serve it with Ollama afterward.
• It’s not a production serving stack. It’s optimized for one machine and a handful of users. High-throughput, multi-GPU serving for thousands of concurrent requests is a different problem with different tools.

The takeaway

Ollama is the easiest way to go from “I’d like to run AI locally” to actually doing it — one install, one run command, and an OpenAI-compatible API ready for whatever you build. Under the hood it’s a well-packaged llama.cpp: GGUF models, quantized weights, Docker-style layered storage, and a local server. Use it when privacy, cost, or offline operation matters; reach for a hosted model when a task truly demands frontier horsepower. To get it running, follow the hands-on Ollama tutorial.