Running LLMs Locally with Ollama: A Practical Guide

You don’t need an API key or an internet connection to run a capable large language model anymore. Thanks to a wave of strong open-weight models and a tool called Ollama, you can pull a model onto your laptop and start chatting in a couple of minutes. This guide walks through installing Ollama, running your first model, using its local API, and customizing a model for your own needs.

Why run an LLM locally?

Hosted models are convenient, but local models have real advantages:

• Privacy. Your prompts and data never leave your machine — essential for sensitive or regulated work.
• Offline. No network, no problem. Useful on planes, in secure environments, or when an API is down.
• Cost. After the hardware you already own, inference is free. No per-token billing.
• Experimentation. Swap models, tweak parameters, and build prototypes without rate limits.

The tradeoff is capability and speed: a model running on your laptop won’t match a frontier model like Claude Fable 5 on the hardest tasks. But for many jobs — drafting, summarizing, classification, coding help, local RAG — a good open model is more than enough.

Step 1: Install Ollama

Ollama runs on macOS, Linux, and Windows. On macOS or Linux, the install script is the quickest path:

curl -fsSL https://ollama.com/install.sh | sh

On macOS you can also download the app, and on Windows there’s a native installer. Once it’s set up, Ollama runs a small background service that manages models and serves a local API.

Step 2: Run your first model

One command pulls a model and drops you into a chat:

ollama run llama3.1

The first run downloads the model (a few gigabytes), then you’re talking to it in your terminal. Type a question, get an answer, and use /bye to exit. Want something else? Browse the model library at ollama.com and swap the name — popular open families include Llama, Mistral, Qwen, and Gemma, in a range of sizes.

Step 3: Manage your models

A handful of commands cover day-to-day use:

ollama list            # show installed models
ollama pull mistral    # download without running
ollama rm llama3.1     # remove a model to free disk space

Most models come in several sizes (often labeled by parameter count, like 8b or 70b) and are quantized — compressed so they fit in less memory at a small quality cost. A smaller, more aggressively quantized model runs on modest hardware; a larger one needs more RAM or a capable GPU but answers better.

Step 4: Use the local API

Ollama exposes an HTTP API on localhost:11434, which is what makes it useful for building apps. Here’s a direct call:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1",
  "messages": [{ "role": "user", "content": "Explain RAG in one sentence." }],
  "stream": false
}'

Crucially, Ollama also offers an OpenAI-compatible endpoint at http://localhost:11434/v1. That means most libraries and tools written for the OpenAI API work against your local model by changing just the base URL — a painless way to develop against a free local backend and switch to a hosted model later.

Step 5: Customize with a Modelfile

To bake in a system prompt or default parameters, create a Modelfile:

FROM llama3.1

SYSTEM "You are a terse senior engineer. Answer in at most three sentences."
PARAMETER temperature 0.5

Then build and run your variant:

ollama create terse-eng -f Modelfile
ollama run terse-eng

Now you have a reusable, preconfigured model — handy for giving a project its own assistant personality.

A note on hardware

The single biggest factor is memory. As a rough guide, a model needs roughly its file size in available RAM (or VRAM on a GPU) to run smoothly. A small quantized model is happy on a modern laptop; large models want a discrete GPU or a machine with plenty of unified memory. If a model is sluggish or won’t load, step down a size or pick a more compressed quantization.

Where it fits

Local models shine when paired with your own data. Point a retrieval-augmented generation pipeline at a local Ollama model and you get a private assistant that answers from your documents without a single byte leaving your machine.

The takeaway

Ollama makes local LLMs genuinely easy: install, ollama run, and you’re going — with an OpenAI-compatible API ready for whatever you build next. Use it for privacy-sensitive work, offline development, and cost-free experimentation, and reach for a hosted frontier model only when a task truly demands the extra capability.