Run Local LLMs with Ollama: Install, Models, and OpenAI-Compatible API

You want an LLM endpoint for your app, but you do not want to send every user message to a third party, and the per-token bill adds up fast when you iterate. Ollama solves both: it runs open models on your own machine and exposes them through an API that speaks OpenAI's protocol, so most existing code only needs a new base_url.

This is a hands-on guide: install, pick a model that fits your RAM, chat via CLI and REST, point an OpenAI SDK client at it, customize models with a Modelfile, and generate embeddings for RAG. Every command below matches the current official documentation (Ollama v0.33.x, September 2026).

Prerequisites

• Linux, macOS, or Windows machine (commands below are Linux-oriented; Windows follows the same flow with a PowerShell installer)
• RAM: 8GB is the practical floor for a 7B model, 16GB for 13B
• Disk: 3-50GB depending on the model (sizes in the table below)
• Docker optional, if you prefer containers

Step 1: Install

Linux and macOS:

curl -fsSL https://ollama.com/install.sh | sh

Windows PowerShell:

irm https://ollama.com/install.ps1 | iex

Verify:

ollama --version

Or run it in Docker:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

For NVIDIA GPUs in Docker you also need the NVIDIA Container Toolkit, then add --gpus=all to the run command. AMD GPUs use the :rocm tag. On Linux the installer registers a systemd service, and Ollama listens on 127.0.0.1:11434 by default.

Step 2: Pull and run a model

One command gets you chatting:

ollama run llama3.2

This pulls the 3B model (~2.0GB) and opens a REPL. Type /bye to exit. Inside the REPL you can change parameters on the fly, for example context size:

/set parameter num_ctx 8192

To download without chatting, use ollama pull. Popular options by size (download size and context window from the Ollama library, September 2026):

Model	Download	Context	RAM needed	Good for
qwen2.5:0.5b	398MB	32K	~2GB	smoke tests, low-end CPUs
llama3.2:1b	1.3GB	128K	~4GB	small machines
llama3.2 (3B)	2.0GB	128K	~8GB	default pick, general chat
qwen2.5 (7B)	4.7GB	32K	8-16GB	stronger quality, multilingual
qwen2.5:14b-instruct-q4_0	8.5GB	32K	~16GB	heavier local workloads
qwen2.5:72b	47GB	32K	~64GB	the realistic ceiling

Ollama's documentation is blunt about memory: 7B models want at least 8GB of RAM, 13B wants 16GB, 33B wants 32GB. Bonus for Indonesian readers: the Qwen family is genuinely strong in Bahasa Indonesia, which matters if your project serves that market.

Check what you have:

ollama list          # models you pulled
ollama ps            # what is loaded right now, and where it runs

ollama ps prints a PROCESSOR column: 100% GPU, 100% CPU, or a split like 48%/52% CPU/GPU. That one command answers most "why is it slow" questions immediately.

Step 3: Chat over the REST API

Ollama runs an HTTP server on 127.0.0.1:11434. Native endpoints: /api/chat for conversations, /api/generate for a single prompt, /api/embed for embeddings.

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    { "role": "user", "content": "Why is the sky blue?" }
  ],
  "stream": false
}'

stream: false makes curl wait for the complete answer instead of a token stream, which is easier to read the first time.

The official clients wrap this API. Python:

pip install ollama

from ollama import chat

response = chat(
    model='llama3.2',
    messages=[{'role': 'user', 'content': 'Why is the sky blue?'}],
)
print(response.message.content)

JavaScript:

npm i ollama

import ollama from 'ollama';

const response = await ollama.chat({
  model: 'llama3.2',
  messages: [{ role: 'user', content: 'Why is the sky blue?' }],
});
console.log(response.message.content);

Step 4: The OpenAI-compatible API, the useful part

Ollama implements part of the OpenAI API under /v1: /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings, and since v0.13.3 also /v1/responses. Point any OpenAI client at it and existing code mostly works:

curl http://localhost:11434/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "llama3.2",
    "messages": [{ "role": "user", "content": "Hello!" }]
  }'

With the standard openai package in Python:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # any value works; no auth is enforced locally
)

resp = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Summarize this in one line: ..."}],
)
print(resp.choices[0].message.content)

That api_key line confuses people: Ollama ignores it locally, but the SDK requires the field. Any string works.

Supported on /v1/chat/completions: streaming, JSON mode via response_format, reproducible outputs via seed, vision, tools (function calling), logprobs, and reasoning controls for thinking models.

Caveats:

• The OpenAI protocol has no field for context size. To raise it, create a Modelfile (Step 5) or set OLLAMA_CONTEXT_LENGTH.
• If a tool hardcodes a model name like gpt-3.5-turbo, copy your model to that name:

ollama cp llama3.2 gpt-3.5-turbo

• /v1/responses works, but only the stateless flavor: previous_response_id and conversation are not supported.

Step 5: Customize a model with a Modelfile

A Modelfile is the Dockerfile for models. The common use case: pin the system prompt and sampling parameters so behavior does not drift between calls.

FROM llama3.2
PARAMETER temperature 0.7
PARAMETER num_ctx 8192
PARAMETER top_k 40
SYSTEM Kamu asisten teknis berbahasa Indonesia. Jawab ringkas, tanpa basa-basi.

Build and run:

ollama create asisten-teknis -f ./Modelfile
ollama run asisten-teknis

Parameters worth knowing: temperature (default 0.8), top_p (0.9), top_k (40), min_p, num_predict, seed, stop. Raise num_ctx when your prompts are long; context consumes RAM, so 8192 is a sane default for most local setups. MESSAGE seeds example conversation history.

ollama show --modelfile llama3.2 prints the full blueprint of any model, which is the fastest way to learn the exact template a model expects.

Step 6: Embeddings for RAG

For retrieval workflows, pull an embedding model:

ollama pull nomic-embed-text   # 274MB

Embed via the native endpoint:

curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "The sky is blue because of Rayleigh scattering"
}'

The same model works through OpenAI-compatible /v1/embeddings, which means LangChain-style retrievers can consume it without custom glue. Those vectors drop straight into pgvector or Qdrant if you are building RAG.

Step 7: Server settings that matter

Three environment variables matter most once this stops being a toy:

• OLLAMA_HOST: the server listens only on 127.0.0.1 by default. To reach it from other machines on your LAN:

sudo systemctl edit ollama
# add under [Service]:
Environment="OLLAMA_HOST=0.0.0.0:11434"
sudo systemctl daemon-reload && sudo systemctl restart ollama

• OLLAMA_MODELS: where models are stored (Linux default: /usr/share/ollama/.ollama/models). Move it when your root disk is tight.
• OLLAMA_CONTEXT_LENGTH: the default context window for models that do not set their own. Raise it, watch RAM climb.

Memory behavior: models unload after 5 minutes idle by default. For an app that hits the API constantly, send keep_alive: -1 in the request so the model stays resident, or run ollama stop <model> to unload immediately. Preload a model with an empty request so the first real user does not eat a cold start.

Flash attention (OLLAMA_FLASH_ATTENTION=1) and a quantized KV cache (OLLAMA_KV_CACHE_TYPE=q8_0) cut memory as context grows; q8_0 uses roughly half the memory of f16 with barely any quality loss.

Security: nothing is authenticated by default. Binding OLLAMA_HOST=0.0.0.0 and exposing port 11434 to the internet without a reverse proxy and auth in front of it is a bad idea.

When to use Ollama vs alternatives

Ollama is the right choice when you want the fastest path from zero to a local model API, need OpenAI-protocol compatibility so existing clients keep working, or you are on a single machine or small team.

• llama.cpp is the inference engine Ollama builds on. Use it directly when you need to compile from source or squeeze performance out of unusual hardware. Ollama is llama.cpp plus model management and an API layer.
• LM Studio delivers the same experience (GGUF models, local server) with a heavier GUI focus. Pick Ollama for CLI-first, scriptable workflows.
• vLLM targets high-throughput serving of large models to many concurrent users on GPU servers, with continuous batching and paged attention. Overkill for a laptop, right for a production inference cluster.
• Cloud APIs (OpenAI, Anthropic) give you stronger models and no hardware, but per-token cost and your data leaves the machine. Local first for prototypes and sensitive data; cloud when the largest models matter more than the bill.

Next steps

• Wire Ollama into an agent loop: /v1 supports function calling, so your existing tool-use code runs against it unchanged.
• Build RAG with local embeddings (Step 6) plus a vector database for retrieval.
• Read the FAQ before exposing the server: proxy setup, ngrok, Cloudflare Tunnel, and OLLAMA_ORIGINS for browser extensions are all documented there.

That is the whole loop: install, pull, switch base_url, done. The model lives on your disk, the API sits on 11434, and the only bill is the electricity.

References

• Ollama GitHub repository
• Ollama OpenAI compatibility documentation
• Ollama Modelfile reference
• Ollama FAQ: memory, environment variables, server configuration