Build a Private Local RAG Pipeline with Ollama + pgvector

You follow any RAG tutorial and it sends your documents to an API provider. Great when the corpus is public. A problem when it is your contracts, your support tickets, your internal runbooks. Every chunk leaves your machine and comes back as a token bill.

A local pipeline changes that. Embeddings, vector search, and the LLM all run on your own hardware, free, with nothing uploaded anywhere. This guide builds one end to end with Ollama and pgvector, two open source tools, and it runs on a normal laptop.

Prerequisites

• Ollama installed and running (https://ollama.com)
• Docker, so we can use the official pgvector image
• About 4GB of free RAM and a few GB of disk
• Python 3 with psycopg2 and requests

pip install psycopg2-binary requests

Step 1: Run pgvector

pgvector is one Postgres image with a vector extension baked in. You do not build anything; you just run it.

docker run -d \
  --name pgvector \
  -e POSTGRES_PASSWORD=localrag \
  -e POSTGRES_DB=ragdb \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql/data \
  pgvector/pgvector:pg16

The -v pgdata:... named volume keeps your data when the container restarts.

Step 2: Create the table and the index

docker exec -it pgvector psql -U postgres -d ragdb

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE documents (
  id SERIAL PRIMARY KEY,
  source TEXT,
  chunk_index INT,
  content TEXT,
  embedding vector(768)
);

CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);

The column is vector(768) because that is what the embedding model below outputs. The index uses vector_cosine_ops, so search runs "nearest by cosine distance".

Two index options exist. IVFFlat needs the table populated before you build the index, because it trains a set of cluster lists. HNSW does not. For new RAG setups use HNSW and only switch to IVFFlat when memory is tight. They share the same operators, so the swap is hnsw to ivfflat with WITH (lists = 100).

Step 3: Pull the models

ollama pull nomic-embed-text
ollama pull llama3.2

nomic-embed-text is a 137M-parameter encoder, 768 dimensions, about 274MB. It is embeddings only. Do not try to chat with it. llama3.2 is the small chat model we use for answers, and it runs on CPU.

Step 4: Ingest documents (ingest.py)

Two details matter before the code.

First, the API endpoint. Ollama has /api/embed (current, batch capable) and the older /api/embeddings, which the docs mark as superseded. Use /api/embed.

Second, the task prefix. nomic-embed-text was trained with prefixes in front of text: search_document: on what you store, search_query: on questions you search with. Ollama passes your input through verbatim and will not add them for you. Skip them and retrieval quality quietly drops.

import os
import sys
import requests
import psycopg2

OLLAMA_URL = "http://localhost:11434"
EMBED_MODEL = "nomic-embed-text"
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50

DB = psycopg2.connect(
    host="localhost", port=5432,
    dbname="ragdb", user="postgres", password="localrag",
)

def embed(texts):
    r = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": EMBED_MODEL, "input": texts},
    )
    r.raise_for_status()
    return r.json()["embeddings"]

def chunk(text):
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + CHUNK_SIZE])
        start += CHUNK_SIZE - CHUNK_OVERLAP
    return chunks

def ingest_file(path):
    text = open(path).read()
    chunks = chunk(text)
    vectors = embed(["search_document: " + c for c in chunks])
    cur = DB.cursor()
    for i, (c, v) in enumerate(zip(chunks, vectors)):
        cur.execute(
            "INSERT INTO documents (source, chunk_index, content, embedding) "
            "VALUES (%s, %s, %s, %s)",
            (path, i, c, v),
        )
    DB.commit()
    print(f"  {path} -> {len(chunks)} chunks")

if __name__ == "__main__":
    for path in sys.argv[1:]:
        ingest_file(path)

Run it with your markdown and text files:

python ingest.py README.md notes/runbook.md

Note the batch: /api/embed takes a list, so one HTTP call covers every chunk of a file. This is the single biggest speed win in any RAG ingest.

Step 5: Query it (query.py)

The query side reuses the same embed call, searches by cosine distance with <=>, then hands the top chunks to the chat model.

import sys
import requests
import psycopg2

OLLAMA_URL = "http://localhost:11434"
EMBED_MODEL = "nomic-embed-text"
LLM_MODEL = "llama3.2"
TOP_K = 4

DB = psycopg2.connect(
    host="localhost", port=5432,
    dbname="ragdb", user="postgres", password="localrag",
)

def embed(text):
    r = requests.post(
        f"{OLLAMA_URL}/api/embed",
        json={"model": EMBED_MODEL, "input": [text]},
    )
    r.raise_for_status()
    return r.json()["embeddings"][0]

def retrieve(question):
    vec = embed("search_query: " + question)
    cur = DB.cursor()
    cur.execute(
        "SELECT content FROM documents "
        "ORDER BY embedding <=> %s::vector LIMIT %s",
        (vec, TOP_K),
    )
    return [row[0] for row in cur.fetchall()]

def ask(question):
    context = "\n\n---\n\n".join(retrieve(question))
    prompt = (
        "Answer using only the context below. If the context lacks the "
        "answer, say so.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\nAnswer:"
    )
    r = requests.post(
        f"{OLLAMA_URL}/api/generate",
        json={"model": LLM_MODEL, "prompt": prompt, "stream": False},
    )
    r.raise_for_status()
    return r.json()["response"]

if __name__ == "__main__":
    print(ask(" ".join(sys.argv[1:])))

<=> is the cosine-distance operator in pgvector, and it returns distance rather than similarity, so smaller is better and ORDER BY ... LIMIT picks the closest chunks.

python query.py "What does the deploy script do?"

Tuning knobs

• Chunk size. 500 characters with 50 overlap is a sane start; documents with long sections can go up to 800. nomic-embed-text accepts up to 8192 tokens, so oversized chunks are usually not the failure you think. Test 400 and 800 and pick the one that answers better.
• top_k. 3 to 6 chunks is enough for most questions. More context dilutes the answer.
• Silent truncation. /api/embed clips input to the model context by default. Pass truncate: false to get an error instead of a silently cut vector.

Common pitfalls

Problem	Cause	Fix
Empty results	Vector dimension does not match column	Embedding must match vector(768) exactly
Weak retrieval	Missing task prefix	search_document: on store, search_query: on query
KeyError embedding	Old code against new endpoint	/api/embed returns embeddings (plural)
Truncated vectors	truncate defaults to true	Set truncate: false

Where to go next

• Filter by metadata before the search, for example WHERE source = 'runbook.md', to scope retrieval to one set of documents.
• Swap in the multilingual nomic-embed-text-v2-moe if your corpus is not English.
• Add a reranker or a simple MMR step if near-duplicate chunks crowd the top results.
• Batch ingest across a folder and keep the model warm with ollama serve.

References

• Ollama API documentation, /api/embed
• pgvector on GitHub
• nomic-embed-text on the Ollama library