You search your team's runbook for "my database keeps running out of connections" and get nothing. The doc that answers that question is titled "PostgreSQL connection pool exhaustion". Keyword search matches words, and your words never appear in it. The meaning matches, the text doesn't.
That gap is what semantic search exists for. Instead of matching tokens, it turns the query and the documents into vectors and returns the ones that sit closest together. Two sentences that mean the same thing end up near each other even when they share zero words.
Chroma is an open source vector database (Apache 2.0) that makes this painless. Install one package, add documents, search. Everything in this post was run for real: the snippets are the exact code I executed, and the outputs are what the code printed.
Prerequisites
- Python 3.9 or newer. That's chromadb's requirement, checked on PyPI.
pip install chromadb. Version 1.5.9 is current as of Aug 2026.- About 80 MB of disk for the default embedding model. Chroma downloads it automatically on first use.
- I verified every example on chromadb 0.6.3 with Python 3.11. The current docs describe the same API for everything used here, but pin your own version and test before trusting a tutorial.
1. Install and run your first query
pip install chromadb
The fastest setup is the in-memory client. Data lives inside the process and dies with it, which makes it perfect for experiments:
import chromadb
client = chromadb.Client()
collection = client.get_or_create_collection(name="ops_runbook")
collection.add(
ids=["doc-0", "doc-1", "doc-2", "doc-3", "doc-4", "doc-5", "doc-6", "doc-7"],
documents=[
"PostgreSQL connection pool exhaustion: check max_connections and increase pool_size in pgbouncer",
"Docker build fails with 'no space left on device': prune old images with docker image prune -a",
"Kubernetes pod stuck in CrashLoopBackOff: check liveness probe timeout and container logs",
"Nginx 502 Bad Gateway: upstream service is down or the timeout is too low in the location block",
"Redis OOM: set maxmemory policy to allkeys-lru and monitor with INFO memory",
"Disk full on CI runner: clean /tmp and Docker build cache in the pipeline",
"Slow page loads: check TTFB, then query times, then CDN cache hit ratio",
"SSL certificate expired: renew with certbot and reload nginx without downtime",
],
metadatas=[
{"service": "postgres", "severity": "high"},
{"service": "docker", "severity": "medium"},
{"service": "kubernetes", "severity": "high"},
{"service": "nginx", "severity": "medium"},
{"service": "redis", "severity": "high"},
{"service": "ci", "severity": "medium"},
{"service": "web", "severity": "low"},
{"service": "nginx", "severity": "high"},
],
)
results = collection.query(
query_texts=["my database keeps running out of connections"],
n_results=2,
)
That's a mini ops runbook. Every entry carries metadata: which service it's about, and how bad the situation usually gets. Querying it with a paraphrase of the Postgres doc gives:
collection count: 8
query: 'my database keeps running out of connections'
rank 1: PostgreSQL connection pool exhaustion: check max_connections and increase pool_size in pgbouncer
distance=1.0598
rank 2: Nginx 502 Bad Gateway: upstream service is down or the timeout is too low in the location block
distance=1.5853
The query never says "PostgreSQL", "pool", or "exhaustion", and the winning document never says "database". The match is on meaning, not words. Three details worth knowing:
add()needs unique string ids. If an id already exists, the record is ignored without an error, so for idempotent ingest useupdate()orupsert()instead.query()embeds your query text with the collection's embedding function automatically. You rarely touch vectors directly.- Distances are Euclidean (L2), and lower means more similar. There's no similarity threshold: you always get
n_resultsback, 10 by default.
2. Metadata filtering
Searching everything at once is rarely what production wants. Usually you want the most similar documents from one slice of the data. That's the where argument:
results = collection.query(
query_texts=["certificate about to expire"],
n_results=3,
where={"service": "nginx"},
)
Only entries whose service metadata equals "nginx" are candidates now:
-- where={'service': 'nginx'} --
SSL certificate expired: renew with certbot and reload nginx without downtime
Nginx 502 Bad Gateway: upstream service is down or the timeout is too low in the location block
The metadata operators are $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, and $and/$or for combining clauses. {"service": "nginx"} is sugar for {"service": {"$eq": "nginx"}}. Filtering by severity before ranking is a handy pattern:
results = collection.query(
query_texts=["everything is on fire"],
n_results=5,
where={"severity": {"$in": ["high"]}},
)
The filter runs inside the query, so your top-k slots don't get wasted on documents the user can't use:
-- where={'severity': {'$in': ['high']}} --
Kubernetes pod stuck in CrashLoopBackOff: check liveness probe timeout and container logs
PostgreSQL connection pool exhaustion: check max_connections and increase pool_size in pgbouncer
Redis OOM: set maxmemory policy to allkeys-lru and monitor with INFO memory
SSL certificate expired: renew with certbot and reload nginx without downtime
3. Full-text search on the document body
Sometimes you need keyword matching on the text itself, not similarity. where_document filters on document content:
results = collection.get(where_document={"$contains": "docker"})
Output:
-- where_document={'$contains': 'docker'} --
Docker build fails with 'no space left on device': prune old images with docker image prune -a
Operators: $contains, $not_contains, $regex, $not_regex, combinable with $and and $or.
The catch: full-text search is case-sensitive. My corpus also contains "Docker build cache" with a capital D, and "$contains": "docker" skips it. If your users type lowercase, normalize at ingest time, or reach for $regex when you need that control.
4. Persistence with PersistentClient
The in-memory client forgets everything when the process exits. For anything real, use PersistentClient, which stores data in a local SQLite-backed directory:
client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_or_create_collection(name="ops_runbook")
Same API, but the data survives restarts. get() retrieves records without similarity ranking, with limit/offset pagination:
results = collection.get(limit=3, offset=0)
print(results["ids"])
# ['doc-0', 'doc-1', 'doc-2']
Updates and deletes:
collection.update(
ids=["doc-0"],
documents=["PostgreSQL pool exhaustion: raise max_connections, add pgbouncer pool sizing"],
metadatas=[{"service": "postgres", "severity": "high"}],
)
collection.delete(ids=["doc-7"])
print(collection.count()) # 7
update() replaces documents and metadata for existing ids. upsert() inserts or replaces, which makes it the right call for idempotent ingest jobs.
5. Client-server mode
When several processes or machines need the same index, run Chroma as a server. The pip package ships the CLI:
chroma run --path /db_path
Connect with the HTTP client:
import chromadb
client = chromadb.HttpClient(host="localhost", port=8000)
I ran exactly this while writing the post: server on localhost:8000, heartbeat endpoint returning 200, and a query through HttpClient coming back with the right document. There's also an async variant, AsyncHttpClient, with the same method signatures. Deployment options, including the official Docker image, are in the server docs.
6. Embedding functions
The default embedding function is Sentence Transformers' all-MiniLM-L6-v2, running locally through ONNX. It produces 384-dimensional vectors, and Chroma downloads the model files (~80 MB) on first use, then caches them. For prototyping and small corpora it's a solid default: no API key, no data leaving your machine, and it gets out of the way.
When quality matters more, swap in a hosted model:
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
collection = client.create_collection(
name="my_collection",
embedding_function=OpenAIEmbeddingFunction(model_name="text-embedding-3-small"),
)
That sends your text to OpenAI, so think about where your data can go. Custom embedding functions work too: implement EmbeddingFunction and decorate it with register_embedding_function. One constraint: all vectors in a collection share one dimension, and query_embeddings must match it. Change the embedding function and you re-embed the collection.
7. A retrieval pipeline for RAG
The part Chroma owns in a RAG app is retrieval: turning a question into the top-k most relevant documents. A small reusable function:
def retrieve(query, top_k=3, where=None):
res = collection.query(query_texts=[query], n_results=top_k, where=where)
blocks = []
for i, doc in enumerate(res["documents"][0], start=1):
meta = res["metadatas"][0][i - 1]
blocks.append(f"[{i}] (service={meta['service']}, severity={meta['severity']}) {doc}")
return "\n".join(blocks), res["distances"][0]
Call it:
query = "the app is slow and I don't know where to start looking"
context, distances = retrieve(query, top_k=3)
print("DISTANCES:", [round(d, 3) for d in distances])
print(context)
DISTANCES: [1.321, 1.638, 1.797]
[1] (service=web, severity=low) Slow page loads: check TTFB, then query times, then CDN cache hit ratio
[2] (service=nginx, severity=medium) Nginx 502 Bad Gateway: upstream service is down or the timeout is too low in the location block
[3] (service=kubernetes, severity=high) Kubernetes pod stuck in CrashLoopBackOff: check liveness probe timeout and container logs
First hit is "Slow page loads", exactly the entry that runbook should surface. From here the context string goes into an LLM call. That step needs your own API key, so it's commented out in my code, but it's only two calls:
# from openai import OpenAI
# llm = OpenAI()
# response = llm.chat.completions.create(
# model="gpt-4o-mini",
# messages=[
# {"role": "system", "content": "Answer using only the provided context."},
# {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
# ],
# )
Keep the retrieved context short and the instruction strict. "Answer using only the provided context" is what stops the model from inventing facts that aren't in your documents.
When to use Chroma vs alternatives
| Tool | Pick it when | Look elsewhere when |
|---|---|---|
| Chroma | you want local, Python-native, running in minutes, small to medium corpora | you need horizontal scale or very high query throughput |
| pgvector | your data already lives in Postgres and you don't want another system to run | you need vector features Postgres doesn't ship, like some distance metrics or hybrid scoring |
| Qdrant | production service with rich filtering and high QPS, Rust server, SDKs in many languages | you want zero infrastructure and a single process |
| FAISS | raw nearest-neighbor speed inside your own process, and you handle persistence yourself | you want a database with CRUD, filters, and durability built in |
| Milvus | very large scale, distributed deployments | you don't need distributed search. Milvus is heavy to operate |
Honest tradeoff: Chroma is the easiest way to get semantic search working, and it stays comfortable well past "prototype". The moment your requirements become "several services share one index" or "thousands of queries per second", the migration is mostly mechanical, because the query API looks similar across all of these tools.
Security note: CVE-2026-45829
This deserves its own section. In May 2026 a critical vulnerability was published in ChromaDB: CVE-2026-45829 (GitHub advisory GHSA-f4j7-r4q5-qw2c), CVSS 9.3. It's a pre-authentication code injection affecting versions 1.0.0 through 1.5.9 in server mode. An unauthenticated attacker can run arbitrary code on the server by sending a malicious model repository with trust_remote_code enabled to the collections endpoint. At the time of writing there is no patched release.
What this means in practice: the in-memory and persistent clients from this article don't listen on a network port, so they're not exposed. But chroma run and the Docker image are servers. If you deploy one anywhere reachable, treat it as unauthenticated: put it behind an authenticated reverse proxy, don't bind it to a public interface, pin the version, and watch the advisory for a fix. This is also a good reason to embed locally, like the default function does, instead of letting the server fetch model repositories.
Gotchas
- Full-text search is case-sensitive. Normalize text at ingest if your users type lowercase.
- Distances are Euclidean and there's no similarity threshold, so
query()always returnsn_resultsitems, 10 by default. - The default model downloads ~80 MB on first run. Offline environments need it cached beforehand.
- Chroma sends anonymized telemetry. In 0.6.x a posthog mismatch makes those events fail and prints harmless "Failed to send telemetry event" warnings to stderr, so nothing leaves your machine in that configuration.
- Since 0.6.0,
list_collections()returns names only. Useclient.get_collection(name)to work with one. - Metadata values must be str, int, float, or bool. Lists and nested dicts raise errors.
Next steps
- Wire the LLM call from section 7 into a full RAG loop, and return the document ids alongside the answer so the model can cite its sources.
- Combine keyword and vector search:
$containshits plus vector hits, merged with reciprocal rank fusion, usually beats either alone. - Re-rank the top 20 with a cross-encoder before the LLM sees anything. It's the slowest step but it improves answer quality noticeably.
- When you outgrow Chroma, the pgvector and Qdrant moves are mostly mechanical. This blog already has a pgvector RAG walkthrough to start from.