Three customers open your support widget in the same hour. One asks "how do I reset my password", another types "i forgot my password", the third sends "password reset link not working". Same intent, same prompt, same correct answer. Without a cache you pay for three full generations and wait three times.
An exact-match cache stores the first string and misses the other two, because string equality is the wrong comparison for language. A semantic cache embeds the question, looks for a near neighbour in a vector index, and returns the stored answer when that neighbour is close enough. The search itself is the easy part. The parts that decide whether this helps or hurts are the threshold and the list of requests you refuse to cache.
What you need
- Docker, for the Redis image that ships the search module
- Python 3.11 or newer, with
pip install redis openai numpy - An OpenAI API key, or a local embedding server like Ollama (the index dimension has to match your model)
- One LLM call that already works
The commands below were run against redis/redis-stack-server with Redis 7.4 and search module 2.10.
Step 1: run Redis with the query engine
Plain redis:7 has no FT.CREATE. Point this code at a vanilla Redis and every call fails with unknown command 'FT.CREATE'. Use the Stack image:
docker run -d --name semcache -p 6379:6379 redis/redis-stack-server:latest
docker exec semcache redis-cli MODULE LIST
The output should include a module named search.
Step 2: decide what a cache key means
Two tiers, because they cost different things:
- Exact tier:
sha256(namespace + normalized question)in a plain string key. No embedding call, no false positives, and it absorbs the traffic that arrives word for word. - Semantic tier: one hash per answer, holding the question, its embedding, the response, the namespace, and a timestamp.
Most of the design lives in the namespace. Put the model, the prompt version, and the tenant in it, for example v3_gpt55_tenant42. Bump the prompt version and stale answers stop being served without flushing anything. Keep it to letters, digits, and underscores: Redis TAG queries treat punctuation as syntax, so a colon or a hyphen inside the tag needs escaping that is easy to forget under deadline.
Step 3: create the index
import numpy as np
from redis import Redis
from redis.commands.search.field import NumericField, TagField, TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType
from redis.exceptions import ResponseError
INDEX = "semcache"
PREFIX = "semcache:vec:"
DIM = 1536
r = Redis(host="localhost", port=6379, decode_responses=False)
def ensure_index() -> None:
schema = (
TextField("prompt"),
TagField("ns"),
NumericField("created_at"),
VectorField(
"embedding",
"HNSW",
{
"TYPE": "FLOAT32",
"DIM": DIM,
"DISTANCE_METRIC": "COSINE",
"M": 16,
"EF_CONSTRUCTION": 200,
},
),
)
try:
r.ft(INDEX).info()
except ResponseError:
r.ft(INDEX).create_index(
fields=schema,
definition=IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH),
)
ensure_index()
VECTOR HNSW 6 means six attributes follow the field name. M and EF_CONSTRUCTION shape the graph at build time: higher values mean better recall and slower inserts. If you leave them out, Redis uses 16 and 200, which is a reasonable starting point for a cache in the low hundreds of thousands of entries.
The index lives on the server, not in your process. Create it once; ensure_index is written so it is safe to call on every start.
Step 4: look up, then store
import hashlib
import time
import uuid
from openai import OpenAI
from redis.commands.search.query import Query
openai_client = OpenAI()
THRESHOLD = 0.92
def embed(text: str) -> np.ndarray:
resp = openai_client.embeddings.create(
model="text-embedding-3-small",
input=text[:8000],
)
return np.asarray(resp.data[0].embedding, dtype=np.float32)
def normalize(text: str) -> str:
return " ".join(text.lower().split()).strip(" ?!.")
def exact_key(ns: str, question: str) -> str:
digest = hashlib.sha256(f"{ns}:{normalize(question)}".encode()).hexdigest()
return f"semcache:exact:{ns}:{digest}"
def semantic_lookup(ns: str, vector: np.ndarray) -> tuple[str, float] | None:
query = (
Query(f"(@ns:{{{ns}}})=>[KNN 1 @embedding $vec AS dist]")
.return_field("dist")
.return_field("response")
.dialect(2)
)
result = r.ft(INDEX).search(query, query_params={"vec": vector.tobytes()})
if not result.docs:
return None
doc = result.docs[0]
similarity = min(1.0, 1.0 - float(doc.dist))
if similarity < THRESHOLD:
return None
return doc.response.decode(), similarity
def ask_model(question: str) -> str:
resp = openai_client.responses.create(
model="gpt-5-mini", # swap for the model you already run
input=question,
)
return resp.output_text
def answer(ns: str, question: str, ttl: int = 86_400) -> tuple[str, str]:
cached = r.get(exact_key(ns, question))
if cached is not None:
return cached.decode(), "exact"
vector = embed(question)
hit = semantic_lookup(ns, vector)
if hit is not None:
return hit[0], "semantic"
reply = ask_model(question)
doc_key = f"{PREFIX}{ns}:{uuid.uuid4().hex}"
pipe = r.pipeline()
pipe.set(exact_key(ns, question), reply, ex=ttl)
pipe.hset(
doc_key,
mapping={
"prompt": question,
"ns": ns,
"created_at": time.time(),
"response": reply,
"embedding": vector.tobytes(),
},
)
pipe.expire(doc_key, ttl)
pipe.execute()
return reply, "miss"
Four details in that code are load-bearing.
decode_responses=False is not optional. With decoding on, redis-py hands you str for every value, and a float32 vector does not survive the round trip. Your stored embedding comes back mangled and every similarity score turns to noise.
Redis reports cosine distance, not similarity. similarity = 1 - dist for the COSINE metric. Float error can push an identical vector to 1.0000001, so clamp at 1.0 before you compare or log.
The hash and the string key both need a TTL. Setting one and forgetting the other is how a cache quietly becomes a database.
The embedding is computed once per request and reused for both the lookup and the write. Calling embed twice doubles your embedding bill for every miss, which is the traffic you were trying to reduce.
Step 5: pick the threshold with numbers, not vibes
At 0.99 you get almost no hits. At 0.80 you start answering "how do I reset my password" with the instructions for changing an email address, delivered with full confidence. Neither failure is visible in a dashboard unless you look for it.
Build a small labelled set from real questions, then sweep:
import numpy as np
from cache import embed
def cosine(a, b) -> float:
a, b = np.asarray(a), np.asarray(b)
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
# same intent, different words: these should hit
same = [
("how do i reset my password", "i forgot my password"),
("how do i reset my password", "password reset link not working"),
("can i export my data as csv", "download all my data to excel"),
("what is the refund window", "how many days do i have to get a refund"),
]
# adjacent topic, different answer: these must not hit
different = [
("how do i reset my password", "how do i change my email address"),
("how do i export my data as csv", "how do i delete my account"),
("what is the refund window", "do you ship internationally"),
("how do i reset my password", "the login page is slow"),
]
same_sims = [cosine(embed(a), embed(b)) for a, b in same]
diff_sims = [cosine(embed(a), embed(b)) for a, b in different]
for t in np.arange(0.80, 0.99, 0.01):
recall = sum(s >= t for s in same_sims) / len(same_sims)
false_accept = sum(s >= t for s in diff_sims) / len(diff_sims)
print(f"{t:.2f} recall {recall:.2f} false accept {false_accept:.2f}")
Read the output as a trade curve, not a score. Pick the highest threshold that still catches the paraphrases you care about, then spot check 20 random hits by hand. A threshold that looks fine on eight pairs can be wrong on your actual traffic, and the only way to know is to read what the cache returned.
If no threshold gives you both acceptable recall and zero false accepts, the embedding model is too weak for the domain. Try a larger model before you lower the bar.
What must never go into the cache
- Requests that carry conversation history. The same question in a different thread has a different answer.
- Anything with tool calls, writes, refunds, or emails. A cache hit skips the side effect, and a retry that silently does nothing is worse than a slow request.
- State that changes by the minute: order status, balances, inventory.
- Answers filtered by permissions. If the cached answer came from an admin session, a customer can read it. Tenant in the namespace is the minimum, not the whole control.
- Personalised output of any kind. Cache the retrieval, not the reply.
Streaming is fine. Store the finished text and replay it as one chunk, or re-stream from the cached string. The user sees a faster answer either way.
Operating it
Count three outcomes, not one: exact, semantic, miss. A rising miss rate with flat traffic means the threshold is too high or the prompt version changed. A rising exact rate usually means clients are retrying.
Log the similarity of semantic hits. If the median hit sits within 0.02 of your threshold, the cache is working at the edge of its accuracy and one bad deploy will push it over.
Memory is predictable. A 1536-dimension float32 vector is 6,144 bytes, so 100,000 entries take about 614MB before index overhead. text-embedding-3-small accepts a dimensions argument if you need to cut that, at some cost in accuracy.
When you change embedding models, create a new index name and drop the old one with FT.DROPINDEX semcache DD. Dimensions cannot be altered in place, and mismatched vectors fail at query time, not at write time.
When to use this, and when not to
Exact cache only: fixed wording, FAQ pages, templated responses. Zero false positives and no embedding cost. Often enough on its own.
Provider prompt caching: the same long system prompt or document, different questions. It cuts input cost on the prefix and does nothing for two users asking the same thing in different words. The two techniques stack.
A library such as GPTCache or RedisVL's semantic cache: more vectorizers, similarity evaluators, and eviction policies out of the box. Reach for one when you do not want to own this code, or when you need adapters for several vector stores.
Skip the semantic tier entirely if your volume is low, if every answer is user specific, or if the agent runs tools on every turn. The cache becomes another stateful system to debug and the hit rate never pays for it.
Next steps
Three upgrades worth the time, in order: add a per-tenant TAG filter so one workspace cannot read another's cached answers, put the eight pairs above into CI as a regression test that fails when a previously safe pair starts matching, and add a negative cache for questions the model refuses, so abuse traffic does not get a free generation every time.