You start with one provider key in .env. Six months later there are four services, three providers, and a Slack thread asking why the Anthropic bill tripled. Every service holds its own copy of the key, runs its own retry loop, and has its own idea of which model is the cheap one.
A shared client library looks like the fix until the retry logic drifts between repos and the version pinning starts a small war. A gateway solves it at the network layer instead: every service calls one OpenAI-compatible endpoint, and the gateway decides which provider handles the request, what happens when that provider starts returning 500s, and how much each key is allowed to spend.
LiteLLM Proxy is that gateway, and it fits on the same VM as your app. By the end of this walkthrough you have two model aliases, a fallback path that survives a dead provider, and a virtual key capped at $5 per 30 days. Everything runs locally. For production you change the image tag, the bind address, and the Postgres password.
What you need
- Docker Engine 24+ with Compose v2 (
docker compose version) - API keys for two providers, here OpenAI and Anthropic
curlandjq- Python 3.10+ only for the SDK test near the end
LiteLLM 1.108.3 (the release this was tested against) supports Python 3.10 to 3.14. The container ships its own runtime, so that range only matters if you install the SDK directly.
Step 1: bring up the gateway and Postgres
Budgets and spend tracking live in the database. Without Postgres the proxy still serves requests, but every budget check fails open. That is worse than having no budget, because you think you are covered.
Create the env file first. LITELLM_SALT_KEY encrypts provider keys stored in the database and cannot be rotated in place, so keep this file somewhere safe.
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\nPOSTGRES_PASSWORD=%s\n' \
"$(openssl rand -hex 32)" "$(openssl rand -hex 32)" "$(openssl rand -hex 16)" > .env
Add your provider keys to the same file:
printf 'OPENAI_API_KEY=%s\nANTHROPIC_API_KEY=%s\n' \
"sk-your-openai-key" "sk-ant-your-anthropic-key" >> .env
Now docker-compose.yml:
services:
litellm:
image: docker.litellm.ai/berriai/litellm:latest
ports:
- "127.0.0.1:4000:4000"
environment:
LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
LITELLM_SALT_KEY: ${LITELLM_SALT_KEY}
DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@db:5432/litellm
OPENAI_API_KEY: ${OPENAI_API_KEY}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
volumes:
- ./config.yaml:/app/config.yaml:ro
command: ["--config", "/app/config.yaml", "--port", "4000"]
depends_on:
db:
condition: service_healthy
db:
image: postgres:16
environment:
POSTGRES_USER: litellm
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: litellm
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 5s
timeout: 5s
retries: 10
volumes:
- postgres_data:/var/lib/postgresql/data
volumes:
postgres_data:
Two details matter here. The port binds to 127.0.0.1 so the gateway is not reachable from the internet before you put TLS in front of it. And the image tag is latest only because that is what the official quickstart uses; check the release notes and pin a version before this touches real traffic.
Step 2: define model aliases
config.yaml is where the routing lives. Your apps will only ever send fast or smart.
model_list:
- model_name: fast
litellm_params:
model: openai/gpt-6-luna
api_key: os.environ/OPENAI_API_KEY
- model_name: smart
litellm_params:
model: anthropic/claude-sonnet-5
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
routing_strategy: latency-based-routing
num_retries: 2
timeout: 30
allowed_fails: 3
cooldown_time: 30
fallbacks: [{"fast": ["smart"]}]
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
model_name is the alias your application sends. litellm_params.model is the real provider string, and the prefix tells LiteLLM which SDK and auth style to use. Keeping the two separate is what makes a provider swap a config change instead of a code change.
The router settings are worth reading once. num_retries: 2 retries within the same deployment. allowed_fails: 3 puts a deployment in cooldown for cooldown_time seconds after three failures, so one broken key does not eat every request for the next minute. fallbacks sends traffic for fast to smart once retries are exhausted.
Start it:
docker compose up -d
curl -s http://localhost:4000/health/readiness | jq
Expected output:
{"status": "healthy", "db": "connected"}
That endpoint checks the process and the database. To check whether the providers actually answer, call the authenticated health route. Be aware it sends a real request to every deployment, so it costs a few tokens each time:
curl -s http://localhost:4000/health \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.healthy_count, .unhealthy_count'
Step 3: issue a key with a budget
Your services should never see the master key. That key can create and delete other keys, read every spend log, and change routing. Generate a virtual key per service instead.
curl -s http://localhost:4000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"key_alias": "checkout-service",
"models": ["fast", "smart"],
"max_budget": 5,
"budget_duration": "30d",
"rpm_limit": 60
}' | jq -r .key
Save the printed value as LITELLM_KEY. The response is the only time the full key is shown.
The models list is an allowlist, not a suggestion. Both aliases are listed because a fallback can land on either one. max_budget is in USD and budget_duration resets the counter every 30 days, so this key gets $5 per month, not $5 total. rpm_limit caps it at 60 requests per minute, which is enough to stop a runaway loop from burning the budget in one minute.
Step 4: call it from your app
The gateway speaks the OpenAI chat completions format, so existing SDK code works with a base URL change.
curl -i http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "fast", "messages": [{"role": "user", "content": "Say hello in five words."}]}'
Check the response headers. x-litellm-model-api-base tells you which upstream actually served the request, which is the fastest way to confirm that a fallback fired.
import os
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000/v1",
api_key=os.environ["LITELLM_KEY"],
)
resp = client.chat.completions.create(
model="fast",
messages=[{"role": "user", "content": "Classify this ticket: my card was charged twice."}],
)
print(resp.choices[0].message.content)
Nothing in that file knows about Anthropic. If you decide tomorrow that fast should be a Gemini model, you edit config.yaml and restart the container. The app keeps sending the same alias.
Step 5: prove the fallback works
A fallback you have never triggered is a guess. Break the primary key on purpose and watch the header change.
OPENAI_API_KEY=sk-invalid docker compose up -d --force-recreate litellm
Shell variables take precedence over .env during interpolation, so the container restarts with a key that will fail auth. Send the same request and inspect the header:
curl -s -D - -o /dev/null http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "fast", "messages": [{"role": "user", "content": "ping"}]}' \
| grep -i x-litellm-model-api-base
You should see the Anthropic API base instead of the OpenAI one. Restore the real key when you are done:
docker compose up -d --force-recreate litellm
This test also tells you something about latency. One failed auth attempt plus retries adds a delay to that request, so keep num_retries low and the timeout tight. A fallback that takes 40 seconds is not a fallback for interactive traffic.
Step 6: read the spend
curl -s "http://localhost:4000/key/info?key=$LITELLM_KEY" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.info.spend'
Cost comes from LiteLLM's model price map, so it is an estimate based on published per-token prices, not your negotiated contract. For a team dashboard that is usually close enough. For billing a customer, reconcile against the provider invoice once a month.
When the key crosses max_budget, requests start failing with a budget error instead of silently continuing. Wire that error into your app logs, otherwise the first person to notice will be a user.
Where this breaks in production
- One replica only. With two proxy containers, cooldowns and rate limits live in separate memory. Add Redis and point
router_settings.redis_hostat it before you scale out. - Master key shipped to an app. Treat it as root. Rotate it if it lands in a frontend bundle or a CI log.
- Rotating
LITELLM_SALT_KEY. There is no in-place rotation. Changing it makes every stored provider key unreadable until you re-enter them. - Budgets with no database. The check is skipped, and you only get one warning line at startup.
When to use this vs alternatives
Direct provider SDKs in each service work fine when you have one provider and one team. The moment a second service needs a fallback, you are copying routing code into a second repo.
OpenRouter is faster to start if you want hosted routing and do not want to run Postgres. The tradeoff is that your traffic and spend data sit behind someone else's gateway, and per-key budgets follow their model.
A custom FastAPI wrapper makes sense only when routing depends on business logic, like sending enterprise tenants to a specific deployment. Even then you will rebuild retries, cooldowns, and a price table, which is most of what this proxy already does.
Next steps
Add a second deployment under the same model_name to load balance across two API keys from one provider. Move from per-service keys to teams when you want a shared budget with per-member limits. And before the second replica, add Redis so the router state is shared.
The whole setup is about 30 lines of YAML. That is the point: routing, failover, and spend control stop being application code and become infrastructure you can read in one screen.