One API for All Your LLMs: LiteLLM Proxy as Your AI Gateway

Your app talks to three providers. Each has its own SDK, its own auth, its own error handling. Switching from OpenAI to Anthropic means rewriting the client layer, and nobody on the team has a clear answer to the simplest question a manager will ask: how much are we actually spending on this? A proxy sitting in front of every provider fixes that. You get one endpoint, one format, and one place to track cost. LiteLLM is the self-hosted gateway most teams reach for, and it is open source.

This guide takes you from a bare install to a running gateway with multiple providers, virtual keys, budgets, spend tracking, and load balancing. Every command below matches the current official docs (September 2026).

Prerequisites

• Python 3.10+. LiteLLM 1.84.0 and newer require it.
• uv installed, or pip. uv tool install provisions a compatible Python for you automatically.
• Docker, only if you do the budget and spend step (for the local PostgreSQL).
• Optional keys: an OpenAI API key, an Anthropic API key, and a running Ollama server.

What you are building

Requests go to one endpoint (http://0.0.0.0:4000). The proxy decides which real provider to call, applies rate limits and budgets, records the cost, and returns an OpenAI-compatible response. Your application only ever knows the OpenAI format.

Step 1: Install

The supported install is:

uv tool install 'litellm[proxy]'

Prefer uv here. A plain pip install 'litellm[proxy]' on an interpreter older than 3.10 silently resolves to the last release that still allowed it, version 1.83.9. If something looks pinned, check python --version and reinstall with uv.

Step 2: One model, first run

With a single provider you do not even need a config file. Set the key in the environment and run:

export OPENAI_API_KEY=sk-...
litellm --model gpt-4o

The proxy starts on http://0.0.0.0:4000. In a second terminal, verify:

litellm --test

This makes a real openai.chat.completions request through the proxy. The test command requires the OpenAI Python package v1.0.0+.

Step 3: Route multiple providers with config.yaml

This is where the gateway earns its keep. Create config.yaml:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: llama3.1
    litellm_params:
      model: ollama_chat/llama3.1
      api_base: http://localhost:11434

litellm_settings:
  drop_params: true

general_settings:
  master_key: sk-your-admin-key

Rules to keep straight:

• model_name is the name clients send. litellm_params.model is the real provider string.
• The openai/ and anthropic/ prefixes tell LiteLLM which provider SDK to use. Ollama has its own ollama_chat/ prefix, which hits the /api/chat endpoint and generally gives better responses than the raw ollama/ variant.
• os.environ/OPENAI_API_KEY reads the key from an environment variable instead of hard-coding it in the file.
• master_key sets the admin key for the proxy and must start with sk-.

Start it:

litellm --config config.yaml

You should see LiteLLM: Proxy initialized with Config, Set models: in the logs. If not, run with --detailed_debug.

Step 4: Call it

From the OpenAI SDK. There is no provider-switching logic left in the client, which is the whole point:

from openai import OpenAI

client = OpenAI(api_key="anything", base_url="http://localhost:4000")

resp = client.chat.completions.create(
    model="claude-opus",          # any model_name from config.yaml
    messages=[{"role": "user", "content": "Summarize this in three sentences."}],
)
print(resp.choices[0].message.content)

The api_key is not checked unless you set master_key. Point any OpenAI-compatible client at http://localhost:4000 and you get the same chat, completions, and embeddings endpoints you already know.

Step 5: Virtual keys and spend tracking

Keys, budgets, and cost need a database. LiteLLM uses PostgreSQL. The simplest local option is a container:

docker run --name litellm-db   -e POSTGRES_USER=user -e POSTGRES_PASSWORD=pass -e POSTGRES_DB=litellm   -p 5432:5432 -d postgres:16

Register it in the config under general_settings::

general_settings:
  master_key: sk-your-admin-key
  database_url: "postgresql://user:pass@localhost:5432/litellm"

Restart the proxy. Now real virtual keys can be issued, and only the master key can mint them:

curl 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer sk-your-admin-key' \
  -H 'Content-Type: application/json' \
  -d '{"models": ["claude-opus", "gpt-4o"], "duration": "30d"}'

Give the returned sk-... key to a teammate. Their app calls the proxy with it and the proxy records every request. Check the spend on a key any time:

curl 'http://localhost:4000/key/info?key=<user-key>' \
  -H 'Authorization: Bearer sk-your-admin-key'

The response includes a spend field in USD. Spend is also tracked per user and per team when you mint keys under /user/new and /team/new. Those rows have a max_budget you can set so a runaway loop gets cut off instead of silently charging your card.

Step 6: Load balancing and rate limits

To spread traffic across several deployments of the same model, give the same model_name to two entries. The proxy load balances between them, and rpm caps each deployment:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 100
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o-backup
      api_key: os.environ/OPENAI_API_KEY
      rpm: 50

The official docs report the proxy handling 1.5k+ requests per second in their own load tests. That is a big-GPU-server number, not something you will hit on one laptop. For most teams the value is the routing, budgeting, and shared keys, not peak throughput.

When to use LiteLLM vs alternatives

• Provider SDKs directly. The right call for a single-provider prototype, but you lose the ability to switch providers later without touching code.
• OpenRouter. A hosted router with free virtual keys, but your data and keys live on their infrastructure and cost analytics are tied to their model catalog.
• Cloudflare AI Gateway. A good fit if you already run on Cloudflare, but it is closed and built around the CF platform.
• Kong AI Gateway or Portkey. Full enterprise API management. Overkill until you need SSO, org-wide policies, and custom plugins.
• LiteLLM. Self-hosted, open source, works with local models and private data, and keeps cost tracking in a database you own. The trade for self-hosting is that you run and watch it yourself.

Next steps

• Add embedding models to the list and use /embeddings for a RAG pipeline.
• Set up a model alias so a key can upgrade requests from gpt-4o to claude-opus without changing client code.
• Wire success_callback to Langfuse or Slack for logging and alerts.
• Read the FAQ before exposing the proxy to the internet. The master key is what keeps people off your providers.

References

• LiteLLM GitHub repository
• LiteLLM Proxy quick start
• LiteLLM config.yaml reference
• LiteLLM virtual keys and spend tracking