Someone opens your chat feature, hits send, and gets a 500. Nothing in your repo changed. One provider returned 429 for forty seconds, and your app had exactly one place to buy tokens from.
The usual patch is a try/except in every service that calls the API. Then the retry logic drifts. One service retries three times, another retries never, a third retries on a 400 and makes the outage longer. Billing is a separate mess: eight services, eight keys, and nobody can answer which feature burned four million tokens yesterday.
A gateway fixes those three things in one deploy. A second provider to fall back to, one key to rotate, and one log line per request with model, status, latency, and token counts. It does not need a framework. Around 150 lines of FastAPI and httpx, and any OpenAI SDK client points at it by changing a base URL.
Prerequisites
- Python 3.11+
pip install fastapi uvicorn httpx- Keys for two OpenAI-compatible providers (OpenAI, Groq, Together, Fireworks, or your own vLLM)
- curl
httpx instead of requests because the gateway is async. A blocking client stalls every other request while one generation is streaming.
Step 1: a provider that fails on demand
You cannot ask a real provider to return 503 at a convenient time. So the fallback path gets tested against a fake upstream first: two instances of the same 30-line app, one answering 503, one answering normally.
# mock_upstream.py
import json
import os
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, StreamingResponse
mode = os.environ.get("MOCK_MODE", "ok")
app = FastAPI()
@app.post("/chat/completions")
async def chat(request: Request):
if mode == "down":
return JSONResponse(
{"error": {"message": "upstream overloaded", "type": "rate_limit"}},
status_code=503,
headers={"retry-after": "0"},
)
body = await request.json()
if body.get("stream"):
async def gen():
for token in ["Hello", " from", " the", " fallback", " provider"]:
frame = {"choices": [{"index": 0, "delta": {"content": token}}]}
yield f"data: {json.dumps(frame)}\n\n".encode()
yield b"data: [DONE]\n\n"
return StreamingResponse(gen(), media_type="text/event-stream")
return {
"id": "chatcmpl-mock",
"object": "chat.completion",
"model": body.get("model", "mock-1"),
"choices": [{
"index": 0,
"message": {"role": "assistant", "content": "Hello from the fallback provider"},
"finish_reason": "stop",
}],
"usage": {"prompt_tokens": 11, "completion_tokens": 7, "total_tokens": 18},
}
Two instances:
MOCK_MODE=down python3 -m uvicorn mock_upstream:app --port 9101 &
MOCK_MODE=ok python3 -m uvicorn mock_upstream:app --port 9102 &
Step 2: the gateway
# gateway.py
import asyncio
import json
import logging
import os
import time
from contextlib import asynccontextmanager
import httpx
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, Response, StreamingResponse
logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("gateway")
# Order matters: the first entry is primary, the rest are fallbacks.
PROVIDERS = json.loads(
os.environ.get(
"GATEWAY_PROVIDERS",
json.dumps([
{
"name": "openai",
"base_url": "https://api.openai.com/v1",
"key_env": "OPENAI_API_KEY",
},
{
"name": "groq",
"base_url": "https://api.groq.com/openai/v1",
"key_env": "GROQ_API_KEY",
},
]),
)
)
RETRY_STATUS = {408, 409, 425, 429, 500, 502, 503, 504}
ATTEMPTS_PER_PROVIDER = 2
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.client = httpx.AsyncClient(
timeout=httpx.Timeout(connect=5.0, read=120.0, write=10.0, pool=10.0),
limits=httpx.Limits(max_connections=200, max_keepalive_connections=50),
)
yield
await app.state.client.aclose()
app = FastAPI(lifespan=lifespan)
def headers_for(provider: dict) -> dict:
headers = {"Content-Type": "application/json"}
key = os.environ.get(provider["key_env"], "")
if key:
headers["Authorization"] = f"Bearer {key}"
return headers
def backoff_seconds(resp: httpx.Response, attempt: int) -> float:
raw = resp.headers.get("retry-after")
if raw:
try:
return min(float(raw), 5.0)
except ValueError:
pass
return 0.25 * (2 ** attempt)
def log_call(provider: str, status: int, started: float, usage: dict | None) -> None:
log.info(json.dumps({
"provider": provider,
"status": status,
"latency_ms": round((time.perf_counter() - started) * 1000),
"prompt_tokens": (usage or {}).get("prompt_tokens"),
"completion_tokens": (usage or {}).get("completion_tokens"),
}))
async def call_with_fallback(client: httpx.AsyncClient, body: bytes):
"""Return (provider_name, response) from the first provider that answers."""
last = None
for provider in PROVIDERS:
for attempt in range(ATTEMPTS_PER_PROVIDER):
started = time.perf_counter()
try:
resp = await client.post(
f"{provider['base_url']}/chat/completions",
content=body,
headers=headers_for(provider),
)
except httpx.HTTPError as exc:
log_call(provider["name"], 0, started, None)
log.info(json.dumps({"error": f"{type(exc).__name__}: {exc}"}))
continue
usage = None
try:
usage = resp.json().get("usage")
except ValueError:
pass
log_call(provider["name"], resp.status_code, started, usage)
if resp.status_code not in RETRY_STATUS:
return provider["name"], resp
last = (provider["name"], resp)
if attempt + 1 < ATTEMPTS_PER_PROVIDER:
await asyncio.sleep(backoff_seconds(resp, attempt))
return last
async def open_stream_with_fallback(client: httpx.AsyncClient, body: bytes):
"""Open a stream, but only fall back while no bytes have been sent yet."""
last = None
for provider in PROVIDERS:
started = time.perf_counter()
request = client.build_request(
"POST",
f"{provider['base_url']}/chat/completions",
content=body,
headers=headers_for(provider),
)
resp = await client.send(request, stream=True)
log_call(provider["name"], resp.status_code, started, None)
if resp.status_code not in RETRY_STATUS:
return provider["name"], resp
last = resp
await resp.aclose()
raise httpx.HTTPStatusError(
"all providers failed",
request=request,
response=last or httpx.Response(502),
)
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
raw = await request.body()
try:
payload = json.loads(raw)
except ValueError:
return JSONResponse({"error": {"message": "invalid JSON body"}}, status_code=400)
client: httpx.AsyncClient = request.app.state.client
if not payload.get("stream"):
result = await call_with_fallback(client, raw)
if result is None:
return JSONResponse({"error": {"message": "all providers failed"}}, status_code=502)
name, resp = result
return Response(
content=resp.content,
status_code=resp.status_code,
media_type="application/json",
headers={"x-gateway-provider": name},
)
try:
name, upstream = await open_stream_with_fallback(client, raw)
except httpx.HTTPStatusError:
return JSONResponse({"error": {"message": "all providers failed"}}, status_code=502)
async def relay():
try:
async for chunk in upstream.aiter_raw():
if await request.is_disconnected():
break
yield chunk
finally:
await upstream.aclose()
return StreamingResponse(
relay(),
media_type="text/event-stream",
headers={
"x-gateway-provider": name,
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
@app.get("/healthz")
async def healthz():
return {
"providers": [
{"name": p["name"], "configured": bool(os.environ.get(p["key_env"]))}
for p in PROVIDERS
]
}
Run it with the two fake upstreams:
export GATEWAY_PROVIDERS='[{"name":"broken","base_url":"http://127.0.0.1:9101","key_env":"BROKEN_KEY"},{"name":"good","base_url":"http://127.0.0.1:9102","key_env":"GOOD_KEY"}]'
python3 -m uvicorn gateway:app --port 9200
Four decisions in that file carry most of the weight.
Timeouts are split on purpose. httpx treats a timeout as five seconds of inactivity by default. Five seconds of read timeout cuts a long generation off mid-sentence, so connect stays at 5 and read goes to 120. Raising one global timeout number instead would also hide dead sockets and slow DNS behind 120 seconds of waiting.
Only retry what is worth retrying. A 429 or a 502 means "try again later". A 400 means your payload is wrong, and retrying it twice adds latency to a bug. So the retry set is 408, 409, 425, 429 and the 5xx family, two attempts per provider, and Retry-After honored up to five seconds because a provider asking for a 30 second pause is a reason to move to the next provider, not to sleep.
Streaming falls back only before the first byte. Once the gateway has relayed a chunk, the client already holds half an answer and no failover can save it. So the stream path checks the status code before relaying anything, and only then starts copying bytes. After that, an upstream failure ends the stream.
The body is forwarded byte for byte. Nothing parses messages, so tools, response_format, temperature, and vendor-specific extras reach the provider untouched. The gateway adds auth and the x-gateway-provider response header, and otherwise stays out of the way.
Step 3: verify it
Health, with no keys set in this sandbox (both report configured: false):
$ curl -s http://127.0.0.1:9200/healthz
{"providers":[{"name":"broken","configured":false},{"name":"good","configured":false}]}
Non-streaming request while the primary is down:
$ curl -s -D- -o /dev/null -X POST http://127.0.0.1:9200/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"mock-1","messages":[{"role":"user","content":"hi"}]}'
HTTP/1.1 200 OK
x-gateway-provider: good
content-type: application/json
The same request with "stream": true:
HTTP/1.1 200 OK
x-gateway-provider: good
data: {"choices": [{"index": 0, "delta": {"content": "Hello"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " from"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " the"}}]}
data: {"choices": [{"index": 0, "delta": {"content": " fallback"}}]}
The gateway log for that request, one JSON line per attempt:
{"provider": "broken", "status": 503, "latency_ms": 3, "prompt_tokens": null, "completion_tokens": null}
{"provider": "broken", "status": 503, "latency_ms": 2, "prompt_tokens": null, "completion_tokens": null}
{"provider": "good", "status": 200, "latency_ms": 2, "prompt_tokens": null, "completion_tokens": null}
Two entries for broken (the retry), then the fallback. Non-streaming calls log prompt_tokens: 11 and completion_tokens: 7 from the mock's usage block, which is the number you would multiply by your price table.
The part that matters: the real OpenAI SDK works unchanged.
from openai import OpenAI
client = OpenAI(api_key="anything", base_url="http://127.0.0.1:9200/v1")
resp = client.chat.completions.create(
model="mock-1",
messages=[{"role": "user", "content": "say hi"}],
)
print(resp.choices[0].message.content)
# Hello from the fallback provider
stream = client.chat.completions.create(
model="mock-1",
messages=[{"role": "user", "content": "say hi"}],
stream=True,
)
print("".join(c.choices[0].delta.content or "" for c in stream if c.choices))
# Hello from the fallback provider
When every provider is down, non-streaming requests pass the last upstream error through unchanged (a 503 with the provider's own body, which is what you want in the logs), and streaming requests get a clean 502.
The bug you will hit
The first version of this gateway set the header unconditionally:
headers = {"Authorization": f"Bearer {os.environ.get(key_env, '')}"}
httpx refuses to send a header whose value is just Bearer and raises:
LocalProtocolError: Illegal header value b'Bearer '
Because the handler catches httpx.HTTPError broadly, the failure looked like a provider outage: status 599 in the log, an empty body for the client, and both providers "down" while both were healthy. Building the header only when a key exists fixed it. A missing key now gets skipped instead of poisoning the request.
Step 4: keep streaming alive behind nginx
nginx buffers upstream responses by default. That turns a token stream into one large delivery when the generation finishes, and your time to first token becomes the full generation time. X-Accel-Buffering: no is already set on the response, and the location block needs the rest:
location /v1/ {
proxy_pass http://127.0.0.1:9200;
proxy_buffering off;
proxy_read_timeout 300s;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
Step 5: count tokens for streaming calls
Non-streaming responses carry usage, so the log line is free. Streaming responses do not, unless the request asks for it. OpenAI-compatible endpoints accept stream_options:
{
"model": "gpt-4.1-mini",
"messages": [{"role": "user", "content": "hi"}],
"stream": true,
"stream_options": {"include_usage": true}
}
With that flag the provider sends one extra chunk just before data: [DONE], with choices: [] and the totals in usage. Every earlier chunk carries usage: null. If you parse streamed responses and assume choices[0] always exists, that extra chunk is a crash waiting for the first production request. The gateway relays it like any other frame, so your cost accounting reads it on the client side or in a small parser added to the relay loop.
That also explains the null token counts in the streaming log lines above: the mock did not send a usage chunk, so there was nothing to log.
When to use this vs the alternatives
LiteLLM proxy if you need translation between provider APIs, virtual keys per team, budgets, or routing strategies like latency-based routing. It covers more providers than you will ever wire by hand, and it is the right call when your gateway has to be a product rather than 150 lines in your repo.
A hosted gateway (Cloudflare AI Gateway, Portkey) if you do not want to run anything. You trade control over the routing logic for zero maintenance.
This code if all your providers already speak /v1/chat/completions and you want the failover rules in your own repository, reviewable in a pull request, deployable next to your app.
One warning about mixing providers. Anthropic publishes an OpenAI SDK compatibility layer, and their docs describe it as intended for testing and comparison, not as a long-term production solution. If Claude is one of your backends and you care about the full feature set (citations, PDF input, prompt caching), write a native adapter for /v1/messages or let LiteLLM do the translation. Do not assume the compatibility layer covers everything.
Next steps
Three additions worth the effort, in order: a per-caller rate limit in front of /v1/chat/completions (one team should not be able to consume the shared fallback), a metrics counter per provider and status code so Grafana shows failovers without grepping logs, and a per-provider model map so a request for a model that provider does not host routes to the equivalent instead of 404ing.