A nightly classification job should not take six hours. It usually does, because someone wrote a loop with 40 workers, hit 429s at minute three, added a sleep, and called it done. The models are not the bottleneck. The request pattern is.
Batch APIs flip that pattern. You hand over a file of requests, the provider runs them on capacity that is not competing with interactive traffic, and you get a results file within 24 hours. Both OpenAI and Anthropic charge half the synchronous price for it, which makes this the cheapest lever most teams never pull.
When batch is the right shape
Good fits:
- Classifying or tagging a backlog of tickets, reviews, or documents
- Backfilling embeddings after a chunking change
- Running an eval suite over thousands of cases
- Moderating a content dump
- Generating synthetic training data
Bad fits: anything a user waits on, anything that streams, anything where you retry within seconds. A 24-hour deadline is the product, not a limitation you work around.
OpenAI also runs batch traffic on a separate rate limit pool, so a big job will not eat the quota your product needs at 9am.
What you need
- Python 3.10+ with
openaiandanthropicinstalled - An API key for whichever provider you use
- A dataset with a stable primary key per row
That last item matters more than it sounds. The whole workflow depends on matching responses back to inputs, and your database ID is a better join key than array position.
Step 1: build the JSONL, keyed by your own IDs
OpenAI takes JSONL: one request per line, with a custom_id you choose.
import json
MODEL = "gpt-6-luna" # one model per file, and only one
with open("batch_input.jsonl", "w") as f:
for row in load_tickets():
f.write(json.dumps({
"custom_id": f"ticket-{row['id']}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": MODEL,
"messages": [
{"role": "system", "content": "Classify the ticket. Reply with JSON only."},
{"role": "user", "content": row["text"][:6000]},
],
"response_format": {"type": "json_object"},
},
}) + "\n")
Three decisions in that snippet are load bearing. custom_id is your ticket ID, not a counter, so results survive a reshuffle. The text is truncated at 6000 characters, because one 400k-token outlier rejects the line and you would rather notice that in code than in a validation report. And response_format is set per line, since each line is an independent request with the same body options as the synchronous endpoint.
Step 2: upload and create the batch
from openai import OpenAI
client = OpenAI()
with open("batch_input.jsonl", "rb") as fh:
uploaded = client.files.create(file=fh, purpose="batch")
batch = client.batches.create(
input_file_id=uploaded.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
print(batch.id, batch.status)
/v1/responses works the same way if you prefer that API. The endpoints batch accepts are a fixed list: responses, chat completions, completions, embeddings, moderations, images, and videos.
The per-batch ceiling is 50,000 requests and 200 MB of input. Embeddings batches also cap at 50,000 inputs across all requests. You can create 2,000 batches per hour, so a 500,000-row job becomes ten files, not one.
Step 3: poll without hammering
import time
while True:
b = client.batches.retrieve(batch.id)
counts = b.request_counts
print(b.status, counts.completed, "/", counts.total, "failed:", counts.failed)
if b.status in {"completed", "failed", "expired", "cancelled"}:
break
time.sleep(60)
The statuses you will actually see: validating, in_progress, finalizing, completed, expired, failed, cancelling, cancelled. A batch that misses its window lands in expired, and the requests that did finish are still in the output file.
Sixty seconds between polls is fine. The window is 24 hours. Polling every second buys nothing and turns one job into thousands of management calls.
Step 4: join results back, retry only the failures
raw = client.files.content(batch.output_file_id).text
rows = {}
for line in raw.splitlines():
item = json.loads(line)
body = item["response"]["body"]
rows[item["custom_id"]] = body["choices"][0]["message"]["content"]
Successful lines carry the full response body. Failed lines do not appear there at all; they go to the error file, addressed by error_file_id:
if batch.error_file_id:
errors = client.files.content(batch.error_file_id).text
for line in errors.splitlines():
item = json.loads(line)
print(item["custom_id"], item["error"]["code"])
Expired requests come back with batch_expired, and since custom_id is your ticket ID, rebuilding a retry file is a join, not an archaeology project. You are charged for the requests that completed even when the batch expires, so skipping the error file means paying twice for the same rows.
Step 5: the same job on Anthropic, different shape
Anthropic does not want a file. You send the request list in the create call, limited to 100,000 requests or 256 MB, whichever comes first.
import anthropic
ac = anthropic.Anthropic()
batch = ac.messages.batches.create(requests=[
{
"custom_id": f"ticket-{row['id']}",
"params": {
"model": "claude-sonnet-5-5",
"max_tokens": 300,
"messages": [{"role": "user", "content": row["text"][:6000]}],
},
}
for row in load_tickets()
])
print(batch.id, batch.processing_status)
Results stream out as JSONL through ac.messages.batches.results(batch.id), and each entry tells you what happened:
for entry in ac.messages.batches.results(batch.id):
if entry.result.type == "succeeded":
text = entry.result.message.content[0].text
rows[entry.custom_id] = text
else:
failed[entry.custom_id] = entry.result.type
result.type is succeeded, errored, canceled, or expired, which is a cleaner signal than parsing a response envelope. Most batches finish in under an hour, and results stay downloadable for 29 days, so a crashed job is recoverable without resubmitting.
Two constraints to design around: stream: true is rejected, and max_tokens must be at least 1. Tool use, vision, and extended thinking all work inside a batch, so a bulk run can do more than classification.
Step 6: the money, with real numbers
Take 500,000 tickets at roughly 800 input tokens and 150 output tokens each. That is 400 million input tokens and 75 million output tokens per run.
Claude Sonnet 5.5 costs $2 per million input and $10 per million output on the standard API, and half that on the Batches API: $1 and $5.
- Standard: 400 x $2 + 75 x $10 = $1,550
- Batch: 400 x $1 + 75 x $5 = $775
Same models, same prompts, $775 saved per run. Nightly, that is the difference between a line item someone questions and one they ignore.
Limits worth writing down
- One model per OpenAI input file. Mixing models means splitting files.
custom_idmust be unique. Duplicates collapse in your dict and you lose rows silently.- Batch results are not permanent. Anthropic keeps them 29 days; OpenAI gives you file IDs, not a promise. Write to your database the day the batch completes.
- Anthropic warns that batches can slightly exceed your workspace spend limit because of concurrent processing. Do not schedule a 100,000-request batch the hour your budget resets and expect a hard stop.
- Management endpoints have their own rate limits. A tight polling loop across fifty batches is its own incident.
Validation failures, and what they look like
The most common rejection is a malformed file, not a bad prompt:
- Blank lines or pretty-printed JSON across multiple lines. One object per line, no exceptions.
- A URL outside the accepted endpoint list.
custom_idmissing or repeated.- A body that the target endpoint would reject anyway, such as
stream: true.
The batch sits in validating, then flips to failed with an error file explaining which line. Print the first ten thousand characters of that file before you resubmit anything; the error usually repeats.
When not to use this
If a human is waiting, batch is the wrong tool, no matter the discount. For interactive traffic, prompt caching and a smaller model do more for cost than batching ever will.
If you run your own GPUs, the equivalent move is offline inference on vLLM against a queue you control. Same principle: separate bulk work from latency-sensitive work instead of interleaving them.
And if the job is a one-off with 200 rows, just loop synchronously. The engineering time costs more than the discount.
Start with a thousand rows
Run a 1,000-row batch end to end before pointing it at the full dataset. Check three things: every custom_id came back, the JSON parses, and the error file is empty. Fix the join while the blast radius is small.
The failure worth avoiding is not a failed batch. It is a 500,000-row batch that completes at 3am with 4,000 unmatched IDs, no error file review, and no way to tell which customers got skipped.