Guide

Batching & concurrency

Two ways to run thousands of requests, and they are not the same thing.

Which one you want

What continuous batching actually is

Worth clearing up, because it is the most commonly confused term here. Continuous batching is a server-side technique. vLLM, TGI and every hosted provider run an iteration-level scheduler that slots new sequences into a forward pass as older ones finish, so the GPU never idles waiting for the slowest generation in a fixed batch.

No client library can perform it. What a client controls is whether the scheduler has anything to schedule — and a loop that awaits each request before sending the next leaves a continuous-batching server running a batch size of one. Same GPU, a fraction of the throughput, and you are billed per token either way.

Keeping N requests outstanding is the entire client-side contribution. That is what map_chat does.

Concurrent dispatch

import polygate
 
questions = ["Summarize this...", "Classify this...", ...]   # 10,000 of them
 
results = polygate.map_prompts(
    "openai", "gpt-4o-mini", questions,
    concurrency=16,
    on_progress=lambda p: print(f"{p.done}/{p.total} at {p.concurrency} in flight"),
)
 
# Same order as the input, one entry per prompt.
for item in results:
    if item.ok:
        print(item.index, item.response.content)
    else:
        print(item.index, "failed:", item.error)

Three things it gets right

Results stay in input order. Completion order is not submission order, so placing results by arrival attaches every answer to the wrong question — silently, with no error and plausible-looking output.

One failure does not kill the run. A single 400 in item 4,000 of 10,000 must not discard 3,999 answers you have already paid for. Errors are captured per item; pass raise_on_error=True when a partial result is useless to you.

Concurrency is bounded, and adapts. Fanning out over ten thousand prompts opens ten thousand sockets and earns an immediate 429. The limiter halves on a rate limit and recovers by one — additive-increase, multiplicative-decrease, the same shape as TCP congestion control and for the same reason: the right level is unknown, differs per key, and changes with load.

Full conversations

# Full conversations rather than bare prompts.
results = polygate.map_chat(
    "anthropic", "claude-haiku-4-5",
    [
        [{"role": "system", "content": "Reply in one word."},
         {"role": "user", "content": text}]
        for text in documents
    ],
    concurrency=8,
)
 
texts = polygate.concurrent.contents(results, default="<failed>")

Offline batch jobs

Several providers run a second, asynchronous tier: hand over a whole job, they process it within a window, and it costs roughly half. Currently supported:

Under the hood these could hardly be less alike — OpenAI wants a JSONL file uploaded and referenced by id, Anthropic takes the requests inline; their status vocabularies and result envelopes differ too. polygate normalizes all of it.

Submit

from polygate import BatchRequest, batch
 
requests = [
    BatchRequest(
        custom_id=f"doc-{i}",
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": text}],
    )
    for i, text in enumerate(documents)
]
 
# Returns immediately with a job id you can poll.
job = batch.submit("openai", requests)
print(job.id, job.status)     # batch_abc123 pending

custom_id is required rather than generated, because results come back unordered and often partially — matching by position would quietly mis-assign every answer in a job where one request failed. Duplicates are rejected up front, since a provider keys results by that id and the clash would otherwise surface hours later at retrieval.

Poll and collect

job = batch.status("openai", "batch_abc123")
print(job.status, f"{job.progress:.0%}", job.completed, job.failed)
 
if job.done:
    for result in batch.results("openai", job.id):
        if result.ok:
            print(result.custom_id, result.response.content)
        else:
            print(result.custom_id, "failed:", result.error)

Statuses normalize to pending, running, completed, failed, cancelled and expired. Successes and failures both come back — on OpenAI they live in two different files, and reading only the output file silently loses every failed request.

Or just wait

# Submit, poll, and return the results. Blocks for as long as the provider
# takes — which can be hours.
results = batch.run(
    "anthropic", requests,
    poll_seconds=60,
    on_progress=lambda job: print(job.status, f"{job.progress:.0%}"),
)

Convenient, and occasionally the wrong tool: prefer submit plus your own polling for anything that must survive the process being restarted. The batch id is durable; a stack frame is not. A timeout_seconds gives up *watching* — it does not cancel the job, which keeps running and can be collected later.

Providers without a batch tier

Calling batch.submit on one raises BatchNotSupportedError naming the alternative. polygate deliberately does not assume every OpenAI-compatible host implements the batch route as well as the chat route — most do not, and a request to a missing /batches returns a 404 that reads like an outage rather than an unsupported feature.