Runpod Batch Inference lets you submit deferred work to a queue-based Serverless endpoint using your own model and container. It is not a new model catalog or an interactive inference speed feature. The useful distinction is operational: work that can wait gets a different execution path from work a person is waiting to receive.
General availability was announced October 28, 2026. This guide explains a documented workflow and the engineering decisions around it. We have not measured Runpod batch throughput, start times or real workload savings.
What the product changes
You can reuse an endpoint’s model and handler rather than deploying a separate application for offline inference. Dedicated batch workers separate that work from normal interactive traffic. That separation is useful, but it does not promise that a large batch starts immediately or finishes before your particular deadline.
Embedding backfills, offline evaluations and document processing are natural candidates. A chat response or a user-triggered task with an immediate deadline is not. Keep those paths distinct in the application even when the model artifact is the same. Our AI deployment foundation covers the wider serving decision.
The official batch documentation requires a queue-based endpoint; load-balancing endpoints are not eligible. Check that before designing a client around an endpoint ID you already use. A reachable endpoint is not sufficient proof that this API path is supported.
Pricing without the wrong discount comparison
Runpod’s launch announcement advertises discounts of up to 30% against on-demand Serverless pricing, depending on GPU. H100, H200, B200 and B300 are specifically described at 20% off. Do not turn the headline into a blanket 30% rate or quote an unrelated provider’s 50% Batch discount.
| Cost component | Treatment |
|---|---|
| GPU execution | Discount depends on the eligible hardware and current quote |
| Token count | Useful workload metric, not a universal Runpod token price |
| Storage and other resources | Check separately for the selected setup |
| Cancellation | Work already running can still be billed |
There is no one dollar-per-million-token figure for arbitrary containers. A long-running image handler and an embedding handler on the same GPU do different amounts of work. Get the endpoint’s current quote, then estimate from representative execution rather than a model parameter count.
For comparison, price a completed and accepted result, including startup, failures and retained resources where applicable. Our Runpod pricing guide separates Pods, Serverless and storage; the Vultr comparison remains the broader provider decision page.
Limits that shape the client
Current documentation lists 5,000 requests per batch, ten active batches per endpoint and a default daily endpoint allowance of 50,000 batch requests. Higher allowances require support or sales approval, up to the published three-million-request level. The launch headline is not the default quota for every account.
Append calls are limited to 10 MiB each. Check serialized request size, not merely the number of documents. A large embedded payload can hit the byte limit long before the request-count limit. Store a local manifest mapping source records to submitted jobs so failed subsets can be identified later.
The timing model is asynchronous and can be multi-hour. Build a deadline policy that decides when to keep waiting, cancel or hand a remaining subset to another path. Do not automatically replay everything through the interactive endpoint when the queue takes longer than expected; that can create duplicate work and unexpected cost.
Create, finalize, observe, retrieve
The API lifecycle has four important stages:
- Create a draft batch with requests, or create an empty draft.
- Append any remaining requests while it is still a draft.
- Finalize the batch, making it eligible for execution and locking its input.
- Monitor processing and retrieve every page of results.
The documented paths are under https://api.runpod.ai/v2/ENDPOINT_ID/batch. Creation accepts a top-level JSON array of request objects; appending uses the requests endpoint and its own documented body shape. Do not reuse one payload shape for both operations without checking the reference.
Here is a shortened request-shape example, not a command executed for this article:
curl -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/batch" \
-H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" \
-d '[{"input":{"text":"A document to process"}}]'Do not finalize until the manifest and submitted count agree. Drafts do not run. Once finalized, persist the returned batch identity before a worker or client can restart. Use server-side secret management rather than placing the key in browser requests.
Status does not replace result accounting
The current API reference distinguishes draft, finalized, completed, failed and cancelled states. The launch walkthrough also illustrates processing while status remains FINALIZED. That label means the batch was submitted, not that its outputs are ready to use.
Inspect counts and request-level results as well as the batch status. A completed batch can contain unsuccessful individual requests. The reference also distinguishes raised handler errors from handlers that return an error-looking value as ordinary output. Application validation must catch the latter.
Results are paginated. Continue while the API indicates more pages and reconcile retrieved results with the manifest. “The first page looked good” is not a completion check. The AI Operations hub covers similar reliability boundaries across inference services.
Retry and cancellation safety
Give downstream writes a durable idempotency key based on the source record and processing version. A lost network response can leave you unsure whether work was accepted. Retrying without a manifest can duplicate both inference and side effects. Our idempotency foundation explains that distinction.
Prefer a handler that returns a result rather than sending emails or modifying customer state during inference. Apply outputs through a separate validated stage. This makes it easier to rerun a failed subset without repeating customer-facing actions.
Cancellation stops pending work, but in-progress work can finish and remain billable. Reconcile late results before assuming the cancelled batch has no remaining outputs. Record failures separately from cancellations and use API failure handling rather than an unbounded retry loop.
A useful first evaluation
Choose a representative sample with short, long and malformed inputs. Pin the container, model revision and handler configuration. Compare accepted output quality, queue wait, execution duration, failures and total bill. Keep source and output data handling consistent with your privacy requirements.
Run an interrupt-and-restart test on the client. Verify that it can resume polling and result retrieval from the saved batch ID. Test a handler returning a structured validation error, not just an HTTP failure. These are proposed acceptance checks, not claims about tests we have performed.
For LLM evaluation workloads, keep the dataset and scoring version alongside the batch manifest. Otherwise a model change and a dataset change become indistinguishable. See evaluation datasets and LLM regression testing.
My take
Runpod Batch deserves a separate guide because its draft/finalize/results lifecycle creates a real integration choice beyond ordinary Serverless. Use it for deferred workloads you can reconcile and retry safely. Choose an interactive path when a person is waiting, and never budget from the largest advertised quota alone.
FAQ
Is Batch Inference generally available?
Yes. Runpod announced GA on October 28, 2026. Availability does not remove account-specific limits or capacity considerations.
Does every GPU get 30% off?
No. The launch rates vary by GPU, with the named H100/H200/B200/B300 group at 20% off. Confirm the actual quote.
Can I use a load-balancing endpoint?
The current batch documentation requires a queue-based Serverless endpoint, not a load-balancing endpoint.
Does FINALIZED mean all work succeeded?
No. Inspect completion state, counts and per-request outputs. Finalization is the step that submits and locks the batch.
Does cancellation mean no bill?
No. Work already in progress can complete and be billed. Pending work and running work have different cancellation consequences.