Continuous Batching in LLM Inference Explained

Continuous batching is a scheduling technique for LLM inference servers that adds new requests to an in-progress batch and removes finished ones token by token, instead of waiting for a fixed group of requests to all complete before starting the next batch. It’s one of the main reasons modern LLM-serving frameworks handle far more concurrent users per GPU than naive batching does.

The problem with static batching

Batching multiple requests together is essential for GPU efficiency — running one sequence at a time leaves most of the GPU’s compute idle, since generating a single token is a memory-bandwidth-bound operation rather than a compute-bound one. Grouping requests together and processing them in the same forward pass amortizes that cost.

The naive way to batch is statically: collect a fixed group of requests, run the full generation loop for all of them together, and only start the next batch once every request in the current one has finished. The trouble is that requests don’t take the same amount of time. If one prompt asks for a two-sentence answer and another asks for a two-page one, the whole batch is stuck waiting on the slowest member — the GPU sits partially idle, processing padding tokens for requests that finished long ago, while new incoming requests queue up outside the batch entirely.

How continuous batching fixes it

Continuous batching (sometimes called iteration-level scheduling) operates at the granularity of a single decoding step rather than a whole request. After each token is generated for every sequence currently in the batch, the scheduler checks: did any sequence just finish (hit an end token or its length limit)? If so, it’s evicted immediately, and a new queued request can be inserted into the now-open slot — starting its own generation on the very next step.

The effect is that the batch composition changes continuously as generation proceeds, and the batch is kept as full as the hardware allows at all times, rather than shrinking toward the end of a fixed round and leaving capacity unused. Two changes make this practical:

• Per-sequence key-value cache management. Because each sequence in the batch can be at a different point in its own generation, the server needs to track a separate KV cache per sequence and be able to allocate or free that cache’s memory as sequences join and leave the batch, rather than assuming a uniform batch shape.
• Fine-grained memory allocation. Techniques like paged attention allocate KV cache memory in fixed-size blocks, similar to how an operating system pages virtual memory, so that inserting and evicting sequences doesn’t require contiguous memory or expensive copying.

Static vs. continuous batching

	Static batching	Continuous batching
Scheduling granularity	Whole batch	Per generation step
New requests	Wait for the next batch	Join mid-flight
Finished requests	Occupy a slot until the batch ends	Evicted immediately, freeing the slot
GPU utilization	Drops as the batch’s shortest requests finish	Stays close to full
Throughput under mixed request lengths	Poor	Strong
Implementation complexity	Simple	Requires per-sequence KV cache management

Scheduling and fairness

Admitting a new request the instant a slot opens up isn’t quite the whole story — a serving system also has to decide which queued request gets that slot when several are waiting. Simple first-come-first-served ordering is common, but production schedulers often weigh in additional factors: a request with a very long prompt takes longer to process during the initial prefill step than one with a short prompt, so naively interleaving them can starve short requests behind a long one’s prefill. Some servers separate prefill and decode into distinct scheduling phases for exactly this reason, admitting new sequences’ prefill work in between decode steps for the sequences already in flight, rather than letting a single long prefill block the whole batch.

Interaction with other serving techniques

Continuous batching is usually deployed alongside, not instead of, other inference optimizations. Speculative decoding speeds up how fast tokens are produced for a given sequence; continuous batching decides which sequences occupy the batch at any given moment — the two are complementary. Quantization shrinks the memory footprint of both model weights and the KV cache, which indirectly increases how many sequences can fit in a continuously batched server at once. And the choice between batch and real-time inference is a higher-level decision about workload shape; continuous batching is specifically the technique that makes real-time, interactive serving efficient when many users are generating tokens concurrently.

Why it matters for cost

Inference cost is dominated by GPU time, and GPU time is expensive whether or not it’s doing useful work. A server that keeps its batch full generates more completed tokens per GPU-hour than one that lets utilization sag between requests, which translates directly into lower cost per token served. That’s the same economic pressure behind techniques like prompt caching — anything that gets more useful output out of the same hardware-hour lowers the effective price, a factor worth keeping in mind when comparing providers with a token cost calculator.

It also improves the user-facing metric that matters most for interactive applications: time to first token and overall latency stay more consistent under load, because a new request doesn’t have to wait for an entire unrelated batch to finish before it even starts.

The takeaway

Continuous batching replaces batch-level scheduling with step-level scheduling: instead of waiting for a fixed group of requests to all finish together, an LLM server evicts finished sequences and admits new ones after every generated token, keeping the GPU’s batch full and utilization high. It requires per-sequence KV cache management to pull off, but it’s now the default scheduling strategy in production LLM-serving frameworks because it turns wasted GPU idle time directly into served throughput.