Speculative Decoding Speeds Up AI Models by Changing the Work, Not Reducing It

Speculative decoding speeds up language models by pairing a small draft model with a larger one that checks proposed tokens in a single forward pass. It does not reduce total computation; FLOPs actually rise. Instead, it uses spare GPU compute left idle during memory-bound decoding, so a single request can finish faster.

Speculative decoding makes language models faster, but not because it does less work. That is a common misconception, and it is wrong. For the same sequence of text, the total number of floating point operations, or FLOPs, actually goes up. So what does a good draft model buy, if not computing efficiency? According to a technical explainer on the topic, the answer lies in the type of work the GPU is doing.

How does speculative decoding work?

The technique works by pairing two models. A small draft model proposes a likely sequence of tokens. A larger model then checks that entire sequence in a single forward pass. Traditional decoding, by contrast, produces one token at a time, and each step requires a full trip through the model.

Memory is the bottleneck

A GPU does two broad kinds of work. One is the math itself, such as matrix multiplies and other tensor operations. The other is moving data from global memory to local memory and back. When the chip sits idle waiting for data to arrive, the operation is considered memory bound. When all the data is in place and the chip is busy finishing the math, the operation is compute bound.

Why is memory the bottleneck when decoding language models?

Single-token decoding is heavily memory bound. For every token it generates, the system has to stream all of the model’s active parameters. The explainer uses Qwen3.5-27B as an example, which has 27 billion parameters. Batched decoding, where thousands of tokens are processed in one step, tends to be compute bound instead.

That imbalance leaves capacity on the table. When decoding is memory bound, the GPU has spare computing power it is not using. If a system decodes two sequences instead of one, it takes no more time per token. Speculative decoding puts that spare capacity to work.

Breadth versus depth

The explainer illustrates the tradeoff with a simple example. Suppose the optimal number of tokens for a given forward pass is five. Running fewer leaves the GPU memory bound. Running more pushes it into being compute bound.

A system could handle that load in two ways. It could run five separate sequences, each decoding one token at a time. That approach spreads progress across many sequences, which the explainer calls “breadth.” Or it could use speculative decoding to run five tokens within one sequence, which the explainer calls “depth.”

In both cases, the GPU does roughly the same amount of work. In the first case, all five sequences move forward a little. In the second, one sequence moves forward a lot. The difference is that speculation lets a single request finish faster, as long as the system has room to spare.

When does speculative decoding speed up AI models?

That room disappears under heavy load. If a system adds more sequences and speculates to the same depth, it slows all of them down, because the GPU becomes compute bound. The benefit appears at small batch sizes, where the batch size is well below the optimal number of tokens.

Why the extra work matters

The distinction matters once a system serves real traffic. If the optimal batch size is five but speculative decoding is still switched on, the GPU may be spending compute on work that could have gone toward moving memory. The extra work also grows costly when the acceptance rate falls short of 100 percent, since every rejected token is wasted computation.

For that reason, the vLLM inference library lets operators turn speculative decoding off at high throughput. The setting is controlled by a parameter called --speculative-disable-by-batch-size.

The analysis also raises a design question. Why speculate to the same depth at every step? A system could speculate deeply when it has spare compute at small batch sizes, then move to shallower speculation, or even set the depth separately for each sequence, when the batch grows. The explainer describes this as the core intuition behind research ideas such as dSpark.

Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.