Multi-token prediction (MTP) speeds up language model decoding by drafting several candidate tokens in a single pass and verifying them all at once against the target model. The central challenge lies in choosing the speculation horizon: a horizon that is too long risks having much of the draft rejected and wasted, while one that is too short limits speedup potential. Unlike standard autoregressive drafting, where generation can be halted token by token, MTP requires the horizon to be fixed in advance, since all prediction heads fire simultaneously in a single forward pass.
Researchers propose framing this horizon selection as an online control problem. Their method, called Adaptive Multi-Token Prediction (AMTP), is a lightweight controller that uses verification feedback to learn, context by context, how far a draft can be trusted. The approach requires no retraining of the underlying model and leaves the exact verification mechanism untouched. Before each decoding round, AMTP estimates the likelihood that a draft prefix will be fully accepted, then selects the horizon expected to yield the highest number of tokens per second.
The authors also provide a theoretical grounding for the approach: they show that an online learner's prediction error translates directly into throughput loss relative to the best possible horizon for each context, and ultimately into a bound on wall-clock completion time. Using online gradient descent, this regret grows sublinearly, which guarantees the controller converges progressively toward near-optimal behavior over time.
On the experimental side, AMTP was tested across five draft-target model pairings, including Llama-3.1-8B-Instruct with Medusa heads, Meta's own MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on the AIME 2025 and GSM8K benchmarks. Results show decoding speedups of up to 4.1x compared to standard decoding, with throughput gains of up to 10% over a fixed speculation window, matching or beating it in eight out of ten configurations. The work reflects a broader trend in LLM inference research: squeezing more speed out of existing architectures through smarter control mechanisms rather than structural changes, a pragmatic direction likely to find quick adoption in production inference pipelines.