MTP: The Low-Risk, High-Reward Bet Behind Faster Generation

What if you could make a low-risk, high-reward bet? You'd probably take it without hesitation.

As it turns out, LLM text generation has exactly this kind of bet quietly built into it.

Have you ever used an AI chat and thought, "I wish this answered a little faster"? That feeling gets stronger with "Thinking" models that reason carefully before answering, or with coding agents that iterate through trial and error. For AI systems that burn through thousands, even tens of thousands, of tokens to reach a good answer, generation speed directly shapes the user experience.

So why does AI text generation take so long in the first place? The reason is simple: LLMs can only produce one token at a time. To decide the next word, the model has to re-run its massive computations from scratch, taking into account everything generated so far. Every single token requires rebuilding this heavy computation from scratch. That cost adds up, and the longer the response, the longer you wait.

This raises a natural question: "Instead of dutifully recomputing everything one word at a time, why not glance a few words ahead while you're at it?"

That idea has, in fact, been turned into a real technique. It's the subject of this article: MTP (Multi-Token Prediction). Every time the model generates a word, it simultaneously computes, almost as a freebie, a guess at what the next few words are likely to be. If the guess turns out right, those guessed tokens get accepted all at once, confirming several words in a single step. If the guess is wrong, the model simply falls back to generating one token at a time as usual. The downside is minimal; the upside, when it pays off, is a big speedup.

This article (the concept edition) walks through how this "small bet, big win" mechanism is actually implemented inside real models (Qwen and Gemma), and how the two approaches differ. In the next installment — the implementation/benchmark edition — we'll actually run this technique and measure exactly how much faster it gets. Later in the series, we'll also cover "DFlash," which pushes this idea even further.

What Is MTP, Exactly?

MTP (Multi-Token Prediction) is a technique where an AI predicts several tokens ahead simultaneously while generating text. By predicting tokens in advance, the model can confirm multiple tokens at once when the prediction turns out correct, speeding up output.

Similar efforts exist elsewhere: instead of generating text sequentially, some "diffusion" models generate text more like image generation, producing output in a more randomized order to speed things up (as of 2026, Google DeepMind and ELYZA, among others, appear to be researching diffusion language models). However, output quality for these remains unstable and hasn't reached a practical level yet.

At least as of 2026, MTP is arguably the most reliable LLM speedup technique available today.

Which Models Support MTP?

The models currently supporting MTP fall mainly into two categories. Each uses a different implementation approach, so support in inference engines needs to be checked individually.

Qwen-3.5 / 3.6: MTP is natively baked into the model itself — the model and its MTP drafter are a single package. It's supported out of the box in engines like vLLM. llama.cpp added experimental support relatively early on, and Unsloth has released an MTP-enabled model for Qwen-3.5-9B in GGUF format.

Gemma: MTP is achieved by pairing the model with a separate model called "Gemma-4-Assistant" — think of it as an add-on bolted onto the main model rather than something built in. llama.cpp support for this arrived somewhat later than for Qwen.

The Basic Mechanics of MTP

Let's look at how behavior differs with MTP disabled versus enabled.

Conventional inference (token-by-token). One forward pass predicts one token; the next step runs another forward pass. Because output comes one token at a time, this is relatively slow.

Step N (full forward pass) → token generated → Step N+1 (full forward pass) → token generated → ...

Inference with MTP. A single forward pass computes a certain number of tokens ahead.

Step N (forward pass, run once) ──┬─→ Token N     [confirmed]
                                   ├─→ Token N+1   [speculative / bonus]
                                   └─→ Token N+2   [speculative / bonus]

• Token 1: output as confirmed.
• Token 2 onward: probability values are calculated in parallel as speculative values (a "bonus").
• Because output comes in multi-token units, this is relatively faster than the conventional approach.

Handling Predicted Tokens (Speculative Decoding)

Predicted tokens go through a follow-up step where they're checked against the correct answer; once they clear the acceptance criteria, they're confirmed all at once.

• 1. Draft. A lightweight method quickly predicts upcoming tokens.
• 2. Verification. The production model computes the probability distribution for the predicted tokens in parallel.
• 3. Acceptance. The system checks how closely the probability distribution from verification matches the distribution the model would have produced without MTP, and determines whether it meets the acceptance threshold. If it matches, the tokens up to that point are accepted as a "bonus" and confirmed all at once. If it doesn't match, drafting is cut off at the point of mismatch, and the result falls back to the production model's own inference.

Draft → Verification → Acceptance ─┬─ match ────→ accept all at once (bonus!)
                                    └─ mismatch ─→ cut off, fall back to the
                                                    production model's answer,
                                                    then Draft again

Because tokens appear to be generated "simultaneously," this is often mistaken for a diffusion model, but the underlying process is still strictly sequential.

How the Two Approaches to MTP Differ

As mentioned, this article covers two implementations, and they differ considerably in their starting point, goals, and mechanics. Let's look at each in more detail.

Qwen's Approach to MTP

Qwen's approach is said to build on techniques researched for DeepSeek-V3, and has been used starting with Qwen3-Next.

The reason this approach is built directly into the model is that the technique itself originally started out as "a method for training a smarter model." MTP turned out to be useful, and the circuitry built into the model for that purpose is now also used, as a side effect, for speculative token prediction.

Input token (N) → Qwen3.5 Main Model → state h_N^0 → Token N+1 [confirmed]
                                             │
                                             ▼ (also feeds the MTP path)
                                       MTP Module 1 → h_N^1 → Sampling & accept
                                             │
                                   match? ───┴─── no match?
                                     │                │
                        predicted token accepted   main model's token adopted
                          → confirmed as N+2         → confirmed as N+2, chain stops
                             │
                             ▼ (only if accepted)
                       MTP Module 2 → h_N^2 → Sampling & accept → ...same check for N+3

Here's how it works, broken down. Even with MTP disabled, when a token is input, the model's main processing generates a state h_N⁰ based on the token sequence so far, which is then detokenized to produce token N+1.

When MTP is enabled, as soon as state h_N⁰ is produced, the requested number of MTP modules are created. If n tokens' worth of prediction is requested, n modules are prepared, each producing an embedding Emb_Nⁿ that corresponds to a predicted word, based on state h_N⁰.

After that, as shown in the diagram above, the embedding information for token N+1 — the token that would have been output even without MTP — is combined to produce state h_N¹. This state is sent to the main model, where its sampling and acceptance mechanism checks whether it matches what the main model would have predicted on its own. If it matches, that token is accepted and the process moves on to predicting the next token. If it doesn't match, the state predicted by the main model is used instead, and verification stops there.

The defining feature is that the prediction and verification mechanisms are chained together one token at a time, forming a sequential flow throughout.

Gemma's Approach to MTP

Gemma's approach was built from the ground up for speed, and its key difference from Qwen is that it processes prediction and verification together, in a batch.

Input token (N) → Gemma 4 Main Model → state h_N^0 → Prediction: N+1
                                             │
                                             ▼ (KV-cache update)
                                         KV-Cache ←──────────────┐
                                             │                   │ (shares cache)
                                             ▼                   │
                                    Gemma 4 Assistants ──────────┘
                                             │  (generates sequentially inside the
                                             │   draft model, then sends as a batch)
                                             ▼
                            Prediction: N+2, N+3, N+4, ... (candidate list)
                                             │
                                             ▼
                     Main model: causal-attention masking, probabilities computed
                     in parallel  ──┬── include only as many as fit into the output
                                    └── if none fit, exclude from output

In Gemma, the draft model is external. Once the main model confirms the first token, the draft model predicts, all at once, however many tokens are allowed.

The main model first receives the input tokens, generates a state, detokenizes it, and outputs token N+1. At this point, the draft model shares a KV cache with the main model (in Qwen's case, the MTP component and the main component maintain independent KV caches).

Upon receiving the updated KV cache, the draft model generates however many predicted tokens are needed, and hands them off to the main model. The main model then checks, in a batch, whether these predicted tokens are correct using probability distributions, and determines how many of them are acceptable.

As a result, however many predicted tokens the main model accepts get output together with token N+1, all at once.

Which Approach to MTP Is Better?

Because the two approaches to MTP start from different premises, it's hard to say one is unconditionally superior. That said, evaluating primarily on maturity and compatibility with inference engines, as of July 2026, Gemma's approach appears more mature.

First, there's a difference in the scope of sequential processing. Because token prediction fundamentally assumes "predicting the next token based on the state of the previous one," the process is inherently sequential. Comparing the flow of each approach, with time running left to right:

Qwen:   Input(N) → Main model → Predict N+2 → Judge N+2 → Predict N+3 → Judge N+3 → ...
                        │             confirmed:N+2 ↑          confirmed:N+3 ↑
                        └→ confirmed: N+1

Gemma:  Input(N) → Main model ──┬→ Predict N+2 ─┐
                        │       ├→ Predict N+3 ─┼→ Judge (batch) → up to n OK → confirmed: N+2, N+3, N+4, ...
                        │       └→ Predict N+4 ─┘
                        └→ confirmed: N+1

With Qwen's approach, the main model predicts token N+1, uses that state to predict token N+2 and passes it to the main model, then predicts token N+3 only after seeing the verification result — repeating this process step by step.

By contrast, with Gemma, after predicting token N+1, the draft model predicts however many tokens are needed all at once (starting from N+2), and hands them to the main model as a single batch. Processing on the main model's side is also parallelized, making this approach far more efficient than Qwen's. Because of this structural difference, Gemma's approach comes out ahead.

Second, there's a difference in structural flexibility. Because Qwen's MTP functionality is integrated into the main model, improving the drafting logic requires additional post-training. Gemma, on the other hand, keeps the draft model separate, so it can simply be swapped out whenever better logic becomes available. This separation is also advantageous operationally.

Finally, there's the ease of implementation in inference engines. Looking at llama.cpp's implementation, Gemma's approach can run even in multimodal configurations, while Qwen's approach doesn't support multimodal use. This difference stems from the fact that Qwen embeds MTP internally, requiring the branching logic to be handled inside the model itself. Gemma's approach, by contrast, simply toggles the feature on or off depending on whether input passes through the draft model, which is a more favorable structure for engine-side implementation.

Summary (And a Preview of What's Next)

MTP, at its core, is a betting mechanism: confirm one token, and while you're at it, speculatively predict a few tokens ahead almost for free — though pushing the bet too far by speculating too many tokens ahead can add enough overhead to slow things down — then confirm them all together if the guess is right. Qwen bakes this bet directly into the model itself; Gemma hands it off to a dedicated external assistant.

In the next installment — the implementation/benchmark edition — we'll actually run both approaches on llama.cpp and measure, with real benchmarks, exactly how much faster they get and how often the bet actually pays off. Later in the series, we'll also cover "DFlash," which takes this idea even further.