MTP in Practice: Benchmarking Gemma's Speculative Decoding on a Real GPU

In the concept edition, we saw that MTP (Multi-Token Prediction) is a "low-risk, high-reward bet," and that Qwen and Gemma implement this bet in completely different ways.

Fine, the theory makes sense. But how much faster does this actually make things in practice? Is the bet really paying off?

In this installment — the implementation/benchmark edition — we'll actually run Gemma's MTP on llama.cpp and measure, with real numbers on an ordinary consumer PC, just how much of an effect it has.

Running It in llama.cpp

For Qwen Models

Run with the following arguments added. Specifying spec-type activates the model's internal MTP drafter. A dedicated model is provided for MTP.

/opt/llama/bin/llama-server --model /opt/llama/models/Qwen3.5-9B-MTP-UD-Q3_K_XL.gguf \
-t 4 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -ctv q8_0 -ctk q8_0 --fit off --no-warmup --no-cache-prompt -c 32768 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 \
--reasoning on

Argument summary:

• --spec-type draft-mtp: specifies the speculative decoding type (draft-mtp for Qwen).
• --spec-draft-n-max 4: how many tokens ahead to speculate at most. Setting this too high increases the penalty when a guess fails.
• Also note: concurrent processing isn't supported, and multimodal input isn't supported either.

For Gemma Models

Run with the following arguments. For Gemma, the drafter-MTP model is bolted on externally, so you enable MTP by pointing the --model-draft argument at that file.

/opt/llama/bin/llama-server --model /opt/llama/models/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \
--model-draft /opt/llama/models/mtp-gemma-4-12B-it.gguf \
-t 4 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 --host 0.0.0.0 --port 8001 \
--device CUDA0 -mg 0 -sm layer --fit on -fa on -c 163840 -ctv q8_0 -ctk q8_0 \
--no-warmup --no-cache-prompt --cache-ram 0 \
--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-device CUDA0 \
--chat-template-kwargs '{"enable_thinking":true}'

Argument summary:

• --model-draft <gguf file>: specifies the draft model.
• -sm layer: how the model is dispatched across multiple CUDA devices. At the time of writing, only layer-level distribution is supported.
• --spec-type draft-mtp: specifies the speculative decoding type.
• --spec-draft-n-max 4: how many tokens ahead to speculate at most. Setting this too high increases the penalty when a guess fails.
• --spec-draft-device CUDA0: the device that runs drafting. Required when multiple CUDA devices are present.

Benchmark: Using MTP on Gemma-4-12B-it

In this section, to investigate how effectively MTP actually works and what kind of benefit it brings to users, we compared performance with and without MTP.

To also demonstrate that this kind of model verification is achievable even on ordinary consumer hardware, the benchmark was run on the author's own personal machine.

Test Environment

The experiment used the following simple test setup.

• Machine: modified HP Z440 Workstation — Intel® Xeon® E5-2690 v4 x1, 48GB DDR4 RDIMM, Intel® DC S3700 Datacenter SSD (400GB SATA), NVIDIA GeForce RTX 3060 (Ampere, 12GB GDDR6 VRAM).
• CUDA: 13.1
• Engine: llama.cpp Build 9574
• Model: gemma-4-12B-it-qat-UD-Q4_K_XL.gguf — the base model is Gemma-4-12b-it-QAT, a Google DeepMind model trained with quantization-aware training in mind, on top of which we use the UD-Q4_K_XL model[2], dynamically quantized by Unsloth[1], which provides open-source fine-tuning tools.
• Drafter model: mtp-gemma-4-12B-it.gguf, downloaded from the same repository as the base model — the Assistant model prepared by Unsloth for Gemma-12B, simply quantized to 4-bit, with the maximum number of speculated tokens set to 4.
• KV cache: quantized to 8 bits to fit efficiently into VRAM.
• Context size: 163,840 tokens, capped at 160k to stay within VRAM capacity.
• Prompt cache: off. Reasoning: on.

Test Cases

Goal: measure and compare speed with and without MTP. Only speed is measured — the actual content of the generated output is not evaluated at all. We measure elapsed time, token count, and token speed during generation, taken directly from llama.cpp's own log output.

Method: using the llama.cpp web frontend, we asked the following four turns of questions.

• 1. "Write JavaScript code for Breakout (as an HTML page)."
• 2. "Make it look cooler."
• 3. "Slow the ball's movement down a bit."
• 4. "Check for bugs and optimize it."

Resource Usage

Memory usage was as follows. The memory consumed specifically for MTP came to 759.68 MiB of VRAM on the GPU side and 324.79 MiB of RAM on the CPU side.

Since we used a 4-bit quantized model for the draft model here, choosing a 16-bit model instead would consume roughly 4x this amount for the weight data. Anyone planning to use a 16-bit model should keep this in mind.

Table 1. Memory usage in this test environment. Units: MiB.

Token Speed Comparison Results (MTP On vs. Off)

Comparing token ingestion speed turn by turn, we saw no major variation. With MTP enabled, ingestion speed dropped slightly compared to without MTP, coming in at roughly 70–73% of the baseline performance (see the "Read" rows in the table below).

Token generation speed, on the other hand, was roughly twice as fast overall with MTP enabled (see the "Generate" rows).

Below are the actual measurements. Note that time is measured in ms.

Table 2. Actual data from the logs.

Without MTP, the whole run took about 293,705 ms in total — just under 5 minutes — while with MTP it completed in about 173,150 ms, just under 3 minutes. That's roughly a 40% reduction in time for this case.

To look more closely at the trend, we also checked how speed changed as token output progressed within each turn (again using the logs).

Below is the token output profile without MTP. Speed dips slightly but stays roughly flat overall, hovering in the mid-30s to low-40s tps across all four turns from start to finish.

Generally, with token-by-token generation, output speed tends to gradually decline as context size grows. How much it degrades depends on the model, but this can become a non-trivial problem as more turns accumulate.

By contrast, here is the token output profile with MTP enabled. Speed starts around 35–40 tps early on and climbs to 60–65 tps later, showing that tokens come out faster and faster as the conversation progresses.

A quick note before we go further: I'm a native Japanese speaker, and this model is my daily driver, so the test prompts and responses are in Japanese rather than English. Tokenizer behavior can differ quite a bit across languages, so this also happens to double as a look at how MTP performs outside the English-first benchmarks you usually see.

Next, we examined the actual content of the responses and found that each one opens with a Japanese explanation followed by code. (The original screenshot of that exchange is in Japanese, so it isn't reproduced here — the token breakdown below covers what matters for this analysis.)

Based on that, we checked the token length of the Japanese explanation portion at the start of each turn, with the following results.

Table 3. Token length of the Japanese explanation at the start of each turn.

We can't fully prove causation here, but turns where speed rose sharply (Turn 1, Turn 3) tended to have less Japanese output, while turns where speed rose more gradually (Turn 2, Turn 4) tended to have somewhat more Japanese output. This suggests the code-generation portion is where MTP's effect shows up most strongly.

A likely reason is that, regardless of programming language, code follows fairly strict grammar rules, making it relatively easy to predict what string comes next, and since the characters are essentially alphabetic, predicting the character type is also easier.

Even outside the code-generation portions, we confirmed output speeds of 40–50 tps right from the start — faster than the conventional approach — suggesting MTP was working effectively there too.

The author believes this largely comes down to characteristics of the Gemma tokenizer. The Gemma tokenizer has a vocabulary of roughly 256k entries. Whereas many other tokenizers fall back to character-level tokenization for Japanese, Gemma's tokenizer is able to register most Japanese text as whole "words."

Because of this, the model can predict the next token at the "word" level for Japanese much like it does for English, which we believe is why Japanese output was noticeably faster with MTP than without.

Checking Token Acceptance Rates

llama.cpp is designed to log an "acceptance" value showing what fraction of tokens were accepted during the verification phase when using an MTP model. Here's an example:

prompt eval time = 9559.06 ms / 7030 tokens ( 1.36 ms per token, 735.43 tokens per second)
eval time = 45531.93 ms / 2858 tokens ( 15.93 ms per token, 62.77 tokens per second)
total time = 55090.99 ms / 9888 tokens
graphs reused = 3983
draft acceptance = 0.82931 ( 2196 accepted / 2648 generated)

Using this value, we checked acceptance for each turn, with the following results.

Table 4. Token acceptance rate during the test cases.

Table 4 (summary). Max / min / average acceptance across the four test-case turns.

As mentioned above, likely because the output contained a lot of code, these tokens showed a high acceptance rate. So what does the acceptance rate look like for everyday use? Based on several days of logs, we recorded the maximum, minimum, and average, shown below.

Table 5. Token acceptance rate for everyday interactions with the Gemma-4-QAT-Assistant model.

Acceptance roughly halved, and speed dropped somewhat as a result. The corresponding speeds are shown below.

Table 6. Token speeds under the conditions in Table 5.

So while this isn't exactly blazing fast, it's reasonable to say it's still faster than running without MTP at all.

Summary

In this article, we looked at the new "MTP" technique — what it is, and how it actually changes generation speed in practice.

In a word, MTP is essentially the "speculative execution" of CPUs, brought into the world of LLMs. When it lands, it can be dramatically faster; when it misses, it can be slower — a slightly gambling-like trait.

We found this feature pairs especially well with code generation. Because programming languages follow strict grammar, once you're partway through a line, it's relatively easy to predict what string should come next. This lets MTP compensate for the parts where the tokenizer tends to slow things down, and our benchmark reflected that with solid results.

The current strength of LLMs comes precisely from their sequential nature — deciding the next word based on the words that came before. MTP, because it speeds things up without sacrificing that strength, is likely to play a genuinely important role as this technology continues to develop.

It's still early days, and there are some constraints, but we think this is a technique well worth watching as it matures and spreads.

Even as we speak, other groups are developing and releasing their own versions of this idea:

• DeepSeek (China): a token-prediction technique called DSpark.
• Z-Lab at UC San Diego: a token-prediction technique called DFlash.

These, too, are spreading quickly now that inference engines are starting to support them. We hope to take a closer look at these technologies in a future article.

Supplementary Information (llama.cpp Status, Gemma-4 Architecture)

llama.cpp Support Status

• 2026/5/16: Support for Qwen's MTP feature merged (#22673).
• 2026/5/22: A further updated version merged — logit computation skip optimization (#23433).
• 2026/6/8: PR adding Gemma-4 MTP support merged (#23398), enabling conversion to GGUF format including the MTP drafter.
• 2026/6/8: Support for the Assistant drafter models for Gemma-4 E2B and E4B merged (#24282); a bug caused Gemma-4-E4B specifically to malfunction, apparently fixed in the PR below.
• 2026/6/30: Fix for the Gemma-4 E4B bug (#25148) merged.

Gemma-4 Model Architecture

Below is the architecture of the Gemma-4-12B-it model.

Input strings ─┬─ Vision/Audio encoders (minimal, mostly folded into the model) ─┐
                └─ tok → Emb ──────────────────────────────────────────────────┴─(+)─┐
                                                                                       │
     ┌── RoPE / p-RoPE ───────────────────────────────────────────────────────────────┤
     │                                                                                 ▼
     │      [ LA → LA → LA → LA → LA → GA ]  x6 groups   →  LNR → SoftMax → Output Probabilities
     └──────────────────────────────────────────────────↑
            (48 blocks total = 6 layers x 8 groups)

• Max context size: 256k tokens (limited to 160k in our test environment).
• Blocks: 48 overall (6 layers × 8 groups). Hidden size: 3,840.
• FFN type: Dense — activation function Gelu_Pytorch_tanh, activation dimension 15,360.
• The attention structure is a hybrid of Local Attention (LA) and Global Attention (GA), a structure that debuted with Gemma-4. Local Attention uses sliding window attention for local context — low compute cost, fast, low memory. Global Attention uses group query attention for accurate whole-sequence understanding — a high-performance coordinator for long context and complex reasoning.
• Unified Encoder: integrates audio/vision encoders into the model; only the CLIP portion remains an externally applied encoder.
• Per-Layer Embedding (PLE): keeps embedding information independent at each layer, contributing to richer representational power during inference.
• KV Cache Sharing: shares and synchronizes the KV cache (normally kept independent) to achieve more consistent inference — when dispatching across multiple GPUs, this causes frequent inter-GPU communication, so caution is needed without NVLink.
• DualRoPE: Local Attention layers use RoPE, embedding token position via rotation matrices for reliable long-text positioning. Global Attention layers use p-RoPE[3], a modified RoPE that rotates only the top p% of K/V matrix pairs — suppressing noise and clarifying positional information for the coordinating Global Attention layers.

Gemma-4-12B-Assistant Model Architecture Details

The Gemma-4-12b-it-assistant model works together with the main model to predict tokens as follows.

Main model (Gemma-4-12b-it):
  Input string → tok → Emb → [LA...LA(45) → GA(46) → GA(47)] → LNR → SoftMax → N+1 output token
                                     │ shares KV cache (layers 45-47) with the Assistant's LA/GA
                                     ▼
Assistant model (MTP-Drafter):
  Query (H_N^0 + Emb of N) → LNR → LA → LA → LA → GA → Norm ─┬→ LNR → predicted state vectors
                                                              └→ Masked Embedder → predicted Token IDs (n tokens)
                                     │
                                     ▼
Verification (inside the inference engine):
  compares the Assistant's predicted state vectors against the Main model's own state vectors
  → decides how many tokens to accept → token output correction → final output

The "Assistant" model that adds MTP capability to Gemma-4-12B-it keeps Gemma-4's architecture but trims its structure down to a genuinely minimal design.

• Max context size: 256k tokens. Blocks: 4 overall (a single group of 3+1). Hidden size: 1,024.
• FFN type: Dense — activation function Gelu_Pytorch_tanh, activation dimension 8,192.

Based on the input string, it predicts the next (N+1) token. To do this, it combines the generated state vector h_N⁰ with the embedded vector of token N, and feeds the result in as a query to begin processing.

Gemma-4's attention layers normally operate as self-attention, but the structure inside the Assistant model uses cross-attention instead.

For KV data, it combines the LocalAttention block at layer 46 of the main model with the LocalAttention blocks at layers 0–2 on the Assistant side, and also integrates the GlobalAttention block at layer 47 of the main model with the GlobalAttention block at layer 3 on the Assistant side, via a shared KV cache. Through this, KV information is bound based on the query.

The output consists of two parts: the predicted tokens — the n tokens predicted by the Assistant model — and the state vector used during that prediction process.

These outputs are sent combined, just before the main model's attention blocks. They pass through all of the main model's attention blocks, generating, in parallel, the state vectors needed for verification.

Once the state vectors to be compared are ready, their contents (probability distributions) are compared to decide how many tokens should be accepted. This is handled in parallel by the inference engine (for example, the generate function in the transformers library), which outputs the resulting number of tokens to accept.

Based on this result, the candidate tokens — accounting for the accepted token count — are combined with token N+1 and passed on for output. This is how the process achieves faster output compared to conventional step-by-step inference.