DFlash-2: Benchmarking Z-Lab's Successor to DFlash for Accuracy and Throughput Gains

A while back, we covered DFlash, a draft-token prediction technique that uses a diffusion model. At the time, we tested it on Gemma-4-12b-it-QAT, and the native Assistant model came out ahead — DFlash wasn't able to show a clear advantage.

Recently, though, a successor called "DFlash-2" surfaced, with a number of enhancements on top of the original design.

As of August 2026, only a handful of models support it. Using the base model alone, DFlash, and DFlash-2 in turn, we looked at where each one actually pays off, and what parameter settings get the most out of it.

What Is DFlash-2?

DFlash is a technique that predicts upcoming output tokens ahead of time to speed up generation, built by Z-Lab. Several DFlash models have been released for different LLM families.

[Input Token (N)] --> [Main Model] -- h_on ----------------------------------------> [Predicted: N+1]
                                       |                                             ^
                                       v                                             |
                                  [KVCache]                                          |
                                       |                                             |
+--------------------------------------+--------------------------------+            |
| DFlash Drafter                       v (KV data injection)            |            |
|                                 [KVCache]                             |            |
|                                      |                                |            |
|                              [Diffusion Model]                        |            |
|                                      |                                |            |
|                               [Token Decoder]                         |            |
|                                      |                                |            |
|                            +---------v--------+                       |            |
|                            | Predicted N+2    |                       |            |
|                            |       +          |                       |            |
|                            | Predicted N+3    |                       |            |
|                            |       +          |                       |            |
|                            | Predicted N+4    |                       |            |
|                            +---------+--------+                       |            |
+--------------------------------------+--------------------------------+            |
                                       |                                             |
+--------------------------------------+---------------------------------------------+-------+
| Processing inside Main Model         v                                             |       |
|                             * Uses causal attention to mask;                       |       |
|                               computes probs in parallel                           |       |
|                                      +------------------------------+              |       |
|                                      v                              v              |       |
|                         (Match found)                  (No match)                  |       |
|                         Include matching portion       Nothing included in output  |       |
|                         -> Predicted: N+2                                          |       |
|                         -> Predicted: N+3                                          |       |
+------------------------------------------------------------------------------------+-------+

Figure 1. The DFlash mechanism (this structure is unchanged in DFlash-2; reused from our earlier DFlash piece).

DFlash speeds up draft generation by using a non-causal model (the diffusion model shown above), aiming for better throughput than conventional autoregressive drafting.

The tradeoff is that the diffusion model also introduces some accuracy loss — in our earlier piece, it fell behind Gemma-4's Assistant model in a head-to-head comparison.

inco.ai has been working on fixing that. Details on the company are scarce, but the maintainer of inco.ai's page on Hugging Face is Zhijian Liu, who also heads Z-Lab — so it's likely a spinoff team or startup connected to the lab.

DFlash-2 adds two things on top of the original mechanism:

• Lightweight Path Selector — picking the correct token ordering. The original DFlash's strength was that its internal diffusion model could output a whole block of tokens, in order, in one shot. The catch is that each token's ordering is generated independently, so the resulting sequence could end up internally inconsistent, and it looks like that was a common reason predictions got rejected (indeed, compared against Gemma-4-Assistant, we saw a large gap in acceptance values, which we confirmed was reflected directly in the difference in output speed).
• Local Convolution — fixing the drop-off in accuracy toward the end of a block. This layer limits information exchange to each token's immediate neighbors when linking tokens together. Working alongside the Path Selector, it counters "suffix decay" — the sharp drop in acceptance rate for predicted tokens toward the end of a block. The Local Convolution layer is inserted in multiple places between the attention and MLP layers.

Put together, DFlash-2 is best described as an enhancement of DFlash that specifically improves the accuracy of its draft-token ordering.

Two models are currently available:

• Qwen3.8-27B-DFlash2
• Muse-Glimmer-30B-DFlash2

Both are published on the Z-Lab and inco.ai repositories, with GGUF builds available for llama.cpp.

We'd originally planned to run this on Gemma-4-12b-it-qat, but since that's the only pairing currently offered, and given the lighter inference footprint, we decided to apply it to Muse Glimmer 30B, the model Meta released, and evaluate it there instead.

Figure 2. How Muse Glimmer 30B connects to DFlash-2 (diagram labels are in Japanese; it illustrates the Path Selector / Local Convolution placement already described above, so it isn't required reading).

The reason for picking Muse Glimmer 30B is simple: it's recently become our own day-to-day model of choice, replacing Gemma-4-12b-qat-Assistant.

Running It in llama.cpp

Where does llama.cpp stand as of August 2026?

As of August 23, 2026, DFlash-2 support hasn't landed on llama.cpp's master branch yet (it did land shortly after, on August 28, 2026 — officially supported from version 0.3.0-dev, build 10658, commit b10f9ca onward; to stay consistent with the figures measured for this piece, we've kept the write-up based on the pre-release build used at test time). That said, Z-Lab has already submitted a pull request, and building from it gets you an inference engine with DFlash-2 support.

spec: add DFlash2 support (local convolution + candidate selector) — #27342
https://github.com/ggml-org/llama.cpp/pull/27342

Building llama.cpp from this PR gives you an inference engine that supports DFlash-2:

# Build llama.cpp with PR #27342 applied
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342
# Build command for NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j

From there, running it as follows gets you a working inference engine:

../dflash/llama.cpp/build/bin/llama-server \
--model ./models/Muse-Glimmer-30B-UD-Q2_K_XL.gguf \
-t 12 -np 1 --prio 2 --temp 1.0 --top-p 0.95 --top-k 64 \
--port 8001 --host 0.0.0.0 --fit off --no-warmup --no-cache-prompt \
-fa on --cache-ram 0 -c 131072 --reasoning on --log-verbosity 4 \
--model-draft ./models/Muse-Glimmer-30B-DFlash2-Q4_K_M.gguf \
--spec-type draft-dflash --spec-draft-n-max 4

A quick rundown of the arguments:

Verification: Using DFlash-2 on Muse Glimmer

Test Environment

We ran the following simplified verification setup.

The setup connects through an access server over an SSH tunnel to reach the target instance (for more detail on this setup, see our earlier piece, Using GPUSOROBAN). llama.cpp's service port is relayed between localhost and the remote instance through port 8001 on both ends.

 Home                                 Highreso: GPUSOROBAN
+----------------+                  +------------------+      +--------------------+
| [>_]  8001/tcp |==(SSH tunnel)==> | [>_]  8001/tcp   |---->| [llama.cpp]        |
|    Client      |                  |  Access Server   |      |  8001/tcp          |
+----------------+                  +------------------+      |  Target Instance   |
                                                                +--------------------+

Figure 3. Connection setup via GPUSOROBAN.

Instance Used

We used an NVIDIA RTX A4000 GPU for this test. Full instance specs below.

Table 1. GPUSOROBAN instance specs.

Model Files Used

Even at 4-bit quantization, Muse Glimmer's weights alone exceed 16GiB. Accounting for KV-cache overhead, we used the 2-bit quantized build optimized with Unsloth Dynamic 2.0.

We downloaded the following files from Unsloth's Muse Glimmer repository:

Table 2. Files applied to llama.cpp.

Test Cases

• Goal: measure and compare throughput across three configurations — main model only, DFlash, and DFlash-2. Only speed is measured; output content is not evaluated. We record wall-clock time, token count, and token throughput at generation — these figures come directly from llama.cpp's own log output.
• Method: using the llama.cpp web frontend, we ran two patterns, clearing the session between each:
  • Single-turn, Japanese-heavy pattern — "Tell me about the geological characteristics of Kyushu."
  • Multi-turn, code-heavy pattern — (1) Write JavaScript code for Breakout (as an HTML file). (2) Make it look cooler. (3) Slow down the ball's movement a bit. (4) Check for bugs and optimize it.

Memory Usage Comparison

Here's how memory usage came out. (Main-model-only is excluded from this comparison.)

Table 3. Memory usage comparison (MiB).

Aside from a modest increase in the MTP model's weight data and compute buffer, memory usage is essentially unchanged. The added layers (the selector and the convolution layers) account for the increase, but combined they add up to under 100MiB.

Token Throughput Comparison (Main / DFlash / DFlash-2)

Single-Turn, Japanese-Heavy Results

Table 4. Read and generation throughput for each configuration.

Here's how generation throughput moved over the course of the response:

Figure 4. Generation throughput over time by configuration.

Throughput started out relatively fast during the initial English portion, then dropped noticeably once the Japanese section kicked in. Even so, DFlash-2 stayed ahead of DFlash throughout.

For this single-turn, Japanese-heavy case at least, neither draft model contributed much.

Multi-Turn, Code-Heavy Results

This is where things get interesting — here's the multi-turn, code-heavy comparison.

Table 5. Read and generation throughput for each configuration.

Both DFlash and DFlash-2 posted strong throughput here. The upward trend across turns is characteristic of MTP in general. In a coding-agent-style workload, DFlash-2 delivers a substantial throughput gain.

Figure 5. Generation throughput over time by configuration.

This confirms something our earlier DFlash testing didn't cover: in the "multi-turn, code-heavy" scenario, DFlash is genuinely useful for models like Muse Glimmer that, unlike Gemma-4, don't ship with a purpose-built Assistant-style MTP model of their own.

On top of that, DFlash-2's new additions push acceptance even higher, which translates directly into faster output.

"Multi-turn, code-heavy" is also a reasonably good proxy for typical coding-agent usage, suggesting this setup is especially effective in workloads that generate large volumes of tokens.

Token Acceptance Rate Comparison

Let's look at the predicted-token acceptance rate. For each case, we read Acceptance and mean length off the llama.cpp logs.

Mean length is calculated as follows:

mean length = 1 + <number of accepted tokens> ÷ <number of verification rounds>

The "1" here represents the bonus token (the one token the target model itself is guaranteed to produce). So if acceptance were 0%, mean length would still be 1, which is its floor value.

One thing worth flagging: the value you set with --spec-draft-n-max does not include the bonus token. So with --spec-draft-n-max 4, the maximum possible mean length — bonus token included — is actually 5.0.

Single-Turn, Japanese-Heavy

Acceptance is fairly low here. DFlash in particular dips under 2, which would normally be read as "not worth using."

Table 6. Acceptance and mean-length values for each configuration.

Multi-Turn, Code-Heavy

Acceptance is markedly higher here. DFlash-2 in particular reaches a mean length of 4.12 by turn 3, a notably strong result.

Table 7. Acceptance and mean-length values for each configuration.

Higher acceptance means more predicted tokens get used, which drives throughput up — the two appear to move together.

In other words: the higher the acceptance rate, the more useful the predicted tokens become, and the faster generation gets.

Getting More Out of DFlash

Finding the Right Draft Length

We reran the multi-turn, code-heavy pattern with the predicted-token length set to 6. Results below.

Table 8. Acceptance / mean-length comparison by draft length.

Acceptance drops at 6 tokens, and mean length does increase, but only by less than one token — not much of a net gain.

Let's look at how this affects throughput over time.

Figure 6. Generation throughput over time by draft length.

Overall, the 6-token setting came out lower in throughput. There were a few stretches in turn 2 where the existing (4-token) setting briefly fell behind, but turns 1 and 3 clearly favored the 4-token setting.

This suggests mean length roughly tracks the probability that a given draft length "lands." So rather than pushing for a longer draft length, it looks better to pick the smallest draft-n-max setting where draft-n-max − (mean length − 1) stays greater than 0 — in other words, the smallest setting that still clears the average number of accepted draft tokens.

DFlash's internal model tries to generate predicted tokens all the way out to whatever maximum length you set. If the realistic prediction horizon is around 4 tokens, anything predicted beyond that is wasted work.

That said, mean length alone isn't enough to settle on the right draft length. In practice, actual throughput should be the primary signal, with mean length used as a rough guide for where to start.

Checking the Impact of KV-Cache Quantization

Next: how much does quantizing (or not quantizing) the KV cache — a meaningful factor in the speed/memory tradeoff — affect DFlash-2's performance?

4-bit quantization usually calls for caution even in ordinary inference, while 8-bit is generally considered to have "little practical impact." That said, it's reasonable to expect some impact when precision in token prediction specifically matters.

Here's what we found, looking at acceptance and mean length:

Table 9. Acceptance / mean-length comparison by KV-cache precision.

The difference isn't large. Let's turn to raw output speed next.

Figure 7. Generation throughput over time by KV-cache precision.

A slight offset in where each turn starts — especially pronounced around turn 3 — makes the data around the 5,000-token mark a bit noisy. Early on the impact looks minor, but it appears to compound somewhat as the run goes on.

To check this more carefully, let's look at each turn individually.

Figure 8. Throughput distribution broken down by turn.

There's a roughly 3-4 tps gap across the board.

Acceptance and mean length were both close between the two settings, but token and character counts can't be subdivided below whole numbers, which is likely what's driving that gap.

The takeaway: when using token prediction, it's best to keep the KV cache at f16 (the default) wherever possible — though since the gap doesn't appear to widen much turn over turn, it's worth weighing against the extra cache memory it costs.

Does It Hold Up for Everyday Use?

We've made the case that DFlash-2 is genuinely useful for coding-agent workloads — but is it actually "not worth enabling" for more ordinary, Deep-Research-style everyday use? We tried it out.

Here's a sample of the questions we actually asked over the course of one day, along with the relationship between token count and throughput.

We use Muse Glimmer to speed up research when writing articles — it digs into things diligently. Some representative examples:

Table 10. A sample of everyday questions asked in one day.

These are fairly ordinary day-to-day questions. Some are research for an article; others are just things that came up in day-to-day life around that time.

Here's the relationship between token count and output speed for that session:

Figure 9. Distribution of token count vs. output speed (everyday use).

Short exchanges are noticeably fast, and speed tapers off as length increases — but it never dropped below the no-MTP baseline (roughly 12 tps, the red dashed line).

Since turn count isn't factored in here, the longer exchanges are more likely to be single-turn, so it's hard to draw a firm conclusion. Still, for everyday use it's clearly a net positive, and as of August 2026 DFlash-2 has become our main day-to-day setup (we also tried Qwen3.8-27B, but it tends to spend too long thinking, which held us back from using it as much day-to-day — even though its raw reasoning quality is excellent).

Conclusion

This time we covered DFlash-2, from the little-known startup inco.ai.

Through testing, we came away with the following:

• DFlash-2 adds a consistency-checking mechanism on top of the existing DFlash design, aimed at more reliable token prediction.
• Under normal conditions (single-turn, Japanese-heavy), the impact is modest.
• In multi-turn, code-heavy settings — i.e., coding-agent use — it becomes a genuinely powerful tool. DFlash-2 shows a clear improvement over the original DFlash here.
• A couple of things to keep in mind when using token prediction, which further improve results: measure the realistic draft length rather than assuming one, and don't quantize the KV cache.

Causal models still have real advantages — the act of linking tokens sequentially reinforces the direction of generation and helps keep output coherent and high-quality — so we expect them to remain the mainstream approach for the foreseeable future.

That makes techniques like this one, which speed up output while preserving that property, worth continuing to watch.

As AI practitioners, we think the job going forward is to keep tracking techniques like this with a full picture in view, not just the surface-level claims.