DFlash：122B模型推理速度飙升至1000+ tokens/s

Speculation is all you need!

A new technique took a 122B model from 250 to 1000+ tokens/sec with zero quality loss.

It builds on speculative decoding, which is already an effective technique used in production to get around the single-token bottleneck of regular LLM inference.

In speculative decoding, a small draft model generates the next several tokens, and the large target model verifies all of them at once in one forward pass.

This produces zero quality loss because the kept tokens are exactly what the target would have produced on its own.

But the bottleneck here is that a normal drafter still generates one token at a time (autoregressive generation), which limits real speedups to around 2-3x.

DFlash swaps the autoregressive drafter with a block diffusion model that generates all tokens in one parallel pass.

To improve the acceptance rate, the drafter does not work from the token sequence alone.

As the target model reads the context, each of its layers produces hidden states, and DFlash pulls those representations from several layers and feeds them into the draft model.

So the drafter starts from the target's own internal view of the context instead of guessing from raw tokens.

This means its proposed tokens line up with the target more often, which pushes the acceptance length up.

Modal recently released a new set of DFlash draft models for several Qwen models on HF that take this even further.

They trained drafters on the target's own outputs and on real production traffic, so that they propose tokens the target actually emits.

On top of the existing DFlash baseline, these drafters added another 5 to 20%. And it also pushed acceptance length from a baseline of 3 to 9+.

They benchmarked Qwen 3.5 27B on one B200 at concurrency 1.

An acceptance length of: - 2 gave 1.86x speedup. - 4 gave 3.57x speedup. - 8 gave 5.62x speedup.

Running Qwen 3.5 122B-A10B with one of these tuned drafters reached over 1000 tokens/sec, and the side-by-side speedup is evident in the video below.

If you are running speculative decoding in production, the quality of the drafter will always decide how high the acceptance length can reach.

So training the drafter on the workload it serves is the fastest way to get there.

I wrote a full breakdown of how speculative decoding works, covering drafting, verification, rejection sampling, and several other optimization techniques.

Read it below.