推理引擎加速开关导致MoE模型能力显著下降实测

I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights, or because they automatically turn on "hacky" flags to gain decode/prefill performance by default. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most popular "speed" flags:

• Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted.
• Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.

Both are faster because fewer experts come from RAM, but they change the output. I added them to the engine as opt in flags so I could measure the real cost with everything else fixed.

DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.

Rig: RTX 5090 + RTX 5060 Ti and 128 GB of DDR5. Settings tested:

• Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards.
• Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use.

100 chunks of wikitext, decode path, against a llama.cpp Q8 reference. Exact IQ3_XXS and Q2_0 are in there for scale.

Note: even exact Q8_0 is 0.031 away from the reference. That's mostly the reference's own noise, not damage: llama.cpp's Q8_0 rounds a few spots of this model more than Basalt does, which adds about 0.01-0.02 to every row equally, so the rows still compare fairly.

2. Does it really matter?

KLD by itself is just a number, so I also ran a task eval: thinking off, temp 1.0, top_p 0.95, top_k 20, two different sets of questions:

• Ladders: 300 chained steps, scored by step.
• Facts: 1200 questions with 5 popularity bands (rare to famous), 3 seeds.

* Small card: the 5090 alone capped to 16gb, the experiment was run this way because it was still much faster than the 5060 Ti.

On a big card the shipped flags barely matter (NVFP4: 272 -> 271 tok/s) and cost about 1.5 points, 4% of the facts the model knew. On a small card you get +56% speed but a much bigger hit. At the worst setting facts fall from 38% to 3%, and the model says things like "The Great Composer" with total confidence. Easy tests barely notice. The damage is mostly in chained reasoning and recalled facts.

3. Bonus: a loop in the model weights

Not every bad output is the engine's fault. One prompt made Qwen 3.8 Flash Next get stuck into an endless run of 0 in its reasoning:

The odds of another 0 are the same on every quant and both engines: 0.99 after 10 zeros, 95%+ past 100 or so. top_p 0.95 leaves 0 as the only choice, and temp 1.0 doesn't fix it. The API fixes it by cutting the run at ~500 zeros. Basalt patched the issue as well, after 256 identical reasoning tokens it closes the thinking block and keeps going.

TLDR: there's no free lunch.

• Engines can buy speed on MoE models by turning on speed "hacks".
• With values some engines ship, every pack drops to somewhere between IQ3_XXS and Q2 on KLD. A Q8 ends up next to Q2. This is as bad as it gets.
• In practice the model forgets things it knew and loses the thread on multi step problems. Small cards get the biggest speedup and pay the most for it.
• If your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded.