Qwengram-0.8B：将PLE n-gram记忆迁移至小模型

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model.

I trained the 0.8B setup with limited resources, mostly using free Kaggle notebook GPUs.

The setup keeps both the Qwen3.5-0.8B backbone and the roughly 51B-parameter PLE memory frozen. A small R=1 reader is trained at decoder layers 3 and 9, with a token-dependent linear gate controlling the later injection.

There is no backbone fine-tuning.

The current balanced checkpoint uses a reader trained for 15M tokens and a separately calibrated dynamic gate. On the frozen full-validation set:

Qwen3.5-0.8B stock

Perplexity reduction: 5.05%

This is a language-model validation result, not a claim of 5% higher benchmark accuracy.

A few findings shaped the final design:

• The real pretrained PLE outperformed both random-memory and permuted-memory controls.
• Reader loss kept improving well beyond 5M training tokens. The 20M reader improved aggregate LM loss further, but regressed on math, so 15M remains the balanced checkpoint.
• Strong fixed late-layer memory injection hurt LAMBADA. Dynamic token-level arbitration recovered much of that tradeoff.
• The gate is genuinely dynamic: its memory strength varies substantially across tokens rather than behaving like a learned constant.
• With the exact same memory budget, learned token placement beat shuffled placement. Routing memory toward high-uncertainty positions recovered part of the advantage, but still did not match the learned gate.
• A warm-started R=4 reader produced a small aggregate LM-loss improvement, but introduced code and math regressions. I therefore kept R=1 as the balanced architecture.

I also implemented the inference path in llama.cpp.

The public artifacts are:

Model / GGUFs

Training, controls, and evaluation

Modified llama.cpp runtime

The GGUF contains the Qwen3.5 backbone plus the trained reader and arbitration tensors. The large PLE remains an external quantized sidecar, rather than being packed into the model GGUF.

I also tested quantization retention on a separate fixed WikiText-2 GGUF runtime test:

• Q8_0 retains 99.1% of the BF16 reader NLL gain

This is a separate runtime measurement, not the frozen Kaggle validation benchmark above.

I’d welcome attempts to reproduce or improve the reader, PLE caching, routing, or runtime.

Next I’d like to try larger Qwen backbones, particularly the 35B-A3B MoE. Experiments at that scale require substantially more compute than free Kaggle notebooks can provide, but the 0.8B study gives a much clearer recipe for reader scaling and dynamic memory arbitration.

Disclosure: I’m the author of Qwengram and the linked repositories. English is not my first language, so I used AI to help proofread grammar and improve phrasing in this post. The experiment itself was also developed with the assistance of coding agents, primarily ChatGPT Sol, for implementation, debugging, experiment orchestration, and analysis support. I designed the experiments, made the research decisions, reviewed the results, and am responsible for the final conclusions.