5种LLM量化技术详解：从RTN到QAT
5 LLM quantization techniques, clearly explained:
5 LLM quantization techniques, clearly explained:
5种LLM量化技术，清晰解析：
(bookmark this)
（请收藏本文）
A 70B model in FP16 needs 140GB for weights alone. At 4-bit, that drops to 35GB, which fits on one card.
一个70B模型在FP16精度下，仅权重就需要140GB内存。若降至4位，则只需35GB，可单卡运行。
But naive rounding fails on large models. Roughly 0.1% of hidden dimensions carry values up to 20x larger than anything else in the tensor, and they wreck the quantization grid for everything else.
然而，简单舍入在大型模型上会失效。大约0.1%的隐藏维度携带的值比张量中其他任何值大20倍，它们会破坏其他所有值的量化网格。
Each of these 5 methods handles those outliers at a different point:
这5种方法各自在不同环节处理这些异常值：
1. RTN: ignores them. Rounds every weight to the nearest grid level with no calibration data. Cheapest option, weakest at low bit widths.
1. RTN：忽略异常值。将每个权重舍入到最近的网格级别，无需校准数据。成本最低，但在低比特宽度下表现最弱。
2. GPTQ: repairs after rounding. Quantizes a layer column by column and adjusts the remaining weights to absorb the error before moving on.
2. GPTQ：舍入后修复。逐列量化层，并在继续前调整剩余权重以吸收误差。
3. AWQ: protects before rounding. Finds the ~1% of weight channels that matter most and scales them up so they survive quantization. Everything still ends up in plain INT4.
3. AWQ：舍入前保护。找出约1%最重要的权重通道并放大它们，使其在量化中幸存。最终所有内容仍以标准INT4格式存储。
4. LLM. int8(): isolates at inference. Outlier dimensions run in FP16, the other 99.9% run in INT8, and the results are merged.
4. LLM.int8()：推理时隔离。异常值维度以FP16运行，其余99.9%以INT8运行，结果合并。
5. QAT: solves it during training. The model is fine-tuned with rounding baked into every forward pass, so it adapts to the damage before quantization is actually applied.
5. QAT：训练期间解决。模型在每次前向传播中融入舍入进行微调，使其在实际量化前适应损伤。
All five produce the same artifact, a model at a fraction of its trained precision. They differ only in where the outlier problem gets addressed.
这五种方法产生相同的结果：一个以远低于训练精度的模型。它们仅在处理异常值问题的环节上有所不同。
The visual below nicely summarises these techniques.
下图很好地总结了这些技术。
As further reading, the article below is a first-principles guide to LLM inference that walks through everything between your prompt and the streamed response, covering tokenization, embeddings, attention, the prefill and decode split, KV caching, and quantization.
作为进一步阅读，下面的文章是一份LLM推理的第一性原理指南，从你的提示到流式响应，涵盖分词、嵌入、注意力、预填充和解码分割、KV缓存以及量化。
It will give you a complete mental model of how inference actually works under the hood.
它将为你提供一个完整的推理实际运作的思维模型。
Read it below.
请在下方阅读。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力