8种LLM精度格式详解：从FP32到NF4

8 LLM precision formats, clearly explained:

(bookmark this)

A 12GB consumer GPU cannot hold a 7B model in FP32, which needs 28GB just for the weights.

Yet, running `ollama run mistral` on that card works anyway, because Ollama pulls a 4-bit build that fits in about 4GB.

Every format below FP32 does the same thing, i.e., it trades numeric detail for memory.

The visual below explains 8 such LLM precision formats:

Before we dive into them, some background:

A floating-point number splits its bits three ways.

One sign bit, then exponent bits that set how large a value can get, then mantissa bits that set how finely nearby values are told apart.

• Reducing exponent bits causes large values to overflow. - Reducing mantissa bits causes nearby values to round into each other.

Different formats focus on different sides.

1) FP32 has 8 exponent bits and 23 mantissa bits.

It costs 4 bytes per parameter and by default still holds the optimizer states even in low-precision runs, because gradient updates are small enough that 16-bit storage would round them away.

2-3) BF16 and FP16 are both 16 bits and split them in opposite directions.

BF16 keeps FP32's 8 exponent bits and drops to 7 mantissa bits, so casting down never overflows.

FP16 keeps 10 mantissa bits and only 5 exponent bits, so it resolves values eight times more finely but covers a far smaller range, roughly up to 65504.

Small gradients fall below that floor and flush to zero, which is why FP16 training scales the loss up before the backward pass and divides it back out before the update.

4) TF32 sits between them and never touches memory.

It computes with FP32's exponent and FP16's mantissa inside the tensor core, so matmuls get faster while storage stays exactly the same.

5) FP8 ships two layouts because one is not enough.

• E4M3 reaches 448 and carries weights and activations, - E5M2 reaches 57344 and carries gradients, which span more orders of magnitude.

6-7) INT8 and INT4 leave floating point behind.

Values map onto 256 or 16 evenly spaced levels with the scale factor stored separately.

But even spacing is a problem here. A single outlier stretches the range so far that every ordinary value falls inside the first step and rounds to zero.

This creates most problems on activations, and LLM[.]int8 and SmoothQuant were built to handle this.

8) NF4 spaces its 16 levels unevenly, following the distribution that pretrained weights actually have.

That allows QLoRA to hold the frozen base model in 4 bits while LoRA adapters train in BF16 on top.

Halving the bits halves the memory, but the bits have to come out of either the exponent or the mantissa.

If you take them from the exponent, large values cannot go past a certain range.

If you take them from the mantissa, values that were close become identical, and that error compounds layer after layer.

That said, weights are only one part of the memory costs that quantization addresses.

The KV cache is the other, and it behaves differently, since it grows with every generated token.

On long contexts, it outgrows the weights entirely.

I wrote a full walkthrough of that side of inference.

It covers the split between prefill, which is compute-bound and processes all input tokens in parallel, and decode, which is memory-bound and produces one token at a time.

It goes through what the KV cache is useful for, its costs, and how DeepSeek V4 redesigned attention around the cache itself to cut it to a tenth of the previous version at 1M context.

Read it below.