Kimi K3 引入注意力残差，2.8T 参数模型 API 已上线

Kimi did it again!

In Kimi K3, they built a new way to handle residual connections in Transformers that has not been changed since ResNet in 2015.

In a standard Transformer, every layer adds its output back to the input with a fixed weight of 1.

So every layer gets equal importance, regardless of what the token actually needs.

Across 40, 60, 80+ layers, the hidden state becomes an equal-weighted sum of everything that came before.

Deeper layers have to produce increasingly large outputs just to have any influence. This destabilizes training as models scale.

Attention Residuals fix this by replacing that fixed addition with softmax attention across depth.

Each layer now learns how much to draw from every previous layer, input-dependent, so different tokens can retrieve different layer representations based on what is actually useful.

The idea mirrors what the original Transformer did for sequences.

RNNs compressed all prior token information into a single state across time. Transformers replaced that with attention. Attention Residuals apply the same logic to depth.

When Kimi first validated this on their 48B Kimi Linear model in March, Block AttnRes matched the performance of a baseline trained with 1.25x more compute, at under 2% inference latency overhead.

K3 is the first model to deploy this at frontier scale, with 2.8T parameters, paired with Kimi Delta Attention for up to 6.3x faster decoding at million-token contexts.

The API is live, and weights will be released on July 27.

I wrote a full breakdown of the Attention Residuals paper when it was released in March.

Read it below.