微软新论文：NextLat让Transformer学习紧凑世界模型

New Microsoft paper argues that transformers generalize better when they learn compact internal states, not just next tokens.

The problem is that normal transformers can look back at every earlier token, so they do not have to squeeze the past into a clean summary. token prediction alone can reward shortcuts that do not become coherent world models.

That can work beautifully on familiar data and still fail when the model has to plan, detour, reason, or carry a hidden structure forward.

NextLat fixes this by adding a training task where the model must predict its next hidden state, not just the next word.

A hidden state is the model’s private summary of what it has seen, so predicting the next one pushes the model to learn how situations change over time.

The authors tested this on map-like world modeling, math reasoning, graph planning, story prediction, and regular language modeling.

The main result is that NextLat often learned more compact and useful internal states, solved planning tasks better, and sped up generation by up to 3.3x.

Overall, it gives transformers some of the useful memory behavior of recurrent models without changing the transformer architecture or slowing normal inference.

Link – arxiv. org/abs/2511.05963

Title: "Next-Latent Prediction Transformers Learn Compact World Models"