前沿语言模型复制长重复字符串失败，2D-RoPE 可改善

Frontier language models can solve advanced reasoning tasks yet still fail to copy long, repetitive strings exactly.

The paper finds that copying becomes less reliable as inputs grow longer or repeat the same patterns.

The failure is partly architectural. With standard 1D RoPE, producing output token y_k requires retrieving x_k from a relative offset that changes with input length.

The choice of positional encoding can determine whether a model copies precisely or follows nearby patterns.

2D-RoPE changes the layout.

It treats the source and output as separate rows, so corresponding tokens line up in the same column.

In synthetic tests, one-layer models trained on lengths 1–100 copied perfectly at lengths up to 1,000 times longer.

The advantage also appeared in pretrained models up to 1.4B parameters, with comparable common-sense performance at the tested scales.

The current design still leans on line breaks, so it is not a universal fix.

– arxiv. org/abs/2607.16072

Title: "Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D"