跳到主内容
@wquguru
精选88Ahead of AI(RSS)技巧与观点

深入理解现代大模型中的位置编码机制

Understanding Positional Embeddings in Modern LLMs

原文
发到 X
推荐理由

LLM 从业者必读的技术深潜,用代码和数学讲透 RoPE 原理,直接指导架构选型与调试。

This article explains how LLMs encode and process token position information by covering classic and contemporary methods like RoPE. It’s essentially a comprehensive journey where we will work through the mechanisms with visual examples, a little math, and a small PyTorch implementation. And, of course, we will then also look at how modern architectures adapt or omit explicit positional encodings.

本文通过介绍 RoPE 等经典和现代方法,解释了大语言模型(LLM)如何编码和处理 token 的位置信息。这本质上是一段全面的探索之旅,我们将结合可视化示例、少量数学推导以及一个小型 PyTorch 实现来逐步剖析这些机制。当然,我们还将探讨现代架构如何适应或省略显式的位置编码。

Below, you can find a preview of which positional embedding choices are used in 2026 open-weight models. We will discuss these choices later after covering the underlying mechanisms.

下文展示了 2026 年开源权重模型中采用的位置嵌入选择预览。在深入探讨底层机制后,我们将稍后讨论这些选择。

Figure 1: A preview of the gallery’s 2026 architectures, grouped by positional encoding. Architecture figures taken from my LLM gallery.

图 1:按位置编码分组的画廊 2026 年架构预览。架构图取自我的 LLM 画廊。

But why spending so much time writing/reading about this rather minor building block? LLMs keep evolving, and as I was preparing a new article on the latest LLM architecture trends this year, I realized that I never covered the positional information methods used in LLMs in any article so far. So, instead of letting the other article get out of hand, I thought it might make sense to publish this as a standalone building block for the upcoming LLM architecture and trends analysis.

但为什么要在这样一个相对次要的构建模块上花费大量时间进行写作/阅读?LLM 不断演进,今年我在准备一篇关于最新 LLM 架构趋势的新文章时,意识到迄今为止我从未在任何文章中涵盖过 LLM 中使用的位置信息方法。因此,与其让另一篇文章失控,我认为将其作为即将发布的 LLM 架构与趋势分析的独立构建模块发布是合理的。

Also, over the years, many readers have asked me for a tutorial on relative positional embeddings (aka RoPE) to follow up on the learned absolute embeddings used in GPT. It’s been on my TODO list for a long time, so I finally revisited my old draft and put this walkthrough together, which will cover

此外,多年来许多读者要求我提供一篇关于相对位置嵌入(即 RoPE)的教程,以便跟进 GPT 中使用的学习型绝对嵌入。这一直是我待办事项清单上的长期任务,因此我终于重新审视了旧草稿并整理出这篇指南,它将涵盖

  • different positional methods in LLMs;
  • (finally) how RoPE works;
  • what methods are used by recent open LLMs.
  • LLM 中的不同位置方法;
  • (终于)RoPE 的工作原理;
  • 近期开源 LLM 使用的方法。

Note that this article assumes some familiarity with basic LLM architectures and works well as a follow-up to Build a Large Language Model (From Scratch). If the concepts and computation of the attention internals, particularly query, key, and value vectors, are a little fuzzy, I recommend checking my self-attention walkthrough as a refresher:

请注意,本文假设读者对基本 LLM 架构有一定了解,并且非常适合作为《从零构建大型语言模型》(Build a Large Language Model (From Scratch))的后续读物。如果你对注意力机制的内部概念和计算,特别是查询(query)、键(key)和值(value)向量有些模糊,我建议先回顾一下我的自注意力机制指南:

The article is also very technical and likely too detailed for some. However, I really wanted to have this foundation out there for future articles to reference. And no worries, the next one will return to a more comfortable style.

本文技术含量很高,对某些读者来说可能过于详细。然而,我非常希望建立这一基础,以便未来的文章可以引用。别担心,下一篇文章将回归更轻松的风格。

PS: I started experimenting with Substack’s new “perks” system. As a thank-you to supporters, I’ve added image exports and metadata from the architecture gallery for you to use in your own analyses.

PS:我开始尝试 Substack 的新“福利”系统。作为对支持者的感谢,我为你们添加了来自架构画廊的图片导出文件和相关元数据,供你在自己的分析中使用。

PPS: If you are already familiar with how position embeddings work, feel free to fast-forward to Section 6, Positional choices in modern architectures.

PPS:如果您已经熟悉位置嵌入的工作原理,可以跳到第 6 节“现代架构中的位置选择”。

Note that this article may be best read in a web browser than an email client. I recommend checking out the web view here, which also has a navigation bar on the left side.

请注意,本文在网页浏览器中阅读效果可能优于电子邮件客户端。建议在此处查看网页视图,左侧还设有导航栏。

1. Where position information enters an LLM

1. 位置信息如何进入大语言模型

Let’s start with the overall architecture. A tokenizer encodes the input text into token IDs. Next, the input token embedding layer of an LLM maps each of these IDs to a vector of numbers. We usually refer to these entries in that vector as “features” (similar to classic ML). The transformer blocks inside the LLM then modify these vectors using attention and feedforward layers. Lastly, the model’s output layer produces logit scores (converted into probabilities) for the next token.

让我们从整体架构开始。分词器将输入文本编码为 token ID。接下来,大语言模型的输入 token 嵌入层将这些 ID 中的每一个映射为一个数值向量。我们通常将该向量中的这些条目称为“特征”(类似于经典机器学习)。随后,大语言模型内部的 Transformer 模块通过注意力机制和前馈层修改这些向量。最后,模型的输出层为下一个 token 生成 logits 分数(转换为概率)。

The following shows this path in GPT-2 (a relatively classic LLM) and MiniMax-M3 (a more recent LLM released this year; a placeholder for most other recent LLMs). As we can see, the architectures are not too different from each other, but before we get to the positional information parts, let’s focus on the orange blocks for now (the token embedding layers) and the transformer blocks above them; the other architecture differences aren’t needed for this explanation.

下图展示了 GPT-2(一个相对经典的大语言模型)和 MiniMax-M3(今年发布的一个较新的大语言模型;可作为大多数其他近期大语言模型的占位符)中的这一路径。我们可以看到,两者的架构差异不大,但在深入探讨位置信息部分之前,让我们先关注橙色块(token 嵌入层)及其上方的 Transformer 模块;其他架构差异在此解释中并非必需。

Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3’s first three out of 60 feedforward layers are dense not MoEs.)

图 2:GPT-2 和 MiniMax-M3 中的 token 嵌入层(以橙色高亮显示)。(旁注:M3 的 60 个前馈层中的前三个是稠密层,而非 MoE。)

A token embedding layer, shown in the figure above, maps each token ID to a vector. Position information is then added afterward. Concretely, and as we will discuss in more detail later, GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block. On the other hand, MiniMax-M3 adds the positional information in each transformer block using a method called RoPE inside attention.

如上图中所示,token 嵌入层将每个 token ID 映射为一个向量。随后会添加位置信息。具体来说,正如我们将在后文详细讨论的那样,GPT-2 在每个 token 嵌入进入第一个 Transformer 模块之前,会向每个 token 嵌入添加一个位置向量。另一方面,MiniMax-M3 则在每个 Transformer 模块中使用一种名为 RoPE 的方法,在注意力机制内添加位置信息。

We will talk about these methods in more detail later, but first, let’s start with the big picture information explaining why explicit positional information is added or needed.

我们将在后文更详细地讨论这些方法,但首先,让我们从宏观层面了解为什么需要添加或显式使用位置信息。

1.1 Why add positional information?

1.1 为什么要添加位置信息?

Consider the following two input sentences as a simple example:

考虑以下两个输入句子作为简单示例:

  • “The red fox jumps over the gray fox.”
  • “The gray fox jumps over the red fox.”
  • “红狐狸跳过灰狐狸。”
  • “灰狐狸跳过红狐狸。”

Changing the word order changes which fox does the jumping, as illustrated in the following figure.

改变词序会改变哪只狐狸在跳跃,如下图所示。

Figure 3: Changing the word order changes which fox jumps.

图 3:改变词序会改变哪只狐狸在跳跃。

For the examples in this article, we will treat each word as one token for simplicity. (As you know, real tokenizers can split words into smaller subword tokens.) The first sentence then has eight token positions, from 0 through 7.

为了本文中的示例,为简化起见,我们将每个单词视为一个 token。(正如你所知,实际的 tokenizer 可以将单词拆分为更小的子词 token。)因此,第一句话有八个 token 位置,从 0 到 7。

Figure 4: Token IDs and positions in the toy sentence.

图 4:玩具句子中的 Token ID 和位置。

代码 · 11 行
# Assume the following small vocabulary.
# Real vocabularies typically have 50k to 200k entries
vocab = {
    "The": 0,
    "red": 1,
    "fox": 2,
    "jumps": 3,
    "gray": 4,
    "over": 5,
    "the": 6,
}

(If you are interested in how vocabularies are built, check out my tokenizer materials on GitHub.)

(如果你对词汇表是如何构建的感兴趣,可以查看我在 GitHub 上的 tokenizer 资料。)

Note that “fox”, which appears twice in the input sentence, gets the same token ID 2 in both cases. And since they both have the same token ID, irrespective of the position in the input sentence, they would get the same embedding vector from the embedding layer.

请注意,“fox”在输入句子中出现了两次,在这两种情况下都获得了相同的 token ID 2。而且,由于它们具有相同的 token ID,无论它们在输入句子中的位置如何,它们都会从嵌入层获得相同的嵌入向量。

Now, let’s consider a regular self-attention mechanism with no positional encoding (and no causal mask), where every input token can attend to every other token in the input. In this scenario, swapping any two words, like “red” and “gray”, would still result in the same embeddings and attention scores between each token pair.

现在,让我们考虑一个没有位置编码(也没有因果掩码)的常规自注意力机制,其中每个输入 token 都可以关注输入中的任何其他 token。在这种情况下,交换任意两个单词(如“red”和“gray”)仍会导致每对 token 之间的嵌入向量和注意力分数相同。

GPT-style models do use a causal (autoregressive) mask, which restricts each token to itself and earlier tokens. Interestingly, this causal mask gives the computation some implicit order information. In our example, “jumps” can attend to “red” in the first sentence and “gray” in the second. So, a causal LLM can theoretically already distinguish these sentences even without explicit positional embeddings. (We will return to this idea later when discussing the “NoPE” concept.)

GPT 风格的模型确实使用了因果(自回归)掩码,它将每个 token 限制为其自身及之前的 token。有趣的是,这种因果掩码为计算赋予了一些隐式的顺序信息。在我们的例子中,“jumps”在第一句话中可以关注“red”,在第二句话中可以关注“gray”。因此,理论上因果 LLM 即使没有显式的位置嵌入,也能区分这些句子。(我们将在讨论“NoPE”概念时再次回到这个想法。)

Nonetheless, GPT-2 and most newer LLMs include explicit positional information because it gives attention a direct signal about positions. That’s because even though the causal mask determines which tokens are accessible and provides implicit order information, it doesn’t distinguish a separation of one position from a separation of ten, for example. So, explicit positional methods are still useful (or even essential), since they supply that information.

尽管如此,GPT-2 和大多数较新的 LLM 都包含了显式的位置信息,因为它为注意力机制提供了关于位置的直接信号。这是因为尽管因果掩码确定了哪些 token 是可访问的,并提供了隐式的顺序信息,但它无法区分相隔一个位置和相隔十个位置的情况。因此,显式位置方法仍然有用(甚至是必要的),因为它们提供了这些信息。

If this all sounded confusing, don’t worry, it was more meant as an overview, and we will step through these concepts a bit more slowly in the remaining article.

如果这一切听起来令人困惑,别担心,这更多是作为一个概述,我们将在文章的剩余部分更缓慢地逐步讲解这些概念。

2. Adding a position vector to each token embedding

2. 将位置向量添加到每个 token 嵌入中

We briefly talked about implicit positional information through causal attention masks (and we will revisit this again in more detail later). However, we now switch gears and talk about the explicit ways to add positional information.

我们简要讨论了通过因果注意力掩码实现的隐式位置信息(我们稍后会更详细地重新审视这一点)。然而,我们现在转换话题,讨论添加位置信息的显式方法。

The perhaps simplest and most direct (but not “cheapest”) approach is to give each sequence position its own vector. We add that vector to the token embedding, so the resulting, modified embedding vector is then passed to the first transformer block. This exact procedure is discussed in the following subsections.

也许最简单且最直接的(但并非“最便宜”)方法是为每个序列位置分配一个独立的向量。我们将该向量加到词元嵌入上,因此得到的、经过修改的嵌入向量随后被传递到第一个 Transformer 块。这一确切过程将在以下小节中讨论。

2.1 Learned absolute position embeddings in GPT-2

2.1 GPT-2 中的学习型绝对位置嵌入

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件