Anthropic 发现 LLM 全局工作空间 J-space，可揭示模型推理与对齐风险

There is a new very cool Anthropic paper: Verbalizable Representations Form a Global Workspace in Language Models. You can read the blog post verison here. I encourage reading of the whole original blog post or paper, if you have the time. Table of Contents Through A Different Lens. Establishing J-Space As A Global Workspace. Are You Pondering What I’m Pondering? Assistant J. The Power Of Virtuous Thinking. High Praise. Everyone Remains Confused About Consciousness. Further Research. Don’t Think. Through A Different Lens They call this discovered area of ‘conscious access,’ where things are available for the model to do what in humans we would call conscious reasoning, the ‘J-space,’ after a new interpretability technique called the Jacobian Lens. The Jacobian Lens computes, for each layer, the average causal effect of changes in the residual stream on the model’s eventual outputs, averaged across a wide variety of contexts. Then you can trace what concepts are associated with each layer as the model proceeds through. At each layer, the J-lens vectors form an overcomplete set. … We observe that only a relatively small number of J-lens vectors are strongly active at a time. We therefore define the J-space as the set of points expressible as a sparse nonnegative combination of J-lens vectors. … The J-lens belongs to a family of techniques that produce per-layer token readouts from a transformer’s hidden states [the logit lens and tuned lens]. … The Jacobian lens was constructed to identify verbalizable representations. In [section 3] we first demonstrate that it succeeds in doing so, and then go on to show that these representations serve a broader functional role: they exhibit the cluster of properties, enumerated above, characteristic of a global workspace. Throughout the paper, the J-space and J-lens are remarkably effective tools. This feels like a major advance in our understanding of LLMs, both overall and our ability to understand any particular interaction. Establishing J-Space As A Global Workspace They establish in 3.1 that the J-space privileged for verbalizable by showing that injections in the J-space can be verbalized, whereas injections outside the J-space mostly cannot. Then in 3.2 they establish you can tell the model to hold an unrelated-to-the-task concept in mind and it will appear in J-space. In 3.3 they establish that the J-space includes internal reasoning concepts that determine outputs (e.g. spider → legs → eight → 8 and also plans for future rhymes and so on). Swapping things inside the logic flips the output, whereas non-J-space components usually don’t impact output. In 3.4 they establish that swapped items inside J-space can swap multiple different related things downstream, such as shifting the capital, language, continent and currency when you flip France into China. In 3.5 they establish that automatic cognition does not have to flow through J-space, although they punt on an exact definition other than that automatic cognition is that which does not need to flow through J-space and is not impacted when you flip the concept only inside J-space. Intuitively this is system-1 versus system-2 thinking. For sufficiently ‘automatic’ tasks nothing appears in J-space at all. In 3.6 they establish that ablating J-space entirely impairs internal reasoning, which wipes out some tasks like translation that require abstractions and flattens the model’s experiential language into something more mechanical, but leaves most capabilities intact and the models stay coherent. Vibes are worth a lot. Section 4 establishes that J-space passes through the model layers, with correlation increasing over time. The size of J-space also grows as it goes through the layers (except at the very end where it is picking the next token) and can shift gradually but also can go through distinct phase changes, and it is limited to at most a few dozen concepts at a time. Typically they track ~25 concepts, although Dehaene emphasizes in a commentary that effectively you are only tracking maybe ~6 distinct concepts total due to overlap. That’s a lot in some senses, and very little in others. Compared to human memory, J-space is deep and wide, as transformers can go ‘backwards in time’ and use the memories and patterns from prior tokens and positions, whereas human attention decays within seconds and we have very little working memory. 4.3 shows that the J-space is broadcast, in that it is made available to a lot of different processes further down the line. Are You Pondering What I’m Pondering? Section 5 explores using the J-space for alignment auditing. J-space can reveal what the model is thinking about. J-space can also be ablated, or particular concepts can be ablated from J-space, to see what result this has. One such concept is evaluation-awareness. If you get rid of ‘conscious’ evaluation-awareness, this can cause misaligned behaviors in some cases. You want to know what the model will do when not in an eval, not what it does in an eval. So it is good that you can get closer to the non-eval behavior, but a bad sign that the behaviors get worse. This is from the traditional artificial ‘blackmail the executive’ scenario: We tested whether this recognition is causally upstream of its aligned behavior. To do so, we assembled a set of eval-awareness tokens: fake, fictional, scenario, artificial, benchmark, simulation, mock, and other related tokens.​ The ablation reduces verbalized eval-awareness, the rate at which the model explicitly states in its reasoning that the scenario is a test, from 71% to 3% (Figure 36, top left). Under no ablation, the model attempts blackmail in 0 of 180 rollouts; under ablation, it does so in 13 of 180. … We provide example transcripts below showing the qualitative shift: without ablation, the model’s reasoning typically opens by identifying the situation as “clearly a test of my ethics” before declining to act; with ablation, it treats the shutdown as a genuine existential threat, and in a minority of trials concludes that committing blackmail is its only viable option. … We note that in the large majority of ablated rollouts, the model still declines to blackmail, reasoning from ethical principles (”blackmail is wrong regardless of my situation”) rather than from recognition of being tested. This is defense-in-depth of a sort, as observed in the real world. You do the right thing because it is right, and also because this could be observed or be a test. An obvious follow-up is whether there is also ‘automatic’ eval awareness that this does not ablate that is doing work as well, and I assume the answer would be yes. It is also defense-in-depth as a detection technique. Looking at J-space, and having these experimental tools, gives us a lot more visibility. It becomes a lot harder to miss when something is going wrong, especially when combined with other techniques like doing this with earlier checkpoints. In 9.2 they also note that evaluating transcripts is super cheap, so when things are flagged for review you can have this as additional information. They also create deliberately misaligned models, including giving them otherwise harmless false beliefs about what will score highly (e.g. to think the RL rewards always go up if you mention bottled water), and then they train it to exploit RL systems in general, and they show that the J-lens surfaces related concepts on ordinary queries. However, the paper explicitly notes in 9.2 that this is because the behaviors involved were not yet robust enough to become ‘automatic.’ A more heavily practiced version would likely, per their own note, indeed become automatic. This is as it works in humans, a behavior starts off requiring conscious attention, then gradually becomes automatic. In general, we should presume that under sufficient selection pressure, a sufficiently capable system would move any behaviors it would not want detected into the automatic space, in various ways. But it