Anthropic Claude文本水印技术原理深度解析

I recently posted a Substack note about Claude’s new watermarking process and implementation. Since it’s such a popular topic and sparked such a lively discussion, I thought it might be interesting to go into a bit more detail when explaining how it works.

Instead of the usual text article, I recorded a little lecture on the topic (to change it up a bit from my usual articles). So, below is the video along with a transcript.

Originally, I planned to make 10 slides and record a short 10-min video. However, while putting it together, I added some crucial details here and there, resulting in >50 slides and a 48 min recording.

I hope that this now explains it well, though! Happy watching!

• I also have a YouTube version if you prefer using the YouTube player.
• And here is a link to the slides

Subscribe now

Video Transcript

Note: The transcript below is slightly edited and cleaned up for readability but preserves the overall order and flow of the video lecture above.

0:00

How Claude’s Text Watermarking Works

Slide 2 of 52, time stamp 0:00

Hi everyone. So, a few days ago, Anthropic announced that they will watermark the text outputs of their Claude models. I then did a social media post briefly explaining how that works. And yeah, this was quite the popular post. So not the watermarking itself was popular, but I guess the explanation or the mechanism behind it. Then, it might be worthwhile expanding this a bit to explain it in more detail, because this post only had one figure, and there were a lot of questions and discussions.

So, I thought, well, let’s make a few more figures. I actually originally planned to do like 10 slides and walk you through it. It ended up being 50 slides, but I hope this really explains how this watermarking technique works well, how watermarking itself can fail or be removed, and so forth. So I think it might be an interesting topic because a lot of people use LLMs these days and also consume a lot of text on the Internet that might be generated by LLMs.

And now there’s going to be this watermarking, and there’s this, I guess, fear of watermarking making text worse, or what’s actually the benefit of this watermarking? And so what does it mean? And I think if we understand a bit better what watermarking is, that goes a long way, and then we can make up our own minds about whether that’s a good thing or not, and so forth, like the pros and cons.

So, my goal here is really to explain how the underlying mechanism works and how they are going to implement this type of watermarking, text watermarking.

Slide 2 of 52, time stamp 1:41

It’s also a great example to illustrate why understanding things from scratch is actually quite useful. This watermarking technique is also a nice way to explain how conventional models or LLMs in general work under the hood. So yeah, you may know I like doing things from scratch. Like, I have my books: Build a Large Language Model From Scratch, Build a Reasoning Model From Scratch. I have some articles labeled from scratch.

So, for me, “ from scratch often includes coding. So this one will not be coding-related, but coding from scratch is actually a very, very useful technique because it really helps you understand how something is implemented. And then from that we can derive our understanding, figures, concepts, because if we don’t really implement things, if there’s no code, it’s really sometimes ambiguous. And of course, you know, as I realized, not everyone is coding from scratch anymore.

Like back in the day, coding something from scratch was all we had. I mean, there were only humans coding. Nowadays, coding can be done by LLMs. However, that doesn’t mean reading code is no longer useful, because it carries a lot of information. So in this case here with this watermarking, spending some time coding an LLM from scratch really makes you realize how this sampling inside is implemented. We still have some relevant code snippets.

And then that really, in turn, helps us understand, oh, the watermarking is applied at this position, and this has so-and-so consequences and so forth. So I think even though people may not be coding from scratch, at least not all the time anymore, it is still useful being able to, let’s say, build something from scratch for educational purposes to understand something deeply and then also for research purposes to manipulate this in a transparent way that is not hidden away in tons of layers of abstraction.

But that aside, I think it’s just a coincidental nice relationship here because, for this slide deck, I actually used a lot of figures from my from-scratch coding materials.

Slide 3 of 52, time stamp 3:58

So a few days ago (this is August 14), there was this article, How Claude’s Text Watermark Works, and there was this article here; it’s just like a screen recording, so it can have everything in the slides, but there’s plenty of detail. They updated it actually a couple of times, so originally when I read this, it was a way shorter. Still, it is very, I guess, conceptual; there’s like this overview, and there’s, I mean, there’s not a single figure in there.

And so it’s kind of still hard to understand what they’re trying to do. So they explain a lot about why they’re going to do it, but they don’t explain how. They’re linking to one paper somewhere there, which is very technical also. So I do think it makes sense maybe to take a step back and start at the beginning to kind of understand what they’re trying to implement here with this watermarking technique.

And so the motivation, by the way, of watermarking is for them to identify if someone posts some text that they can say, oh, this text was generated by our Claude Opus 4.8 model, for example, so that they have a way to tell, OK, this text is AI-generated because it carries this watermark. And this watermark is invisible to users, so only they can decode it and find out whether the text has their watermark.

Why can only they do it? We will get to that later in this (hopefully not too long a video), but one thing at a time.

5:38

How LLM Text Generation Works

Slide 4 of 52, time stamp 5:35

So I wanted to start with a brief prelude to explain how text generation works in LLMs, because based on that we can then more easily understand how the watermarking works and that this is actually not a huge, expensive thing on top of it. It’s really just like a minor, I guess, tweak inside the regular text generation process.

Slide 5 of 52, time stamp 6:01

So when we are using something like ChatGPT, for example, let’s say I ask the question, the capital of Germany is, and yeah, ChatGPT or other LLMs, so this is just like an example would, for example, answer “Berlin”. So here, in this case, it’s generating two tokens, like “Berlin” and the period. But for simplicity, let’s assume it’s generating one token. So the next token is the “Berlin” token. How is this token generated internally?

What is happening under the hood when we type something here like the capital of Germany is and receive a token like “Berlin” back? What is actually going on there behind the scenes?

Slide 6 of 52, time stamp 6:41

So in the next couple of slides, I want to briefly talk about what happens under the hood when this next token is generated.

Slide 7 of 52, time stamp 6:50

So assume again that our prompt is the capital of Germany is. And the first step here is to convert this into token IDs. So tokenizing it and converting it into token IDs is one of the main steps at the beginning. This is outside. It’s not inside the LLM; it’s outside of the LLM. So we are simply converting the text into token IDs. It’s just a format that embedding layers can work with.

Slide 8 of 52, time stamp 7:22

And then this passes through the LLM. And the LLM gives us a score distribution for the next token.

Slide 9 of 52, time stamp 7:31