Jeff Dean谈百万倍算力跃升后的AI未来

There used to be a chat group internally called Data Centers on fire that would have like exciting [laughter] uh exciting events happening.

A distant supernova goes off. A cosmic ray hits a memory cell and a zero flips to a one. Does that really happen?

Oh, yeah.

So, my question is do you enjoy these Chuck Norris-style jokes about you?

It could be true. Um

[laughter]

One problem that you solved, tried to solve many times, but have never been able to crack. I cannot believe that this is happening, but I got to talk to a legendary engineer, the chief scientist of Google, Jeff Dean. He led Google Brain, one of the most legendary AI labs in history. He co-created MapReduce, which taught thousands of computers to work together as one. He co-built TensorFlow, the engine behind a huge chunk of AI research.

And for all this, they call him the Chuck Norris of computer science. Yes, I will tell him a joke about that, too. Now, when I see interviews with these executives, everyone is asking about China and taxes and all that. Look, I know nothing about that. I am just a student who loves to talk about research. So, my goal was to try to go a bit deeper and ask him questions that maybe only he knows the answer to, which is incredible.

I'll also ask him about problems that even he couldn't solve yet. And I will ask him about some of the secret sauce at Google and see if we get something. And more. And I am so happy to share it with you, fellow scholars, so we can learn together. I am not sure if I saw Jeff smile and laugh this much before, so I hope he enjoyed it, too. And once again, this is an incredible honor. I cannot believe that I was sitting there.

There were some production issues with the video part. I apologize for those. Also, I was super nervous. I could barely hold on to my papers. Now, fellow scholars, let's learn together with Jeff Dean. Thank you so much for doing this, Jeff. We talked a bit last year, and I learned so much from you. It was incredible. And then I got a message that we we get to do this, and I was so happy. So, thank you so much for this, and and we get to share your knowledge a small part of your knowledge with with the fellow scholars.

So, that's that's absolutely

great chatting with you last year, and I'm looking forward to this.

Thank you. Thank you. So, everyone says that we are running out of training data for LLMs, but you you said that there is still plenty of data out there. What did you mean?

Yeah, I mean, I think everyone has this view that uh we're running out of training data. And um it's true we've like used quite a lot of of the public text data in the world. Um but I think there's lots of interesting video data that we're not really training on yet. Uh there's lots of interesting kind of um ways to generate synthetic data and then use that for training.

Mhm.

And then I also think we can start doing things like uh making more passes over the data that we do have to make more and more capable models, and also come up with algorithmic techniques that enable us to get a lot more information from every piece of data that we do have. So, I I'm not too worried about that as like an impediment to making progress. It seems like there's lots and lots of things we can do.

People also say that with so much simulation data as as you mentioned, sooner or later most of the data will be AI-generated, which is then used to train a different AI, and then suddenly everyone starts to, you know, learn on the same thing. But you said, wait, it still helps. I think the argument was that if you have enough compute, you can crunch through a lot of data. And if there is just a little needle in the haystack that's useful, the system is able to learn from it.

Is that true? Because my previous crappy little experiments, it it was not true at all. So, you had to be very careful with the data.

Yeah, I mean, I think it is true in general. I mean, if there's a lot of details to get right to make this a reality. Think about, for example, doing RL training and rollouts to, you know, figure out how to solve some fairly high-level phrased coding question.

Mhm.

Right? So, you might explore 100 or 1,000 different ways of generating solutions to these problems, and you might have some, you know, some filters that you apply to these things like, does the code even compile? Well, you can throw out 800 of them right right off the bat. Does it pass the unit test? Does it like perform well? And so, you can really start to hone in on like which of these, you know, potentially many solutions to the problem is the one that actually sort of generates the highest, you know, characteristics that you're looking for, the reward in some [clears throat] sense.

And that I think is is definitely true. Like, more compute will generate you more interesting solutions, and then those can then be put into the training data. They can be enriched with like data augmentation techniques. You know, I generated the solution in Python. Now, I could generate a solution in Go, and have more Go programming language training data.

That's like a an incredible kind of augmentation. Like, augmentation before with convolutional neural networks, you know, it was just just shift the image by a couple pixels and whatnot. And here, the augmentation can be like completely different programming language and and whatnot.

Yeah, I mean, I think, you know, a lot of times we think about coding-based problems as you go from natural language, which is often very under-specified. Like, you know, make me a cool Space Invader game or something.

Mhm.

Um, but actually, if you have a program that already works, that does what you want, and you want want it, that's awesome. Cuz in effect your prompt is the fully specified behavior of the system you want and you just want it in a different language for whatever reason maybe better performance or better safety characteristics or whatever. So, that we've seen internally with some tools that have been written in Python and people have been able to sort of just say, "Please use all the tests for this code and the actual Python code base and make different versions of it."

And found, you know, much faster solutions. So, you can you can suddenly

get so much more out of the same amount of data, basically.

Yeah. So, so that's that's why you're not worried about the data. [clears throat] Okay, nice. Now, Bill Dally has said that something like 90% of what happens in modern data centers is not training anymore, which I I found really surprising. It's inference. Like, there's more less training and more using, like, relatively speaking. Um how does that shift the way you design hardware at Google?

Yeah, I mean at first there's a lot of other things that are not either inference or training happening in data centers, like all the applications we run and search and Gmail and so on. But of the sort of machine learning workloads, you know, I it is the case that training uh is becoming, you know, less proportion of the overall compute that we want to do because [clears throat] there's so much, you know, inference workload you want to do.

And the inference workload includes both like offline inference, uh sort of RL rollouts during RL training, uh and then also online inference for handling user requests or agent-based behavior. Because of that shift and the different characteristics of those two kinds of computations, it makes a ton more sense to now specialize much more for inference workloads in hardware, for example, because the characteristics are quite different.

You need lower precision, you you know, are handling a very large volume of requests on this particular model. The model weights don't necessarily change uh at inference time. All these things lead to very different solutions for hardware and much more energy efficiency can be gained by specializing. And so I think you'll see a lot more in this ar

原文超出正文长度上限，此处截断——上游还有内容，完整版见上方「原文 ↗」。