黄仁勋谈英伟达：AI革命与极致协同设计

• The following is a conversation with Jensen Huang, CEO of NVIDIA, one of the most important and influential companies in the history of human civilization. NVIDIA is the engine powering the AI revolution, and a lot of its success can be directly attributed to Jensen's sheer force of will and his many brilliant bets and decisions as a leader, engineer, and innovator. This is Lex Fridman Podcast. And now dear friends, here's Jensen Huang.

You've propelled NVIDIA into a new era in AI, moving beyond its focus on chip scale design to now rack scale design. And I think it's fair to say that winning for NVIDIA for a long time used to be about building the best GPU possible, and you still do, but now you've expanded that to extreme co-design of GPU, CPU memory, networking, storage, power cooling, software, the rack itself, the pod that you've announced, and even the data center.

So let's talk about extreme co-design. What is the hardest part of co-designing system with that many complex components and design variables? - Yeah, thanks for that question. So first of all, the reason why extreme co-design is necessary is because the problem no longer fits inside one computer to be accelerated by one GPU. The problem that you're trying to solve is you would like to go faster than the number of computers that you add.

So you added you know, 10,000 computers, but you would like it to go a million times faster. Then all of a sudden you have to take the algorithm, you have to break up the algorithm, you have to refactor it, you have to shard the pipeline, you have to shard the data, you have to shard the model. Now all of a sudden when you distribute the problem this way, not just scaling up the problem, but you're distributing the problem, then everything gets in the way.

This is the Amdahl's Law problem where the amount of speed up you have for something depends on how much of the total workload it is. And so if computation represents 50% of the problem, and I sped up computation infinitely like a million times, you know, I only sped up the total workload by a factor of two. Now all of a sudden, not only do you have to distribute a computation, you have to, you know, shard the pipeline somehow.

You also have to solve the networking problem because you've got all of these computers are all connected together. And so distributed computing at the scale that we do, the CPU is a problem, the GPU is a problem, the networking is a problem, the switching is a problem. And distributing the workload across all these computers is a problem. It's just a massively complex computer science problem. And so we just gotta bring every technology to bear.

Otherwise, we scale up linearly or we scale up based on the capabilities of Moore's Law, which has largely slowed because Dennard scaling has slowed. - I'm sure there's trade-offs there. Plus you have a complete disparate disciplines here. I'm sure you have specialists in each one of these high bandwidth memory, the network and the NVLink, the NICs, the optics and the copper that you're doing, the power delivery, the cooling, all of that.

I mean, there's like world experts in each of those. How do you get 'em in a room together to figure out- - That's why my staff is so large. - What's the process—can you take me through the process of the specialists and the generalists? Like how do you put together the rack when you know the s- the set of things you have to shove into a rack together? Like what does that process look like of designing it all together? - Yeah.

There's the first question, which is: what is extreme co-design? You're, you, we're optimizing across the entire stack of software from architectures to chips, to systems, to system software, to the algorithms, to the applications. That's one layer. The second thing that you and I just talked about is goes beyond CPUs and GPUs and networking chips and scale up switches and scale out switches. And then of course, you gotta include power and cooling and all of that because, you know, all these computers are extremely, extremely power hungry.

They do a lot of work and they're very energy efficient, but they in aggregate still consume a lot of power. And so that's one. The first question is, what is it? The second question is, why is it, and we just spoke about the reason, you know you want to distribute the workload so that you can exceed the benefit of just increasing the number of computers. And then the third question is, how is it, how do you do it? And that's the, that's kind of the miracle of this company.

You know, when you're designing a computer, you have to have operating system of computers. When you're designing a company, you should first think about what is it that you want the company to produce. You know, I see a lot of companies organization charts, and they all look the same. Hamburger organization charts, soft organization charts, and car company organization charts. They all look the same. And it doesn't make any sense to me.

You know, the goal of a company is to be the company is to be the machinery, the mechanism, the system that produces the output. And that output is the product that we like to create. It is also designed, the architecture of the company should reflect the environment by which it exists. It almost indirectly says what you should do with the organization. My direct staff is 60 people. You know, I don't have one-on-ones with 'em because it's impossible.

You can't have, you can't have 60 people on your staff if you're, you know, gonna get work done and- - So you still have 60 reports. You still have across- - More, yeah. - More. And most stars at least have a foot in engineering. - Almost all of them. There's experts in memory, there's experts in CPUs, there's experts in optical. All, all— - That's incredible. - Yeah, GPUs and— Architecture, algorithms, design, um— - So, you constantly have an eye on the entire stack, and you're having to, like, intense discussions about the designer of the entire stack? - And no conversation is ever one person.

That's why I don't do one-on-ones. We present a problem and all of us attack it. You know, because we're doing extreme co-design. And literally, the company is doing extreme co-design all the time. - So, even if you're talking about a particular component, like cooling, networking, everybody's listening in? - Yeah, exactly. - And they can contribute, "Well, this doesn't work for the power distribution. This doesn't-" - Exactly. - "...

This doesn't work for the memory. This doesn't work for this." - Exactly. And whoever wants to tune out, tune out. You know what I'm saying? And the reason for that is because the people who are on the staff, they know when to pay attention. There's supposed... You know, it's something they could have contributed to, they didn't contribute to, "I'm going to call them out." You know? And so, "Hey, come on, let's get in here." - So, as you mentioned, NVIDIA is this company that's adapting to the environment.

So, at which point can you say, did the environment change and began adapting sort of secretly- ... in the early days from GPU for gaming, maybe the early deep learning revolution to we're now going to start thinking of it as an AI factory? What does NVIDIA do? It produces AI, let's build a factory that makes AI. - Uh, I could, I c- you, you could- I could reason through what just systematically. We started out as a, as an accelerator company.

But the problem with accelerators is that the application domain's too narrow. It has the benefit of being incredibly optimized for the job. You know, any specialist has that benefit. The problem with intense specialization is that, of course, your market reach is narrower, but that's, that's even fine. The problem is, the market size also dictates your R&D capacity. And your R&D capacity ultimately dictates the influence and impact that you can possibly have in computing.

And so, when we first started out as an accelerator,

原文超出正文长度上限，此处截断——上游还有内容，完整版见上方「原文 ↗」。