跳到主内容
@wquguru
精选88Avi Chawla技巧与观点

GPU工作组织机制详解与RDU架构性能实测

How GPUs organize work, clearly explained:

原文
发到 X
推荐理由

从底层硬件调度讲透GPU原理,并附带RDU架构的实测数据对比,对做推理优化的工程师极具参考价值。

How GPUs organize work, clearly explained:

GPU 如何组织工作,清晰解释:

(popular AI engineering interview topic)

(流行的 AI 工程面试话题)

A kernel launch can create millions of threads, but they don't all run at once.

内核启动可以创建数百万个线程,但它们并非同时运行。

The GPU assigns groups of threads to available sections of the chip. Within each section, it schedules instructions from threads that are ready to make progress.

GPU 将线程组分配给芯片上可用的部分。在每个部分内,它调度那些准备好取得进展的线程的指令。

The diagram follows this process, covering the function a program launches to the instructions the hardware executes.

该图表遵循这一过程,涵盖了程序启动的功能到硬件执行的指令。

1. The program launches a kernel

1. 程序启动内核

A kernel is a function containing the instructions the GPU should follow.

内核是包含 GPU 应遵循的指令的函数。

For this example, consider that each thread follows the same instructions, i.e., read two numbers, add them, and store the answer.

在本例中,假设每个线程遵循相同的指令,即读取两个数字、将它们相加并存储答案。

Each thread gets an index so it knows which pair of numbers to use.

每个线程获得一个索引,以便知道使用哪一对数字。

2. Threads are organized into blocks

2. 线程被组织成块

Rather than manage every thread separately, the program groups them into blocks.

程序不是单独管理每个线程,而是将它们分组为块。

Suppose there are 1,024 threads, grouped into blocks of 256. This gives four blocks.

假设有 1,024 个线程,每 256 个分为一组。这样就形成了四个块。

Together, these blocks form the grid for this launch.

这些块共同构成了此次启动的网格(grid)。

The GPU assigns blocks to its streaming multiprocessors, or SMs. An SM is a section of the chip that can execute threads.

GPU 将块分配给其流式多处理器(SM)。SM 是芯片上能够执行线程的部分。

All threads in a block run on the same SM. Multiple blocks can share an SM if enough resources are available.

一个块中的所有线程都在同一个 SM 上运行。如果资源足够,多个块可以共享一个 SM。

3. The SM runs threads in groups of 32

3. SM 以 32 个线程为一组运行线程

These groups are called warps.

这些组称为 warp。

Our 256-thread block contains eight warps. When a warp executes the addition instruction, its active threads each add their own pair of numbers.

我们的 256 线程块包含八个 warp。当 warp 执行加法指令时,其活动线程各自对自己的一对数字进行相加。

The grouping lets the hardware apply one instruction across many values.

这种分组允许硬件对许多值应用一条指令。

If threads take different branches, some become inactive while others execute their branch. That means less work gets done by that instruction.

如果线程走不同的分支,一些线程将变为非活动状态,而其他线程则执行各自的分支。这意味着该指令完成的工作量减少了。

4. Waiting threads don't always stop other work

4. 等待中的线程并不总是阻止其他工作

A thread needs its input numbers before it can add them.

线程需要其输入数字才能进行相加。

If a warp is waiting for data, the SM can run an instruction from another warp that's ready.

如果一个 warp 正在等待数据,SM 可以运行另一个已准备好的 warp 的指令。

Keeping several warps on the SM helps it find useful work during those wait times because if none are ready, the execution stalls.

在 SM 上保留多个 warp 有助于它在这些等待时间内找到有用的工作,因为如果没有就绪的 warp,执行将会停滞。

5. Threads can also cooperate

5. 线程还可以协作

Adding separate pairs needs no communication between threads, but there can be cases, like adding all those answers together, which require communication.

添加独立的配对不需要线程间通信,但有些情况(例如将所有这些答案相加)则需要通信。

Threads within a block can exchange partial results through shared memory and synchronize before using them.

同一块内的线程可以通过共享内存交换部分结果,并在使用前进行同步。

Combining results across blocks often happens in a later kernel, once the earlier work has finished.

跨块的组合结果通常发生在后续的 kernel 中,此时早期工作已经完成。

The GPU therefore needs both enough work to run in parallel and a way to handle dependencies correctly.

因此,GPU 既需要足够的并行工作量,也需要一种正确处理方法依赖关系的方式。

Memory management is important as well.

内存管理同样重要。

Registers hold individual threads' working values, shared memory supports cooperation within a block, and L2 caches data across SMs. Larger HBM stores data such as model weights.

寄存器保存各个线程的工作值,共享内存支持块内协作,L2 缓存跨 SM 的数据。更大的 HBM 存储模型权重等数据。

During LLM decode at small batch sizes, moving those weights can take longer than the calculations.

在小批量下进行 LLM 解码时,移动这些权重的时间可能比计算本身更长。

I covered that bottleneck in my RDU article, explaining how on-chip pipelines reduce data movement.

我在 RDU 文章中讨论了这一瓶颈,解释了片上流水线如何减少数据移动。

In my test, an RDU system using multiple chips generated 179 tokens per second, about 4x the estimated maximum for a single H100 running the same model at the same precision.

在我的测试中,使用多芯片的 RDU 系统每秒生成 179 个 token,约为单颗 H100 以相同精度运行同一模型时估计最大值的 4 倍。

Read it below.

请阅读下文。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件