GPU内核工程系列：从Naive GEMM到FlashAttention
Open GPU kernel engineering series: naive GEMM to FlashAttention, H100 and Blackwell (44 sessions, code public) [P]
系统性的GPU内核优化实战指南，覆盖从基础到Blackwell新特性的完整链路，有代码和基准验证，做底层优化的同学值得收藏跟进。
I'm building a free, code-first series on GPU kernel engineering and just published the intro and roadmap video. It's my own channel, and I'm posting to share the resource and get feedback on the plan, not to sell anything.
我正在构建一个免费的、以代码为核心的 GPU 内核工程系列课程，刚刚发布了介绍和路线图视频。这是我的个人频道，发布是为了分享资源并获取对计划的反馈，而非推销任何产品。
Video: https://youtu.be/cibk7G-jHQw
视频：https://youtu.be/cibk7G-jHQw
The core idea behind the series: for most ML workloads, kernel performance is a data-movement problem before it's a compute problem. Two things the series builds on throughout:
该系列的核心思想：对于大多数机器学习工作负载而言，在内核性能成为计算问题之前，它首先是一个数据移动问题。整个系列基于以下两点：
- The roofline model. Attainable throughput is bounded by min(peak compute, memory bandwidth x arithmetic intensity), where arithmetic intensity is FLOPs per byte moved. A kernel's position relative to the ridge point tells you whether to optimize memory access or compute.
- FlashAttention as the worked example. It computes exactly the same attention as the standard implementation. The gain comes from tiling so the full N x N score matrix is never written to HBM, which cuts memory traffic. We build it live.
- 屋顶线模型（Roofline model）。可达吞吐量受限于 min(峰值计算能力, 内存带宽 x 算术强度)，其中算术强度是每移动一个字节的浮点运算次数（FLOPs/byte）。内核相对于脊点（ridge point）的位置告诉你应该优化内存访问还是计算。
- FlashAttention 作为实例演示。它计算出的注意力机制与标准实现完全相同。增益来自于分块（tiling），使得完整的 N x N 分数矩阵永远不会写入 HBM，从而减少了内存流量。我们将现场构建它。
Roadmap (44 sessions):
路线图（共 44 节）：
- Foundations: CPU parallelism, GPU architecture, roofline, first CUDA kernels
- Memory hierarchy
- GEMM: naive kernel to optimized, profiling at every step
- Tensor cores
- Profiling and debugging
- PyTorch integration and Triton
- Attention / FlashAttention
- Hopper and Blackwell: TMA, WGMMA, CUTLASS, FP8/quantization, NVFP4, multi-GPU basics, FlashAttention 4, DeepSeek's kernels, AI-written kernels, capstone project
- 基础：CPU 并行、GPU 架构、屋顶线模型、首个 CUDA 内核
- 内存层次结构
- GEMM：从朴素内核到优化版本，每一步都进行剖析
- 张量核心（Tensor cores）
- 剖析与调试
- PyTorch 集成与 Triton
- 注意力机制 / FlashAttention
- Hopper 和 Blackwell：TMA、WGMMA、CUTLASS、FP8/量化、NVFP4、多 GPU 基础、FlashAttention 4、DeepSeek 的内核、AI 生成的内核、最终项目
Each session is concept, live code, profile, then a check that it actually got faster against a baseline. Prerequisites are C/C++, basic linear algebra, and PyTorch.
每节课包括概念讲解、现场编码、性能剖析，然后通过与基线对比来验证是否确实提升了速度。 prerequisites 是 C/C++、基础线性代数和 PyTorch。
Caveats: it's in progress, and the later sessions (especially Blackwell/NVFP4) may change as the tooling does.
注意事项：课程仍在进行中，后续课程（尤其是关于 Blackwell/NVFP4 的部分）可能会随着工具链的更新而发生变化。
Feedback I'd find most useful:
我认为最有用的反馈：
- If you've written production kernels, what do people usually get wrong that a course should cover?
- 如果你编写过生产环境的内核，人们通常在哪些方面容易出错，而这些正是课程应该涵盖的内容？
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力