PyTorch博客:FBTriton优化表批量嵌入内核性能
Modernizing Table Batched Embeddings with FBTriton
推荐系统工程师必看,这篇详细拆解了TBE内核在B200上的极致优化手段,从负载平衡到寄存器管理都有具体参数和取舍,直接指导高性能算子开发。
This post explores the FBTriton kernel design for Table Batched Embedding (TBE) forward and backward passes. These core operators handle embedding lookups across thousands of sharded GPUs within recommendation systems. Our Triton implementation successfully outperforms the legacy CUDA kernels on these workloads. Here, we explain the architectural design, detail the measured performance gains, and highlight future optimization opportunities.
本文探讨了用于表批嵌入(TBE)前向和后向传播的 FBTriton 内核设计。这些核心算子在推荐系统中处理跨数千个分片 GPU 的嵌入查找操作。我们的 Triton 实现在这些工作负载上成功超越了传统的 CUDA 内核。在此,我们将解释架构设计,详细说明测得的性能提升,并强调未来的优化机会。
1. What’s TBE
1. TBE 是什么
TBE (Table-Batched Embedding) kernel efficiently performs embedding lookups and pooling across many tables in one GPU operation. TBE combines embedding lookup and pooling for many tables in a single GPU launch, reducing launch overhead and improving memory efficiency. You can read more general info here
TBE(Table-Batched Embedding,表批嵌入)内核能够在一个 GPU 操作中高效地执行多个表的嵌入查找和池化操作。TBE 将多个表的嵌入查找和池化合并在一次 GPU 启动中,从而减少启动开销并提高内存效率。您可以在此处阅读更多通用信息
2. Implementation of Triton TBE Forward
2. Triton TBE 前向传播的实现
For each table and bag, forward gathers the indexed rows, optionally multiplies them by per-sample weights, accumulates them in FP32 (FP64 for FP32 weights), and writes one D-wide pooled output. We built two implementations: a generic gather and a fast path implementation with a small-table histogram.
对于每个表和包(bag),前向传播收集索引行,可选地将其与逐样本权重相乘,以 FP32(若权重为 FP32 则使用 FP64)进行累加,并写入一个宽度为 D 的池化输出。我们构建了两种实现:一种通用的 gather 路径,以及一种带有小表直方图的快速路径实现。
The general gather path
通用 gather 路径
- Grid. The generic launch uses ceil(B / BAGS_PER_PROGRAM) programs. Each program loops over T features instead of launching a B×T grid.
- Gather width. The inner loop issues four independent row loads. The tuned two-bag path issues eight
- Bags per program. Large non-VBE, non-FP32 workloads use two bags per program. When a histogram feature is split out, the remaining generic feature ranges use four. Other shapes use one
- Index and offset width. TorchRec accepts config-driven int32 indices and offsets when the linearized range fits below 2^31. This halves index/offset storage and the CUB radix-sort key width while keeping int64 as the default
- Accumulation. FP16/BF16 weights accumulate in FP32. FP32 weights accumulate in FP64 to preserve accuracy at large D
- 网格(Grid)。通用启动使用 ceil(B / BAGS_PER_PROGRAM) 个程序。每个程序循环遍历 T 个特征,而不是启动 B×T 的网格。
- Gather 宽度。内层循环发出四个独立的行加载请求。经过调优的双包路径发出八个。
- 每程序的包数。大型非 VBE、非 FP32 工作负载每程序使用两个包。当分离出直方图特征时,剩余的通用特征范围使用四个。其他形状使用一个。
- 索引和偏移量宽度。当线性化范围低于 2^31 时,TorchRec 接受配置驱动的 int32 索引和偏移量。这将索引/偏移量的存储和 CUB 基数排序键宽度减半,同时保持 int64 作为默认值。
- 累加。FP16/BF16 权重在 FP32 中累加。FP32 权重在 FP64 中累加,以在大 D 值下保持精度。
The small-table histogram and tensor-core path
小表直方图和张量核心路径
The specialized path is selected for one feature with E≤64, 64≤D≤128, L≥64, FP16 weights, FP32 output, no per-sample weights, and no VBE. One program handles 16 bags. It builds a histogram over the first 256 indices and evaluates counts × table with tl.dot; a scalar path handles the remaining indices. Other shapes use the generic kernel.
专用路径适用于满足以下条件的单个特征:E≤64,64≤D≤128,L≥64,FP16 权重,FP32 输出,无逐样本权重,且无 VBE。一个程序处理 16 个包。它在前 256 个索引上构建直方图,并通过 tl.dot 计算 counts × table;标量路径处理剩余索引。其他形状使用通用内核。
Bounds checking
边界检查
The standalone path uses an updated CUDA validation step before the Triton forward pass. On B200, this achieves up to 1.24x speedup on the bounds-check component across workloads. When fused_bounds_check is enabled, the eligible kernel validates and repairs invalid input tensors. Offset validation and repair remain a separate small kernel. Other configurations, such as weighted, variable batched, AMD, and transpose-hoist cases, fall back to standard validation. The option is exposed through TorchRec and defaults off.
独立路径在 Triton 前向传递之前使用更新的 CUDA 验证步骤。在 B200 上,这在工作负载中对边界检查组件实现了高达 1.24 倍的加速。当启用 fused_bounds_check 时,符合条件的内核会验证并修复无效输入张量。偏移量验证和修复仍保持为一个单独的小内核。其他配置(如加权、可变批次、AMD 和转置提升情况)回退到标准验证。该选项通过 TorchRec 暴露,默认关闭。
Forward-state reuse and preprocessing
前向状态复用与预处理
Exact row-wise Adagrad can save the forward histogram. Backward uses those counts for the compensated FP16 high/low GEMM into FP32, then applies the optimizer. Forward and backward remain separate launches; only the histogram counts are reused.
精确的行级 Adagrad 可以保存前向直方图。反向传播使用这些计数进行补偿型 FP16 高/低 GEMM 运算至 FP32,然后应用优化器。前向和反向传播仍为独立的启动操作;仅复用直方图计数。
Another optional core-module path moves index transpose, sort, and run-length encoding into forward and returns the metadata through autograd. It defaults off and is not exposed by the current TorchRec wrapper. On a large B200 configuration, forward moves from 22.844 ms to 33.252 ms, backward from 56.693 ms to 32.931 ms, and combined latency from 79.537 ms to 66.183 ms (−16.8%).
另一个可选的核心模块路径将索引转置、排序和游程编码移至前向传递中,并通过 autograd 返回元数据。该功能默认关闭,且未被当前的 TorchRec 包装器暴露。在大型 B200 配置下,前向传递时间从 22.844 毫秒增加到 33.252 毫秒,反向传播时间从 56.693 毫秒减少到 32.931 毫秒,总延迟从 79.537 毫秒减少到 66.183 毫秒(-16.8%)。
3. Implementation of Triton TBE Backward Kernel
3. Triton TBE 反向传播内核的实现
The core logic of the TBE backward propagation: for every unique (table, row) pair touched anywhere in the batch, sum the upstream gradient rows from every batch position that touched it, then apply exactly one optimizer update to that row.
TBE 反向传播的核心逻辑:对于批次中任何位置触及的每个唯一 (table, row) 对,汇总来自所有触及该行的批次位置的上游梯度行,然后对该行执行恰好一次优化器更新。
Annotations: T is number of tables; E is embedding size (i.e. number of rows); D is embedding dimension (i.e. number of cols); L is pooling factor (i.e. embedding bag size); B is batch size (i.e. number of embedding bags per request)
注解:T 是表的数量;E 是嵌入大小(即行数);D 是嵌入维度(即列数);L 是池化因子(即嵌入包大小);B 是批次大小(即每个请求的嵌入包数量)
Forward-pass. An embedding table T is a 2-D tensor (ExD) located in the GPU.
前向传递。一个嵌入表 T 是一个位于 GPU 上的二维张量 (ExD)。
- Given: a sparse feature (id-list or id-score-list) such as [id1, id2, id3, id4] (here L=4 and B=1) or multiple sparse features (B>1).
- Goal: By computing the sum T[id1%E]+T[id2%E]+T[id3%E]+T[id4%E], we get its forward output. If B=100, we will have 100 forward outputs.
- 给定:稀疏特征(id 列表或 id-score 列表),例如 [id1, id2, id3, id4](此处 L=4 且 B=1)或多个稀疏特征(B>1)。
- 目标:通过计算 T[id1%E]+T[id2%E]+T[id3%E]+T[id4%E],得到其前向输出。如果 B=100,我们将得到 100 个前向输出。
Backward-pass
反向传递
- Given: forward output and output grads (output.backward(grads))
- Goal: calculate weights.grad and update weights
- Calc grads: for each index “id”, compute the sum of all the output grads from the embedding bags containing “id”. (With TxB=4.2M, B=128K, the indice stats can be total/dedup/highest_freq = 83M/3M/125K)
- Update weights: as simple as weight = weight – grad * LR
- TBE backward has no tensor core ops but it requires heavy data move/reduction and suffers from load imbalance issues.
- 给定:前向输出和输出梯度(output.backward(grads))
- 目标:计算 weights.grad 并更新 weights
- 计算梯度:对于每个索引 “id”,计算包含 “id” 的所有嵌入包中的输出梯度之和。(在 TxB=4.2M,B=128K 的情况下,索引统计信息总计/去重/最高频率 = 83M/3M/125K)
- 更新权重:如同 weight = weight – grad * LR 一样简单
- TBE 反向传播不包含 Tensor Core 算子,但需要大量的数据移动/规约,并面临负载不均衡问题。
transpose_embedding_input inverts the batch into runs: one unique row paired with the samples that touched it. This operation is hoisted into the forward pass, off the backward critical path.
transpose_embedding_input 将 batch 转换为 runs:每一行唯一的行与访问过它的样本配对。此操作被提升到前向传播中,从而避开反向传播的关键路径。
Segment length (SL), the number of samples touching a row, is the variable everything keys on, and it spans one to millions inside a single batch. Runs are routed by SL to one of three kernels.
分段长度 (SL),即访问某行的样本数量,是决定一切的关键变量,在单个 batch 内跨度从一到数百万不等。Runs 根据 SL 路由到三个内核之一。
- short_run (SL < 256): one program owns the run start to finish: gather, accumulate in registers, optimizer, store. Exclusive ownership means a plain store, no atomics.
- grad_accum + apply (SL >= 256, default): the run is split into 256-lookup chunks, partials land in a workspace, a second kernel applies the optimizer.
- fused (SL >= 256, Blackwell, very large batches): same split, but a device-scope fence lets the last sub-program apply the update in one launch.
- short_run (SL < 256):一个程序独占该 run 从头到尾的处理:gather、在寄存器中累加、优化器更新、存储。独占所有权意味着普通存储,无需原子操作。
- grad_accum + apply (SL >= 256,默认):run 被拆分为 256-lookup 的块,部分结果存入工作区,第二个内核应用优化器。
- fused (SL >= 256,Blackwell,超大 batch):相同的拆分方式,但设备级栅栏允许最后一个子程序在一次启动中完成更新应用。
Weighted tables differ only in scaling each gradient row by its per-sample weight before accumulating.
加权表仅在累加前按每个样本的权重对每个梯度行进行缩放,其余部分相同。
4. Edge Cases and performance improvements
4. 边缘情况与性能改进
Problem 1. imbalanced programs
问题 1. 程序负载不均衡
A single program walking two million lookups leaves the GPU idle, while a huge tail of one-lookup runs pays full per-run cost for almost no work.
一个程序处理两百万次查找会导致 GPU 空闲,而大量仅含一次查找的 run 却为几乎无用的工作支付全额的 per-run 成本。
Solution 1.1: split long runs. Runs at or above the threshold become fixed 256-lookup chunks, split-K style. A two-million-lookup run turns into roughly eight thousand sub-programs, enough to fill the machine from one row’s work.
解决方案 1.1:拆分长 run。达到或超过阈值的 run 变为固定 256-lookup 的块,采用 split-K 风格。一个两百万次查找的 run 会被拆分为大约八千个子程序,足以利用单行工作填满机器资源。
Solution 1.2: CLC (TLX Blackwell) A kernel will persist once started initially and keep stealing ‘free’ run_id after finishing the current ‘run_id’. If the kernel steals a workload successfully, it will process it in the same block. Otherwise, the kernel exits. To address the load imbalance issues, we no longer need a software-based solution like bifurcating run_id processing by frequency. Instead CLC provides a hardware-based solution.
解决方案 1.2:CLC (TLX Blackwell) 内核一旦初始启动便会持久运行,并在完成当前 'run_id' 后继续抢占空闲的 'run_id'。如果内核成功抢占任务,它将在同一 block 中处理;否则内核退出。为解决负载不均衡问题,我们不再需要基于软件的分叉 run_id 处理方案(如按频率分支)。相反,CLC 提供了一种基于硬件的解决方案。
Problem 2. the gather width is a register cliff
问题 2. gather 宽度存在寄存器悬崖
Each buffered row keeps BLOCK_SIZE 64-bit addresses live, because dout_row_start_ptr[:, None] + col_offsets[None, :] materializes a whole pointer tile. Cost is width × BLOCK_SIZE, so a width tuned at one row width is wrong at another.
每个缓冲行保持 BLOCK_SIZE 个 64 位地址处于活跃状态,因为 dout_row_start_ptr[:, None] + col_offsets[None, :] 会实例化整个指针 tile。开销为 width × BLOCK_SIZE,因此在一个 row width 下调优的 width 在另一个 row width 下是错误的。
Solution: per-target config width, in every tier. Measured on B200:
解决方案:在每个层级使用针对目标的配置 width。在 B200 上测得:
| tier | width | registers | occupancy | effect |
|---|---|---|---|---|
| short run, unweighted | 8 → 2 | 184 → 64 | 12.5% → 49.9% | 0.41 → 1.05 |
| long run, accumulate | 8 → 2 | 158 → 62 | 17.6% → 44.7% | fleet parity 82% → 87% |
| short run, weighted | 4 → 2 | 125 → 64 | 24.8% → 49.3% | weighted parity 51% → 69% |
| 层级 | 宽度 | 寄存器 | 占用率 | 效果 |
|---|---|---|---|---|
| 短运行,未加权 | 8 → 2 | 184 → 64 | 12.5% → 49.9% | 0.41 → 1.05 |
| 长运行,累积 | 8 → 2 | 158 → 62 | 17.6% → 44.7% | 车队均衡 82% → 87% |
| 短运行,加权 | 4 → 2 | 125 → 64 | 24.8% → 49.3% | 加权均衡 51% → 69% |
Problem 3. BLOCK_SIZE is a constexpr
问题 3。BLOCK_SIZE 是一个 constexpr
One launch must size BLOCK_SIZE to next_pow2(max_D) across all tables, so when the lookup-dominant table is much narrower than the widest, most of every gathered row is masked-off lanes.
一次启动必须将 BLOCK_SIZE 调整为所有表中 next_pow2(max_D) 的值,因此当查找主导的表比最宽的表窄得多时,每个收集行的大部分通道都会被屏蔽。
To overcome that, we bucket by dimension. Short runs are routed to one bucket per next_pow2(D) during classification, each launching with its own BLOCK_SIZE. It folds into the classification kernel, so it costs no extra pass; profiling confirms it buys reduced lane waste, not occupancy.
为了克服这个问题,我们按维度进行分桶。在分类期间,短运行被路由到每个 next_pow2(D) 对应的一个桶中,每个桶使用自己的 BLOCK_SIZE 进行启动。它折叠进分类内核中,因此不会增加额外的遍历;性能分析证实它减少了通道浪费,而非占用率。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力