PyTorch博客:将Spyre打造为原生设备
Building Spyre as a Native PyTorch Device
这是一篇极具价值的工程实践指南,详细拆解了如何将非GPU硬件深度集成到PyTorch底层。对于从事异构计算或自定义硬件集成的工程师来说,其中的设备抽象映射策略非常值得参考。
TL;DR
TL;DR(太长不看)
Spyre becomes a native PyTorch device by connecting PyTorch’s existing device, allocator, stream, and compiler abstractions through torch-spyre to the Spyre runtime and firmware. PrivateUse1 gives Spyre a real device identity, PyTorch allocator storage keeps tensors resident on device="spyre", streams provide familiar ordering semantics, and runtime events connect dependent work across streams. FX graphs stay in the Inductor compiler path. This post focuses on the PyTorch-to-Spyre device/runtime integration: how device identity, allocator-backed storage, streams, events, and compiled-program launch map onto Spyre’s runtime constraints. Once a compiled artifact exists, launch is a prepared recipe of typed operations over ordered queues rather than a graph-based runtime interface. The result is one path for eager and compiled execution, lower launch overhead, and a PyTorch surface that maps cleanly onto Spyre’s hardware.
通过 torch-spyre,Spyre 将 PyTorch 现有的设备、分配器、流和编译器抽象连接到 Spyre 运行时和固件,从而成为原生的 PyTorch 设备。PrivateUse1 赋予 Spyre 真实的设备标识,PyTorch 分配器存储使张量驻留在 device="spyre" 上,流提供熟悉的排序语义,而运行时事件则连接跨流的依赖工作。FX 图保持在 Inductor 编译器路径中。本文重点介绍 PyTorch 到 Spyre 的设备/运行时集成:设备标识、基于分配器的存储、流、事件以及编译程序的启动如何映射到 Spyre 的运行时约束。一旦编译产物存在,启动就是一组针对有序队列的类型化操作的预制配方,而非基于图的运行时接口。其结果是统一了急切执行和编译执行的路径,降低了启动开销,并使 PyTorch 接口能够清晰地映射到 Spyre 硬件上。
Spyre is IBM’s dataflow AI accelerator, optimized for inference. It is designed for enterprise teams running AI alongside their applications and data on IBM Z, LinuxONE, and Power systems. Its reduced-precision compute is well suited to the matrix-heavy work in language generation and embedding models.
Spyre 是 IBM 的数据流 AI 加速器,专为推理任务优化。它旨在支持企业在 IBM Z、LinuxONE 和 Power 系统上运行 AI 及其应用程序和数据。其低精度计算能力非常适合语言生成和嵌入模型中密集的矩阵运算。
For developers, the challenge is using that hardware through familiar PyTorch code while keeping up with new models and inference techniques. Efficient execution needs more than a compiler: tensors must be able to stay on the device between operations, launches need to be lightweight, and host work and data transfers should overlap computation wherever dependencies allow. This post explains how IBM’s torch-spyre team builds on the PyTorch community’s interfaces to support those workflows on a dataflow accelerator.
对于开发者而言,挑战在于使用熟悉的 PyTorch 代码来利用该硬件,同时跟上新模型和推理技术的步伐。高效执行不仅仅需要编译器:张量必须在操作之间保留在设备上,启动过程需要轻量级,并且主机工作和数据传输应在依赖关系允许的情况下与计算重叠。本文解释了 IBM 的 torch-spyre 团队如何利用 PyTorch 社区的接口,在数据流加速器上支持这些工作流程。
The dataflow constraints behind the PyTorch surface
PyTorch 接口背后的数据流约束
The card has 32 cores connected by a high-bandwidth ring. Each core has 2 MB of local scratchpad and arrays of processing elements. Up to 128 GB of LPDDR5 provides storage for tensors and programs.
该卡拥有 32 个核心,通过高带宽环形总线连接。每个核心配备 2 MB 本地暂存区(scratchpad)和处理元素阵列。高达 128 GB 的 LPDDR5 内存用于存储张量和程序。
Those two levels of memory have different owners. The runtime stack allocates and manages LPDDR5. Before a program runs, the compiler has already emitted the loads and stores that move tiles between LPDDR5 and each core’s scratchpad. Data moves across that path in 128-byte sticks, so layout and alignment affect how efficiently it can be fetched. The runtime does not allocate or schedule the scratchpad directly.
这两层内存由不同的所有者管理。运行时栈负责分配和管理 LPDDR5。在程序运行之前,编译器已经生成了在 LPDDR5 和各核心的暂存区之间移动数据块的加载和存储指令。数据以 128 字节为单位沿该路径传输,因此布局和对齐方式会影响获取效率。运行时不直接分配或调度暂存区。
A native PyTorch device still has to honor the execution model of the hardware underneath it. For Spyre, that means mapping PyTorch’s device, allocator, stream, event, and launch abstractions onto a runtime built around compiled programs, ordered queues, explicit data movement, and fixed layout contracts. Four constraints drive the design.
原生 PyTorch 设备仍需遵循其底层硬件的执行模型。对于 Spyre 而言,这意味着将 PyTorch 的设备、分配器、流、事件和启动抽象映射到一个围绕编译程序、有序队列、显式数据移动和固定布局契约构建的运行时上。四个约束驱动了该设计。
Computation is triggered by data, not by a single kernel instruction stream. A GPU kernel is typically organized around scheduled groups of threads executing the kernel’s instruction sequence. A compiled Spyre kernel is arranged differently: it contains programs for multiple functional units – such as processing elements, special-function pipelines, and load/store units – and each unit runs the program compiled for that unit. Hardware queues and local memories connect those units, so progress is driven by operands becoming available at the consuming unit. The Spyre compiler builds that device-side producer/consumer schedule; the runtime submits the compiled kernel as a complete device-compute operation with its tensor arguments.
计算由数据触发,而非由单个内核指令流触发。GPU 内核通常围绕执行内核指令序列的调度线程组进行组织。编译后的 Spyre 内核则以不同方式排列:它包含多个功能单元的程序——如处理元素、特殊功能流水线以及加载/存储单元——每个单元运行为其编译的程序。硬件队列和本地内存连接这些单元,因此进展由操作数在消费单元处可用所驱动。Spyre 编译器构建了这种设备端的 producer/consumer 调度;运行时将整个编译后的内核作为带有张量参数的完整设备计算操作提交。
Runtime overlap comes from separate pipelines, not concurrent compute launches. Spyre has a compute pipeline and data-movement pipelines, so a transfer can run while a compiled program is computing. Those pipelines are not interchangeable: a compiled program can use many cores in parallel, but independently submitted compute operations share one runtime compute queue and wait behind each other. Runtime-level overlap therefore comes from moving data while computing, not from launching two compute operations at once.
运行时重叠来自独立的流水线,而非并发的计算启动。Spyre 拥有计算流水线和数据移动流水线,因此在编译程序计算的同时可以执行传输。这些流水线不可互换:编译后的程序可以并行使用多个核心,但独立提交的计算操作共享一个运行时计算队列并相互等待。因此,运行时级别的重叠来自于边计算边移动数据,而非同时启动两个计算操作。
Ordered queues preserve per-stream completion order. Work arrives as a sequence of typed operations – move this in, run this, move that out – and each stream completes those operations in order. To use the separate hardware pipelines without weakening that guarantee, the runtime assigns independent transfer and compute work to different streams and connects them with an event where one consumes the other’s result. Putting all work on one stream preserves correctness but serializes those pipelines.
有序队列保留每个流的完成顺序。工作以一系列类型化操作的形式到达——将此移入、运行此、将此移出——每个流按顺序完成这些操作。为了在不削弱该保证的前提下使用独立的硬件流水线,运行时将独立的传输和计算工作分配给不同的流,并通过事件将它们连接起来,其中一个流消费另一个流的结果。将所有工作放在一个流上可保证正确性,但会使这些流水线串行化。
Programs are compiled ahead of time against a fixed layout. The compiler fixes how values compose into the device’s native fixed-size chunks and what order dimensions appear in, and it can fix addresses for memory belonging to the job itself – intermediates and compiler-owned temporary buffers. What it cannot fix is anything whose address the caller owns: tensor inputs and outputs stay symbolic because the allocator has not placed them yet. The runtime must supply those addresses at launch.
程序在编译时针对固定的布局进行编译。编译器确定值如何组合成设备的原生固定大小块、维度出现的顺序,并可以固定作业自身内存(中间变量和编译器拥有的临时缓冲区)的地址。它无法固定的是由调用方拥有地址的内容:张量输入和输出保持符号化,因为分配器尚未放置它们。运行时必须在启动时提供这些地址。
In the implementation described here, PyTorch does not talk to the card directly. torch-spyre provides the PyTorch device integration, while the Spyre runtime and firmware manage device work and communication. That layering is why the rest of this post separates PyTorch concepts such as tensors, streams, and events from lower-level concerns such as device regions, DMA-visible host staging buffers, and explicit work submission.
在本文描述的实现在中,PyTorch 不直接与显卡通信。torch-spyre 提供 PyTorch 设备集成,而 Spyre 运行时和固件管理设备工作和通信。这种分层结构使得本文其余部分将 PyTorch 概念(如张量、流和事件)与较低层级的关注点(如设备区域、DMA 可见的主机暂存缓冲区和显式工作提交)区分开来。
The PyTorch abstractions that Spyre maps onto
Spyre 映射到的 PyTorch 抽象
PyTorch already has concepts that map onto those constraints.
PyTorch 已经具备映射到这些约束的概念。
A device identity. PrivateUse1 registration starts with two calls:
一个设备标识符。PrivateUse1 注册始于两次调用:
torch.utils.rename_privateuse1_backend("spyre")
torch._register_device_module("spyre", make_spyre_module())Those calls give PrivateUse1 the name spyre and register its device module. With the allocator, copy kernels, and dispatcher registrations behind that identity, tensor.to("spyre") works and the dispatcher routes Spyre tensors to Spyre kernels.
这些调用赋予 PrivateUse1 名称 spyre 并注册其设备模块。有了该标识符背后的分配器、复制内核和分发器注册,tensor.to("spyre") 即可正常工作,且分发器会将 Spyre 张量路由到 Spyre 内核。
Device-management APIs. Backend-specific controls live under torch.spyre, while torch.accelerator gives portable code a common interface for availability, device selection, streams, synchronization, and memory reporting. Tensor allocation and kernel launch still flow through tensors, the dispatcher, the device allocator, and the backend runtime.
设备管理 API。后端特定的控制位于 torch.spyre 下,而 torch.accelerator 为便携代码提供了用于可用性、设备选择、流、同步和内存报告的通用接口。张量分配和内核启动仍然通过张量、分发器、设备分配器和后端运行时进行。
A stream abstraction. Implementing the device-guard hooks the framework asks for is what makes native torch.Stream(device="spyre") and torch.stream(...) work on your device. The framework-native surface works because those hooks are there, not because the backend reimplemented PyTorch’s stream API.
一个流抽象。实现框架请求的设备守卫钩子使得 native torch.Stream(device="spyre") 和 torch.stream(...) 在你的设备上正常工作。框架原生的表面之所以有效,是因为存在这些钩子,而不是因为后端重新实现了 PyTorch 的流 API。
A device allocator interface. Its core contract hands back device storage and a way to release it. Additional hooks connect the backend to PyTorch’s memory statistics, cache controls, and stream association.
一个设备分配器接口。其核心契约交还设备存储及其释放方式。额外的钩子将后端连接到 PyTorch 的内存统计、缓存控制和流关联。
Reference-counted tensor storage. Tensor lifetime becomes the framework’s problem: when the last reference to a Spyre tensor goes away, PyTorch calls back into the backend allocator to release or recycle the device allocation.
引用计数的张量存储。张量生命周期成为框架的问题:当对 Spyre 张量的最后一个引用消失时,PyTorch 会回调后端分配器以释放或回收设备分配。
A compiler path through Inductor. torch.compile(backend="inductor") keeps FX graphs in PyTorch’s compiler pipeline. The same lowering, cache, and launch path can serve compiled models and registered eager operations.
Inductor 的编译路径。torch.compile(backend="inductor") 使 FX 图保留在 PyTorch 的编译器流水线中。相同的降级、缓存和启动路径可同时服务于已编译模型和注册的即时(eager)操作。
An event model. PyTorch defines events as a way to express ordering between streams. Spyre uses the same model internally through runtime events and derived dependency edges.
事件模型。PyTorch 将事件定义为表达流之间顺序关系的一种方式。Spyre 通过运行时事件和派生的依赖边,在内部使用相同的事件模型。
Laid side by side, the correspondence is close:
并排对比,对应关系十分紧密:
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力