跳到主内容
@wquguru
精选86PyTorch 博客(RSS)产品发布/更新

PyTorch媒体处理库重构:TorchCodec接管编解码

Evolution of the PyTorch Media Processing Landscape

原文
发到 X
推荐理由

PyTorch核心媒体栈重大调整,涉及底层I/O重构与ABI稳定性提升,做CV/Audio开发的同学务必关注迁移路径以避免兼容性问题。

TL;DR

TL;DR(太长不看)

If you need to decode or encode media, whether it’s images, video, or audio, use TorchCodec. If you need to transform media, use TorchVision for images and video, and TorchAudio for audio. Everything else that used to live in these libraries (models, datasets, etc.) is better served today by the wider ecosystem, for example the HuggingFace libraries.

如果你需要解码或编码媒体文件,无论是图像、视频还是音频,请使用 TorchCodec。如果你需要对媒体进行变换处理,图像和视频请使用 TorchVision,音频请使用 TorchAudio。以往存在于这些库中的其他所有内容(如模型、数据集等),如今由更广泛的生态系统来支持会更好,例如 HuggingFace 的库。

Images, video and audio are now central to a lot of model development, from vision-language models to diffusion models that generate images and video. Training these models means decoding media into tensors and transforming them, and generative models then need to encode their outputs back into media files. Over the last two years we have consolidated PyTorch’s media stack across three libraries: TorchCodec, TorchVision, and TorchAudio. This post describes where things landed, and why.

图像、视频和音频现在已成为许多模型开发的核心,从视觉-语言模型到生成图像和视频的扩散模型皆是如此。训练这些模型意味着将媒体解码为张量并进行变换,而生成式模型随后则需要将其输出重新编码回媒体文件。在过去两年中,我们将 PyTorch 的媒体栈整合到了三个库中:TorchCodec、TorchVision 和 TorchAudio。本文介绍了当前的布局及其原因。

Before and now

过去与现在

TorchCodec: the one place for decoding and encoding

TorchCodec:解码与编码的唯一去处

A few years ago, media decoding and encoding capabilities were scattered and partially duplicated across TorchVision and TorchAudio, with each one building its own stack. TorchVision had multiple entry points (io.read_video() and io.VideoReader()), spread across three backends: PyAV in Python, a C++ FFmpeg backend, and a CUDA/NVCUVID backend. Only the PyAV backend was available out-of-the-box, as the two other backends required building from source, and were tied to a specific FFmpeg version. TorchAudio had StreamReader and StreamWriter for video and audio, and additional audio decoding utils with various backends (FFmpeg, libsoundfile, libsox). Image decoding lived in TorchVision.

几年前,媒体解码和编码功能分散在 TorchVision 和 TorchAudio 中,且存在部分重复,各自构建了独立的底层架构。TorchVision 拥有多个入口点(io.read_video() 和 io.VideoReader()),分布在三个后端上:Python 端的 PyAV、C++ 端的 FFmpeg 后端以及 CUDA/NVCUVID 后端。只有 PyAV 后端开箱即用,因为另外两个后端需要从源码构建,并且绑定特定版本的 FFmpeg。TorchAudio 为视频和音频提供了 StreamReader 和 StreamWriter,并附带各种后端的额外音频解码工具(FFmpeg、libsndfile、libsox)。图像解码则位于 TorchVision 中。

In response to this, we have consolidated all decoding and encoding capabilities for media to live in TorchCodec: images, video, and audio, on CPU and CUDA. The APIs that used to live in TorchVision or TorchAudio are deprecated, or removed. We did this for three reasons:

针对这一情况,我们将所有媒体的解码和编码功能整合至 TorchCodec:涵盖图像、视频和音频,支持 CPU 和 CUDA。原先位于 TorchVision 或 TorchAudio 中的 API 已被弃用或移除。我们这样做出于以下三个原因:

One centralized location for users. With all media I/O in TorchCodec, users no longer have to work out which library happens to offer the decoder they need, and they get a library designed as a whole rather than a set of independent APIs.

为用户提供一个集中的位置。当所有媒体 I/O 都集中在 TorchCodec 中时,用户不再需要费力去查找哪个库恰好提供了他们所需的解码器,而是获得一个作为整体设计的库,而非一组独立的 API。

Performance optimizations. A single centralized place also means that our development effort goes into one library, instead of being spread across many. When decoding and encoding lived in several implementations, it was never obvious which one deserved the optimization work. Now it is, and anything we make faster in TorchCodec makes everyone faster. TorchCodec is generally more performant than the previous implementations we had in TorchVision and TorchAudio, particularly for CUDA video decoding.

性能优化。单一的中心化位置也意味着我们的开发工作集中在一个库中,而不是分散在多个库中。当解码和编码由多个实现负责时,很难明确哪个实现值得进行优化工作。现在情况不同了,我们在 TorchCodec 中实现的任何加速都能让所有人受益。TorchCodec 的整体性能通常优于我们在 TorchVision 和 TorchAudio 中的先前实现,尤其是在 CUDA 视频解码方面。

Maintenance and dependencies. Media I/O involves a lot of dependencies: six major FFmpeg versions (4 through 9 at the time of writing), NVIDIA’s codec SDK for GPU decoding, and one library per image format (libjpeg, libpng, etc.), each with its own licensing rules about what we can ship. On top of that, none of them are pure Python dependencies: they’re all C/C++ libraries, which makes the build, the CI, and the releases much harder. All of this complexity is now contained to TorchCodec, which greatly simplifies the maintenance burden for both TorchVision and TorchAudio.

维护和依赖项。媒体 I/O 涉及大量依赖项:六种主要的 FFmpeg 版本(截至撰写本文时为 4 到 9),用于 GPU 解码的 NVIDIA 编解码器 SDK,以及每种图像格式对应的一个库(libjpeg、libpng 等),每个库都有各自关于我们可以分发内容的许可规则。此外,这些都不是纯 Python 依赖项:它们都是 C/C++ 库,这使得构建、持续集成(CI)和发布变得更加困难。所有这些复杂性现在都集中在 TorchCodec 中,这大大简化了 TorchVision 和 TorchAudio 的维护负担。

TorchVision and TorchAudio: now focused on transforming media

TorchVision 和 TorchAudio:现在专注于媒体转换

TorchVision and TorchAudio supported a broad product surface, ranging across models, datasets, pipelines, transforms, and the I/O above. Today, both libraries are focused around what they do best: their transforms. The rest is not under active development.

TorchVision 和 TorchAudio 曾支持广泛的产品功能,涵盖模型、数据集、管道、转换以及上述的 I/O 功能。如今,这两个库都专注于它们最擅长的领域:它们的转换功能。其余部分不再处于活跃开发状态。

The driving factor here was the observation that the transforms in TorchVision and TorchAudio are the most active usage areas in the community. In contrast, models, datasets, and pipelines have lagged in usage, as great alternatives have risen, such as the HuggingFace libraries. Narrowing the scope of TorchAudio and TorchVision allows us to double-down on their strengths, and on the pieces that are not covered by other parts of the ecosystem.

这里的主要驱动因素是观察到 TorchVision 和 TorchAudio 中的转换功能是社区中最活跃的使用领域。相比之下,模型、数据集和管道的使用情况有所滞后,因为出现了更好的替代方案,例如 HuggingFace 库。缩小 TorchAudio 和 TorchVision 的范围使我们能够加倍投入其优势领域,并专注于生态系统中其他部分未覆盖的部分。

This transition was particularly disruptive for TorchAudio: a lot of APIs were deprecated and eventually removed. We appreciate the community’s patience and feedback, which allowed us to reconsider our initial plan. Several popular APIs we had originally slated for removal were kept. This migration was difficult, but we believe it was worth it. A smaller TorchAudio is one that can actually be maintained, so it’s one users can continue to rely on.

这一转变对 TorchAudio 造成了特别大的影响:许多 API 被弃用并最终移除。我们感谢社区的耐心和反馈,这使我们能够重新考虑最初的计划。我们原本计划移除的几个流行 API 得以保留。这次迁移虽然困难,但我们认为这是值得的。更精简的 TorchAudio 才是真正可维护的,因此它是用户可以继续信赖的。

How they fit together

它们如何协同工作

In practice the split is simple: TorchCodec turns media files or encoded bytes into tensors (decoding), and tensors back into media files (encoding). TorchVision and TorchAudio transform the tensors in between.

在实践中,拆分很简单:TorchCodec 将媒体文件或编码字节转换为张量(解码),并将张量转换回媒体文件(编码)。TorchVision 和 TorchAudio 则负责处理中间环节的张量。

For video, with TorchCodec’s VideoDecoder and TorchVision’s v2 transforms:

对于视频,使用 TorchCodec 的 VideoDecoder 和 TorchVision 的 v2 变换:

代码 · 9 行
import torch
from torchcodec.decoders import VideoDecoder
from torchvision.transforms import v2
clip = VideoDecoder("video.mp4")[10:20]   # FrameBatch, uint8, (10, C, H, W)
transform = v2.Compose([
    v2.RandomResizedCrop(224),
    v2.ToDtype(torch.float32, scale=True),
])
batch = transform(clip.data)

For audio, with TorchCodec’s AudioDecoder and TorchAudio’s transforms:

对于音频,使用 TorchCodec 的 AudioDecoder 和 TorchAudio 的变换:

代码 · 4 行
from torchcodec.decoders import AudioDecoder
from torchaudio.transforms import MelSpectrogram
samples = AudioDecoder("audio.mp3").get_all_samples()   # AudioSamples
mel = MelSpectrogram(sample_rate=samples.sample_rate)(samples.data)

Images work the same way: decode_image and friends return plain tensors that go straight into torchvision.transforms.v2. Encoding goes in the other direction, with VideoEncoder, AudioEncoder, JpegEncoder and PngEncoder.

图像的处理方式相同:decode_image 及其相关函数返回纯张量,可直接输入 torchvision.transforms.v2。编码则朝相反方向进行,涉及 VideoEncoder、AudioEncoder、JpegEncoder 和 PngEncoder。

In terms of releases, all three libraries are now ABI stable. This means that a given version of TorchCodec, TorchVision or TorchAudio isn’t tied to one single version of PyTorch anymore: it keeps working with the PyTorch versions that come after it. They no longer need to be rebuilt for every PyTorch release, so their release cadence no longer matches PyTorch’s. That’s expected!

在发布方面,这三个库现在都实现了 ABI 稳定。这意味着给定版本的 TorchCodec、TorchVision 或 TorchAudio 不再与 PyTorch 的单一版本绑定:它们能够继续兼容后续发布的 PyTorch 版本。它们不再需要为每个 PyTorch 发行版重新构建,因此其发布节奏也不再与 PyTorch 保持一致。这是预期的结果!

If you’re still using the decoding or encoding APIs in TorchVision or TorchAudio, now is a good time to move to TorchCodec: start with our migration guide, and let us know on GitHub if something you need is missing.

如果您仍在 TorchVision 或 TorchAudio 中使用解码或编码 API,现在是迁移到 TorchCodec 的好时机:请从我们的迁移指南开始,并在 GitHub 上告知我们是否缺少您需要的功能。

There is a lot more to say about the media processing libraries, and we’ll do it in a follow-up post: what’s new in each library, and how to get the most performance out of a full decoding and transform pipeline. Stay tuned!

关于这些媒体处理库还有更多内容可说,我们将在后续文章中介绍:各库的新特性,以及如何从完整的解码和变换流水线中获得最佳性能。敬请期待!

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件