PyTorch 2.14 发布：NVGEMM、分布式容错与 Apple

We are excited to announce the release of PyTorch® 2.14 (release notes)!

The PyTorch 2.14 release features the following changes:

• NVGEMM brings CuTeDSL-generated CUTLASS kernels to Inductor, with epilogue fusion, scaled and NVFP4 GEMM, and grouped-reduction epilogues autotuned alongside Triton and ATen
• A new nccl2 backend for PyTorch Distributed, ported from torchcomms, implementing the full collective contract with nonblocking communicators and eager communicator splitting
• Fault tolerance becomes a first-class c10d concept, with in-place process-group reconfiguration, one-sided RMA windows, and a Flight Recorder that works for any backend rather than only NCCL
• Apple Silicon gains native linear algebra, including Jacobi-kernel SVD, eigh, QR and Cholesky, alongside a five-part reduction rewrite and a further MPSGraph to Metal kernel migration
• torch.switch generalizes torch.cond to multi-way branching, and torch.while_loop can now be captured in a CUDA graph
• Declarative dynamic shapes via @dynamic_spec, shared across torch.compile, torch.export and make_fx
• Broader platform support: ROCm 7.14 wheels are produced from the TheRock pip SDK, Intel XPU adds native graph capture, and Inductor targets Rubin (sm_107)
• Experimental torch.compile support for complex-valued tensors: Opt-in support decomposes supported complex operations into real and imaginary computations, enabling compiler backends to optimize more complex-number workloads.

This release is composed of 2,995 commits from 487 contributors since PyTorch 2.13. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.14. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

Bring any questions you may have about this release to our Q&A Webinar. It will be on Thursday, September 17, 2026 and will feature Andrey Talman (Meta), Natalia Gimelshein (Meta), Joe Spisak (Reflection AI), and Chris Gottbrath (Gottbrath Tech, moderator) who will share an overview of the 2.14 release and provide answers to community questions about PyTorch and the new capabilities in this release. Register today.

Connect with the global PyTorch community at the upcoming PyTorch Conference North America October 20-21, 2026, in San Jose, CA. Explore what’s new with PyTorch framework with sessions spanning compiler and runtime work, distributed communication, device portability, release engineering, CI, observability, accelerator integration, contributor infrastructure, and much more. PyTorch Conference is the place for engineers, researchers and maintainers solving real problems in training, inference, kernels, applications and responsible AI to convene.

Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. PyTorch 2.12 added a device-agnostic torch.accelerator.Graph API and Microscaling quantization export support. PyTorch 2.13 landed FlexAttention on Apple Silicon, gave Inductor a CuTeDSL code path, and introduced torchcomms for large-cluster training.

PyTorch 2.14 builds directly on those threads. The CuTeDSL path matures into NVGEMM, a full GEMM backend with epilogue fusion and low-precision support. torchcomms lands in-tree as the nccl2 backend, with fault-tolerant reconfiguration and one-sided RMA windows raising fault tolerance from a backend detail to a c10d concept. Apple Silicon moves from attention kernels to native linear algebra, and dynamic shapes become declarative through a spec that travels across compile, export and tracing.

PyTorch 2.14 delivers meaningful improvements across performance, reliability, and hardware support. The release introduces NVGEMM, a new GPU math backend that automatically selects the fastest kernel for matrix operations — including support for lower-precision formats that reduce memory use during training and inference. For teams training across many machines, a redesigned distributed communication backend (nccl2) offers better scalability, while new fault-tolerance features allow training jobs to recover from node failures without restarting from scratch.

Apple Silicon users benefit from native linear-algebra routines (SVD, QR, Cholesky, and more) and a broad migration to hand-tuned Metal kernels that cut overhead on Mac GPUs. On the compiler side, new control-flow primitives (torch.switch, torch.while_loop) give model authors more flexibility when writing dynamic logic that still compiles efficiently, and a new @dynamic_spec decorator provides a single, clean way to declare which tensor dimensions can change at runtime — simplifying workflows across compilation, export, and tracing.

Platform support expands to AMD ROCm 7.14, Intel XPU native graph capture, and NVIDIA’s next-generation Rubin architecture. Under the hood, the compiler now overlaps communication with computation by default, batches small GPU kernels more intelligently, and reduces per-call overhead — all of which add up to faster end-to-end model execution without any code changes from users.

Performance Improvements

Large MPS Op Migration to Native Metal

Continuing the migration begun in 2.13, a further set of MPS operators moved off Apple’s MPSGraph framework onto hand-written Metal compute kernels, including index_add, index_select, argmin, argmax, conv3d, median, nanmedian, linspace, arange, nan_to_num, log_sigmoid, sigmoid_backward, mish and GLU.

Reductions received a dedicated five-part rewrite covering full reductions, inner-dimension reductions, strided and batched outer reductions, small-dimension and narrow kernels, and the argmax and argmin split-K paths. The final part migrates min and max off MPSGraph. Skipping input up-casts and using vec4 loads removes work the MPSGraph path could not avoid.

The native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon.

API Unstable

(PR #191101, #191097, #191098, #191099 and #191100 by Irakli Salia, Hugging Face, #187109 and #188802 by Nikita Shulga, Thinking Machines Lab)

MPS Memory and Copy Paths

Long-running decode workloads grew the MPS caching allocator’s reserved footprint faster than necessary. The allocator now buckets large allocations to bound reserved memory and uses placement heaps to reduce fragmentation.

Host and device transfers also got shorter paths. CPU to MPS copies blit directly from pinned buffers with event-deferred reclaim, contiguous same-dtype copies use a compute kernel instead of a graph, elementwise ops are vectorized on inner-contiguous sliced views, and cat gains a vectorized contiguous fast path for any dimension.

API Unstable

(PR #187441 and #190438 by Irakli Salia, Hugging Face, #189512 and #188613 by Nikita Shulga, Thinking Machines Lab, #188483 and #188200 by Joona Havukainen, Apple)

F.linear Decode Path on MPS

Single-token decode passes a [B, 1, K] activation to F.linear, a shape that was falling off the fast path on MPS and costing an 8.5x slowdown on bf16 and fp16 according to the fix. The sequence-length-1 case is now routed correctly, and new GEMV kernels back the vector-matrix shapes that dominate autoregressive decoding. Together, the routing fix and new GEMV kernels close one of the largest remaining performance gaps between MPS and CUDA for autoregressive workloads.

API Unstable

(PR #189855 by Giovanni Versiglioni, Apple, #186927 by Irakli Salia, Hugging Face)

Compute and Communication Overlap On By Default in Inductor