PyTorch 大会 2026 聚焦 vLLM：KV 缓存、硬件移植与内核优化

TL;DR

PyTorch Conference North America 2026 features vLLM across sessions on KV cache management and disaggregated serving, hardware portability, kernel optimization, PyTorch integration, Mixture-of-Experts inference, attention, and production serving.

vLLM at #PyTorchCon NA

PyTorch Conference North America 2026 comes to San Jose, CA, October 20–21, with technical talks, live demos, lightning talks, sponsored sessions, a keynote, and a Birds of a Feather discussion featuring vLLM.

Across the program, vLLM appears in sessions on serving architecture and KV cache work, hardware portability, kernel and performance optimization, and PyTorch integration. Additional sessions cover Mixture-of-Experts inference, attention, production deployment, broader application stacks, and open source contribution.

View the full conference schedule

Register for PyTorch Conference North America 2026

Serving Architecture, KV Cache, and Production Inference

A Developer’s Guide to Attention in vLLM

Lucas Wilkinson, Red Hat; Matthew Bonanni, Red Hat

October 20, 11:45 a.m.–12:10 p.m. | LL20AB | Breakout Session

This session explains how vLLM represents, serves, and optimizes attention as models adopt approaches including sliding windows, sparsity, compression, linear variants, and hybrid attention.

The speakers cover attention backends, KV-cache connectors, and the hybrid memory allocator, along with a recent overhaul of vLLM’s attention abstractions, what changed, and how the new design makes emerging architectures easier and cleaner to support.

State-of-the-Art KV Transfer for Disaggregated LLM Serving in vLLM

Nicolò Lucchesi, Mistral AI; Sunita Nadampalli, Amazon; Zhanqiu Hu, Red Hat

October 20, 2:15–2:40 p.m. | LL20CD | Breakout Session

This session covers developments in vLLM’s disaggregated serving stack for transferring KV cache between prefill and decode. Topics include hybrid-model transfer with heterogeneous tensor parallelism, bidirectional KV transfer, the KV Push connector, and KV cache leases for reliability.

The speakers report that KV Push reduces time to first token and that, on Nemotron, disaggregated prefill/decode Pareto-dominates co-located serving across concurrency levels.

Native Tiered KV Cache Offloading in vLLM: From Storage Offloading to Disaggregated Serving

Or Ozeri, IBM

October 20, 2:50–3:15 p.m. | LL20CD | Breakout Session

This talk presents vLLM’s native tiered KV cache offloading framework, newly integrated upstream with no external dependencies.

The framework routes transfers through CPU memory as a universal transport hub. The design minimizes GPU transfer overhead, consolidates I/O through a CPU buffer, avoids specialized transfer APIs, and remains independent of KV cache memory layout across hardware, attention backends, parallelism schemes, and model architectures.

LMCache: a cluster-wide open source solution for LLM prompt caching

Kuntai Du, Tensormesh, Inc.

October 20, 3:40–3:50 p.m. | LL20CD | Lightning Talk

LMCache provides prompt caching across inference engines including vLLM, SGLang, and TensorRT-LLM and storage systems including Mooncake, Redis, and AWS S3.

The session includes a tutorial on deploying LMCache in Kubernetes along with the techniques and research behind its prompt-caching approach.

vLLM KV Cache Management for Model-Specific Requirements

Mengqing Cao, Huawei

October 20, 5:30–5:40 p.m. | LL20CD | Lightning Talk

This session addresses KV cache requirements that vary across model architectures including MLA, SWA, Eagle, and DeepSeek-V4.

It proposes a model-customized KV Cache Planner built around a default planner plus model-specific planners for requirements such as spec grouping, block-size derivation, cache tensor creation, and max_model_len adjustment.

Elastic Expert Parallelism in vLLM

Itay Alroy, NVIDIA

October 21, 2:50–3:15 p.m. | LL20CD | Breakout Session

Elastic Expert Parallelism enables vLLM deployments to add or remove workers at runtime and redistribute experts across the updated worker set with minimal interruption to serving.

The session covers communication reconfiguration, CUDA Graph recapture, expert rebalancing through the EP Load Balancer, weight transfer to new GPUs, and coordination with model forward execution. It also covers how NIXL EP enables grow and shrink operations under live traffic, fault detection, reporting, and recovery.

Prefix Caching for Autoregressive Stages in Multi-Stage Pipelines

Ricardo Noriega, Red Hat; Alex Brooks, Red Hat

October 21, 4:20–4:45 p.m. | LL20AB | Breakout Session

This talk explores Automatic Prefix Caching for Stage Outputs in vLLM-Omni, an approach for extending vLLM’s prefix caching to multi-stage models while minimizing GPU memory cost.

The approach aligns external CPU tensor caches with vLLM’s native block management. The speakers also describe how vLLM-Omni dynamically discovers cacheable tensors without requiring manual configuration.

Hardware Portability and Accelerator Backends

Sponsored: Unifying Open Source LLM Serving on Google Cloud TPUs with TorchTPU

Rob Mulla, Google

October 20, 10:40–10:50 a.m. | Community Expo | Demo Theater

This 10-minute demo shows TorchTPU’s native, high-performance compilation path for PyTorch models on Cloud TPUs.

The demo highlights TorchTPU as a unified backend for inference engines including vLLM and SGLang, enabling model deployment through those serving engines with minimal code modifications.

Sponsored: PyTorch Ecosystem Running Natively on Trainium

Maen Suleiman, Amazon Web Services

October 20, 10:55–11:05 a.m. | Community Expo | Demo Theater

This live demo covers PyTorch workflows on Trainium through TorchNeuron, including training with TorchTitan or Hugging Face Transformers v5, serving with vLLM-Neuron, profiling with Neuron Explorer, and adding NKI kernels directly to PyTorch code.

The session also demonstrates Neuron Agentic Development, AI-assisted tooling for kernel authoring and optimization.

One Model Definition, Many Accelerators: Scaling vLLM Across Hardware Without Forks

Thomas Parnell, IBM; Richard Zou, Meta

October 20, 4:20–4:45 p.m. | LL20CD | Breakout Session

This talk presents hardware-agnostic model definitions for vLLM, an approach that separates model logic from hardware execution paths so the same model definition can run across accelerators without forks or per-platform maintenance.

The design relies on compatibility with torch.compile, well-defined extensibility hooks, and isolation from hardware-specific paths. The speakers show how the approach supports Intel Gaudi/HPU and IBM Spyre without hardware-specific modeling code.

Portable PyTorch Across AI Accelerators: A Triton Operator Stack from Eager Mode to vLLM

Yonghua Lin, Beijing Academy of Artificial Intelligence

October 20, 4:55–5:20 p.m. | LL20CD | Breakout Session

This session presents FlagOS, an open source system stack using a Triton-based operator, compiler, and runtime layer for PyTorch.

FlagGems implements PyTorch eager-mode operators and LLM-critical kernels in Triton, while the same operator layer connects to vLLM through the vllm-plugin-fl multi-backend plugin.

The speakers report testing FlagOS on 20+ AI chips and architectures and say it has enabled Day-0 adaptation of models including Qwen3.5, MiniMax-M3, MiniCPM-5, and DeepSeek-V4. They report 5–40% inference performance improvement over original vendor adaptation.

Efficient MoE LLM Inference on Arm with vLLM and OpenVINO

Abhishek Jain, Fujitsu Research of India; N Maajid Khan, Fujitsu Research of India

October 20, 4:55–5:05 p.m. | LL21ABC | Lightning Talk

This talk presents a vLLM and OpenVINO inference stack optimized for Arm CPUs, including SVE-optimized SDPA and Paged Attention, U8 KV-cache quantization, operator fusion, KleidiAI integration, and optimized threading for 8-bit and 4-bit inference.