NVIDIA发布Alpamayo 2 Edge:面向Thor的2B推理VLA
NVIDIA Releases Alpamayo 2 Edge: A 2B Reasoning VLA for DRIVE Thor
自动驾驶端侧大模型的重要突破,详细披露了从架构到部署优化的完整技术细节,对做车端Agent的同学极具参考价值。
NVIDIA has released Alpamayo 2 Edge, a compact reasoning vision-language-action (VLA) model for autonomous driving. Alpamayo 2 Edge is a 2B-parameter model that plans trajectories and explains them inside the vehicle. It targets NVIDIA DRIVE AGX Thor, not a datacenter GPU. The goal: reasoning-based planning within onboard compute and latency budgets.
NVIDIA 发布了 Alpamayo 2 Edge,这是一款用于自动驾驶的紧凑型推理视觉-语言-动作(VLA)模型。Alpamayo 2 Edge 是一个拥有 20 亿参数的模型,能够在车内进行轨迹规划并解释这些轨迹。它针对的是 NVIDIA DRIVE AGX Thor,而非数据中心 GPU。目标:在车载计算能力和延迟预算内实现基于推理的规划。
TL;DR
TL;DR(太长不看版)
- Size: 2B-class. Card breakdown: 2.45B backbone + 0.2B action expert (HF metadata says 3B). Context length: not disclosed (measured prompt: 3,236 tokens).
- Runs on: NVIDIA DRIVE AGX Thor with the FP8 + TensorRT Edge-LLM recipe. BF16 PyTorch needs 16 GB+ GPU memory, validated on 1x H100 80GB.
- Performance: Strong closed-loop driver for its size; mid-pack on reasoning QA.
- Best: 15.15 km per at-fault incident in AlpaSim, about 60x Qwen-Drive-1.0 4B (0.25 km).
- Worst: nuReasoning numeric score of 48.67; LingoQA 75.2 trails Alpamayo 2 Super (79.2).
- Bottom line: Best: reasoning-based planning at 160 to 190 ms on in-car hardware. Worst: only H100 validated, and real-time Thor speed needs the FP8 recipe.
- 规模:20 亿级别。参数拆解:24.5 亿骨干网络 + 2 亿动作专家(Hugging Face 元数据显示为 30 亿)。上下文长度:未公开(测量提示词:3,236 个 token)。
- 运行环境:搭载 FP8 + TensorRT Edge-LLM 配方方案的 NVIDIA DRIVE AGX Thor。BF16 PyTorch 需要 16 GB+ 的 GPU 显存,已在 1x H100 80GB 上验证通过。
- 性能:在其体量下表现出色的闭环驾驶能力;在推理问答方面处于中游水平。
- 最佳表现:在 AlpaSim 中,每起有责事故行驶 15.15 公里,约为 Qwen-Drive-1.0 4B(0.25 公里)的 60 倍。
- 最差表现:nuReasoning 数值得分为 48.67;LingoQA 得分 75.2,落后于 Alpamayo 2 Super(79.2)。
- 结论:最佳表现:在车载硬件上实现 160 至 190 毫秒的基于推理的规划。最差表现:仅在 H100 上验证过,且要在 Thor 上实现实时速度需采用 FP8 配方方案。
What is Alpamayo 2 Edge?
什么是 Alpamayo 2 Edge?
Alpamayo 2 Edge is an open NVIDIA VLA model that turns surround-camera video into a 6.4-second plan. It is the in-car member of the Alpamayo family, next to the 34B cloud model Alpamayo 2 Super.
Alpamayo 2 Edge 是一款开源的 NVIDIA VLA 模型,可将环视摄像头视频转化为 6.4 秒的规划方案。它是 Alpamayo 家族的车载成员,与 340 亿参数的云端模型 Alpamayo 2 Super 并列。
Per the model card, the 2.45B backbone pairs a 1.96B language model with a 0.41B SigLIP2 encoder. A 0.2B diffusion action expert sits on top. The backbone comes from NVIDIA Cosmos 3 Edge and is Nemotron-based. NVIDIA markets the model as 2B, while Hugging Face metadata shows 3B.
根据模型卡片,24.5 亿的骨干网络由一个 19.6 亿的语言模型和一个 4,100 万的 SigLIP2 编码器组成。顶部叠加了一个 2 亿参数的扩散动作专家。该骨干网络源自 NVIDIA Cosmos 3 Edge,基于 Nemotron 架构。NVIDIA 将该模型宣传为 20 亿参数,而 Hugging Face 元数据显示为 30 亿。
What goes in and what comes out?
输入和输出是什么?
For trajectory prediction, the model reads 6 cameras and 22 frames per observation. The 2 front cameras give 5 frames each, the other 4 give 3 each. It also takes a navigation instruction and 20 historical ego poses.
对于轨迹预测,模型读取 6 个摄像头和每次观测的 22 帧图像。其中 2 个前置摄像头各提供 5 帧,其余 4 个各提供 3 帧。它还接收导航指令和 20 个历史自车位姿。
Output is 64 waypoints from 0.1 to 6.4 seconds. The action expert predicts acceleration and curvature, then a unicycle model integrates them. The text output is a Chain-of-Causation (CoC) reasoning trace. The model also answers free-form questions about a driving scene.
输出包括从 0.1 秒到 6.4 秒的 64 个航点。动作专家预测加速度和曲率,然后通过单轮模型进行积分。文本输出是因果链(CoC)推理轨迹。该模型还能回答关于驾驶场景的自由形式问题。
How does adaptive reasoning work?
自适应推理是如何工作的?
During routine driving, the model emits special <NOMINAL> tokens instead of a full reasoning trace. It writes CoC reasoning only when a scene needs it. Fewer tokens means less compute per planning step.
在常规驾驶期间,模型会发出特殊的 <NOMINAL> token,而不是完整的推理轨迹。仅当场景需要时,它才会写入 CoC 推理。更少的 token 意味着每次规划步骤的计算量更少。
How was Alpamayo 2 Edge trained?
Alpamayo 2 Edge 是如何训练的?
NVIDIA team lists these ingredients:
NVIDIA 团队列出了以下训练要素:
- Offline distillation from CoC autolabels generated by Alpamayo 2 Super.
- About 115,000 hours of multi-camera driving video and about 3.7M CoC traces.
- Closed-loop RL post-training in AlpaGym, learning from its own actions in simulation.
- Multi-resolution inputs for cheaper encoding of surround views.
- 由 Alpamayo 2 Super 生成的 CoC 自动标签进行的离线蒸馏。
- 约 11.5 万小时的多摄像头驾驶视频和约 370 万条 CoC 轨迹。
- 在 AlpaGym 中进行闭环强化学习后训练,从其在模拟环境中的自身行为中学习。
- 多分辨率输入,以更低的成本编码环视视图。
How fast does it run on DRIVE AGX Thor?
它在 DRIVE AGX Thor 上运行速度有多快?
Hugging Face weights ship in BF16 and run as a PyTorch model. Real-time use on Thor needs the Alpamayo Developer Hub recipe. It stages the checkpoint and the DFlash2 drafter, then calibrates FP8 scales. Next it exports ONNX graphs and builds TensorRT engines with TensorRT Edge-LLM plugins.
Hugging Face 权重以 BF16 格式提供,并作为 PyTorch 模型运行。在 Thor 上进行实时使用需要 Alpamayo Developer Hub 配方。该配方会暂存检查点和 DFlash2 drafter,然后校准 FP8 缩放比例。接下来导出 ONNX 图,并使用 TensorRT Edge-LLM 插件构建 TensorRT 引擎。
NVIDIA reports 160 ms to 190 ms pixel-to-trajectory latency, depending on how much the model reasons. The reference setup is DRIVE AGX Thor-X with DRIVE OS 7.2.7.0, CUDA 13.4 and TensorRT 11.3.1.9. In NVIDIA’s demo video, PyTorch BF16 plans at 0.76 Hz. The optimized FP8 Thor pipeline plans at 13.02 Hz in AlpaSim. Partner Embedl says it reached 10 Hz on AGX Thor in under 48 hours.
NVIDIA 报告像素到轨迹的延迟为 160 毫秒至 190 毫秒,具体取决于模型的推理程度。参考设置是搭载 DRIVE OS 7.2.7.0、CUDA 13.4 和 TensorRT 11.3.1.9 的 DRIVE AGX Thor-X。在 NVIDIA 的演示视频中,PyTorch BF16 的规划频率为 0.76 Hz。优化后的 FP8 Thor 管道在 AlpaSim 中的规划频率为 13.02 Hz。合作伙伴 Embedl 表示,它在不到 48 小时内就在 AGX Thor 上达到了 10 Hz。
On a workstation GPU, the card recommends at least 16 GB of memory. NVIDIA measured 12.06 GiB peak process memory on 1 H100 with 6 trajectory candidates. Other GPU architectures are not yet validated.
在工作站 GPU 上,显卡建议至少配备 16 GB 内存。NVIDIA 在单张 H100 上使用 6 个轨迹候选者时测得峰值进程内存为 12.06 GiB。其他 GPU 架构尚未验证。
How does Alpamayo 2 Edge score on benchmarks?
Alpamayo 2 Edge 在基准测试中的表现如何?
NVIDIA compares Edge with Alibaba’s 4B Qwen-Drive-1.0. In AlpaSim closed-loop driving, Edge covers 15.15 km between at-fault incidents versus 0.25 km. That is about 60x. Open-loop minADE is 0.53 m versus 1.68 m. Both are NVIDIA’s own runs.
NVIDIA 将 Edge 与阿里巴巴的 4B Qwen-Drive-1.0 进行比较。在 AlpaSim 闭环驾驶中,Edge 在发生责任事故之间行驶了 15.15 公里,而后者仅为 0.25 公里。这大约是 60 倍。Open-loop minADE 为 0.53 米,对比 1.68 米。两者均为 NVIDIA 自行运行的结果。
The model card adds:
模型卡片补充道:
- AlpaSim: 15.15 ± 0.04 km per at-fault incident over 875 NuRec scenarios, 3 rollouts.
- Open-loop minADE_6 at 5 s: 0.52 m on 1,392 PhysicalAI-AV samples.
- LingoQA (Lingo-Judge): 75.2.
- SURDS: 86.50, KITScenes scene: 89.46, nuReasoning numeric: 48.67.
- AlpaSim:在 875 个 NuRec 场景、3 次 rollout 中,每次责任事故平均行驶 15.15 ± 0.04 公里。
- 5 秒时的 Open-loop minADE_6:在 1,392 个 PhysicalAI-AV 样本上为 0.52 米。
- LingoQA (Lingo-Judge):75.2。
- SURDS:86.50,KITScenes 场景:89.46,nuReasoning 数值:48.67。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力