Kyutai发布2.4B VLA模型REKI,公开数据预训练带记忆机制
Meet Kyutai REKI: The 2.4B VLA Model Bringing Efficient Memory to Robot Manipulation
做具身智能的同学注意,这个2.4B VLA用极低成本实现了有效记忆,LIBERO-Plus成绩亮眼且代码可复现,值得跟进测试。
Kyutai, with Inria and École des Ponts, has released REKI, a 2.4B vision-language-action model (VLA) for robot manipulation. The REKI vision-language-action model is pretrained only on public data. It also carries a cheap memory of its recent past. On three out-of-distribution benchmarks, only models about twice its size beat it. All weights are live on Hugging Face today.
Kyutai 联合 Inria 和 École des Ponts 发布了 REKI,这是一个用于机器人操作的 24 亿参数视觉-语言-动作模型(VLA)。REKI 视觉-语言-动作模型仅在公开数据上进行预训练。它还携带着对近期过去的廉价记忆。在三个分布外基准测试中,只有规模约为其两倍的模型超越了它。所有权重今天已在 Hugging Face 上开源。
TL;DR
TL;DR(太长不看版)
- Size: 2.4B parameters (2.2B Qwen3.5-2B VLM + 0.2B flow-matching action expert). Memory covers the last 3 frames, 1 s apart.
- Runs on: Benchmarked on 1 NVIDIA H100. 21.8 ms per action chunk, compiled, from 2 cameras at 256×256. Weights ship in BF16. No quantized variants announced.
- Performance: Best among VLAs pretrained on public data on LIBERO-Plus and RoboTwin Clean2Rand.
- Best: 90.4% on LIBERO-Plus, the top score in Kyutai’s comparison.
- Worst: 38.4% on RoboCasa365, well behind Xiaomi-Robotics-1 (57.4%).
- 规模:24 亿参数(22 亿 Qwen3.5-2B VLM + 2 亿流匹配动作专家)。记忆覆盖最后 3 帧,间隔 1 秒。
- 运行环境:在 1 块 NVIDIA H100 上进行基准测试。编译后,从 2 个 256×256 分辨率的摄像头获取每个动作块耗时 21.8 毫秒。权重以 BF16 格式发布。未宣布量化变体。
- 性能:在 LIBERO-Plus 和 RoboTwin Clean2Rand 上,基于公开数据预训练的 VLA 中表现最佳。
- 最佳成绩:LIBERO-Plus 上达到 90.4%,这是 Kyutai 对比中的最高分。
- 最差成绩:RoboCasa365 上为 38.4%,远落后于 Xiaomi-Robotics-1(57.4%)。
What is REKI?
什么是 REKI?
REKI is a vision-language-action model from Kyutai that maps camera views and an instruction to robot actions. It pairs a Qwen3.5-2B backbone with a 200M-parameter action expert. The expert generates action chunks by flow matching, integrated in 10 Euler steps. Robot state and flow time enter through adaptive RMSNorm. Each robot state layout gets its own small state MLP.
REKI 是 Kyutai 推出的一个视觉-语言-动作模型,它将相机视图和指令映射为机器人动作。它由一个 Qwen3.5-2B 主干网络和一个 2 亿参数的动作专家配对组成。该专家通过流匹配生成动作块,并在 10 个欧拉步长内集成。机器人状态和流时间通过自适应 RMSNorm 输入。每个机器人状态布局都有自己独立的小型状态 MLP。
How does REKI’s memory work?
REKI 的记忆机制如何工作?
Most VLAs act on the current frame alone. Prior memory methods feed past frames to the VLM, unroll training, or add modules. Each option adds cost.
大多数 VLA 仅基于当前帧进行操作。先前的记忆方法将过去帧馈送到 VLM、展开训练或添加模块。每种选项都会增加成本。
REKI places memory inside the action expert instead. Before each frame’s action tokens sit 32 learned register tokens. They cross-attend the encoded prompt and views of that frame. They also self-attend the registers of earlier frames.
REKI 将记忆放置在动作专家内部。在每个帧的动作令牌之前放置了 32 个学习到的寄存器令牌。它们对该帧的编码提示和视图进行交叉注意力计算。它们还对早期帧的寄存器进行自注意力计算。
Registers never read action tokens. So they depend only on observations, not flow time or robot state. They are computed once per frame and cached in a rolling window.
寄存器从不读取动作令牌。因此,它们仅依赖于观测值,而不依赖于流时间或机器人状态。它们每帧计算一次并缓存在滑动窗口中。
The cost is small, per the project page:
根据项目页面,成本很小:
- Reading cached registers adds 0.1 ms to a 21.8 ms chunk.
- The cache uses about 2 MiB of GPU memory per past frame.
- Training packs a 3-frame window into 1 batched pass.
- 读取缓存寄存器使 21.8 毫秒的动作块增加 0.1 毫秒。
- 每个过去帧的缓存使用约 2 MiB 的 GPU 内存。
- 训练将 3 帧窗口打包到一个批处理传递中。
Memory also fixes a failure mode. Without it, arms often oscillate between 2 poses. On RoboTwin, oscillating episodes fall from 18.6% to 4.1% with memory. Kyutai attributes this to perceptual aliasing.
记忆还修复了一种失败模式。没有记忆时,机械臂经常在两种姿态之间振荡。在 RoboTwin 上,加入记忆后,振荡 episode 的比例从 18.6% 下降到 4.1%。Kyutai 将此归因于感知歧义。
How does REKI perform on benchmarks?
REKI 在基准测试中的表现如何?
REKI is pretrained once, then finetuned separately per benchmark. Scores below come from the model card and project page.
REKI 仅进行一次预训练,然后针对每个基准测试单独微调。以下分数来自模型卡片和项目页面。
- LIBERO-Plus: 90.4%, ahead of Qwen-RobotManip (89.0%) and π0.5 (85.7%).
- RoboTwin Clean2Rand: 65.8%, behind only Qwen-RobotManip (68.5%).
- RoboCasa365: 38.4%, ahead of RLDX-1 (36.0%) and Qwen-RobotManip (35.9%).
- LIBERO (4 suites): 97.6%.
- RoboTwin 2.0 Easy / Hard: 85.9% / 84.6%.
- LIBERO-Plus:90.4%,领先于 Qwen-RobotManip(89.0%)和 π0.5(85.7%)。
- RoboTwin Clean2Rand:65.8%,仅落后于 Qwen-RobotManip(68.5%)。
- RoboCasa365:38.4%,领先于 RLDX-1(36.0%)和 Qwen-RobotManip(35.9%)。
- LIBERO(4 个套件):97.6%。
- RoboTwin 2.0 Easy / Hard:85.9% / 84.6%。
Pretraining helps most out of distribution. It adds 13.8 points on Clean2Rand and 30.3 under clutter. Memory adds 1.5 points on LIBERO-Plus and 5.4 on RoboCasa365.
预训练在分布外场景下帮助最大。它在 Clean2Rand 上提升了 13.8 分,在杂乱环境下提升了 30.3 分。记忆模块在 LIBERO-Plus 上提升了 1.5 分,在 RoboCasa365 上提升了 5.4 分。
How does REKI compare with other VLAs?
REKI 与其他 VLA 相比表现如何?
| Feature | REKI (Kyutai) | Qwen-RobotManip (Qwen) | π0.5 (Physical Intelligence) | ABot-M0 (AMAP) |
|---|---|---|---|---|
| Total size | 2.4B | 4.7B (per Kyutai) | 3.3B (per Kyutai) | 4.6B (per Kyutai) |
| VLM backbone | Qwen3.5-2B | Qwen3.5-4B | Gemma-based | Not disclosed |
| Pretraining data | Public only | Open-source + human video | Includes own robot data | Open-source (UniACT) |
| Memory | 32 registers, 3 frames | In-context variant | None | None |
| LIBERO-Plus | 90.4 | 89.0 | 85.7 (Kyutai eval) | 80.5 |
| RoboTwin Clean2Rand (mean) | 65.8 | 68.5 | 63.8 | 55.7 |
| RoboCasa365 | 38.4 | 35.9 | 16.9 | Not disclosed |
| H100 latency, compiled | 21.8 ms | Not disclosed | 33.8 ms (openpi, Kyutai) | 46.1 ms (Kyutai) |
| Weights | Open, CC BY-NC-SA 4.0 | Not released | Open, Apache-2.0 repo | Open, Apache-2.0 repo |
| 特性 | REKI (Kyutai) | Qwen-RobotManip (Qwen) | π0.5 (Physical Intelligence) | ABot-M0 (AMAP) |
|---|---|---|---|---|
| 总参数量 | 24亿 | 47亿(据 Kyutai) | 33亿(据 Kyutai) | 46亿(据 Kyutai) |
| VLM 骨干网络 | Qwen3.5-2B | Qwen3.5-4B | 基于 Gemma | 未公开 |
| 预训练数据 | 仅限公开数据 | 开源 + 人类视频 | 包含自有机器人数据 | 开源(UniACT) |
| 记忆模块 | 32 个寄存器,3 帧 | 上下文学习变体 | 无 | 无 |
| LIBERO-Plus | 90.4 | 89.0 | 85.7(Kyutai 评估) | 80.5 |
| RoboTwin Clean2Rand(均值) | 65.8 | 68.5 | 63.8 | 55.7 |
| RoboCasa365 | 38.4 | 35.9 | 16.9 | 未公开 |
| H100 延迟,编译后 | 21.8 毫秒 | 未公开 | 33.8 毫秒(openpi, Kyutai) | 46.1 毫秒(Kyutai) |
| 权重许可 | 开放,CC BY-NC-SA 4.0 | 未发布 | 开放,Apache-2.0 仓库 | 开放,Apache-2.0 仓库 |
Notes: Clean2Rand means are computed from the 6 conditions in Table 5 of the Qwen-RobotManip report. The Qwen report lists π0.5 at 84.4 on LIBERO-Plus.
注:Clean2Rand 的均值是根据 Qwen-RobotManip 报告中表 5 的 6 个条件计算得出的。Qwen 报告列出 π0.5 在 LIBERO-Plus 上的得分为 84.4。
What data was REKI trained on?
REKI 使用了哪些数据进行训练?
The mixture holds 1.0B frames plus 12.2M vision-language samples. Real-robot teleoperation provides 639.8M frames, simulation 193.9M and egocentric human video 170.9M. Sources include DROID, BridgeData V2, AgiBot World, EgoDex and EgoVerse. Pretraining ran 200,000 steps on 32 H100 GPUs. Kyutai keeps each dataset’s native control convention rather than remapping actions.
该混合数据集包含 10 亿帧图像以及 1220 万视觉-语言样本。真实机器人遥操作提供 6.398 亿帧,仿真环境提供 1.939 亿帧,第一人称人类视频提供 1.709 亿帧。数据来源包括 DROID、BridgeData V2、AgiBot World、EgoDex 和 EgoVerse。预训练在 32 块 H100 GPU 上运行了 20 万步。Kyutai 保留了每个数据集的原生控制约定,而非重新映射动作。
How do you deploy REKI?
如何部署 REKI?
Six checkpoints are listed: the pretrained base plus finetunes for LIBERO, RoboTwin (2 variants), RoboCasa365 and RoboCasa-GR1. Real-robot deployment uses the LeRobot async-inference gRPC protocol. On a UR5 arm, REKI was finetuned for 3k steps (1 hour) on 50 demonstrations per task.
列出了六个检查点:预训练基础模型,以及针对 LIBERO、RoboTwin(2 种变体)、RoboCasa365 和 RoboCasa-GR1 的微调版本。真实机器人部署使用 LeRobot 异步推理 gRPC 协议。在 UR5 机械臂上,REKI 针对每项任务的 50 个演示进行了 3000 步(1 小时)的微调。
from reki import RekiPolicy
policy = RekiPolicy.from_pretrained("kyutai/reki-2b-libero", device="cuda")
policy.reset() # clears the register cache at episode start
actions = policy.predict(images={"camera_top": img0, "camera_wrist_left": img1},
state=proprio, task="put the bowl on the plate")Key Takeaways
关键要点
- REKI is a 2.4B VLA pretrained only on public data.
- It scores 90.4% on LIBERO-Plus, top in Kyutai’s comparison.
- 32 cached register tokens give memory for 0.1 ms extra.
- Memory cuts pose oscillations from 18.6% to 4.1% on RoboTwin.
- Weights are open but non-commercial (CC BY-NC-SA 4.0).
- REKI 是一个仅使用公开数据预训练的 24 亿参数 VLA。
- 它在 LIBERO-Plus 上得分 90.4%,在 Kyutai 的比较中位居榜首。
- 32 个缓存寄存器令牌提供了额外 0.1 毫秒的记忆能力。
- 在 RoboTwin 上,记忆模块将姿态振荡从 18.6% 降低至 4.1%。
- 权重是开放的,但仅限非商业用途(CC BY-NC-SA 4.0)。
Check out the Paper, Project Page, Model Weights and GitHub Repo. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
查看论文、项目页面、模型权重和 GitHub 仓库。所有功劳归于该项目的研究人员。此外,欢迎在 Twitter 上关注我们,别忘了加入我们拥有 15 万+成员的 ML SubReddit 并订阅我们的新闻通讯。等等!你还在 Telegram 上吗?现在你也可以加入我们的 Telegram 群组了。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力