JEPA-Anything:用正交预测因子分解统一7类世界模型
Beyond Domain-Specific World Models: JEPA-Anything Uses 1 Recipe for 7 Fields
这篇论文展示了JEPA架构在多模态领域的通用化突破,正交因子分解思路对提升世界模型稳定性有直接参考价值,值得关注。
Researchers from PhAI Labs, CUHK, Fudan, Stanford, Oxford and Princeton have released JEPA-Anything, a domain-agnostic framework for building world models. Instead of designing a new predictive model for each field, it applies one shared learning recipe to very different systems. It extends joint-embedding predictive architectures (JEPAs) with a method called Orthogonal Predictive Factorization (OPF). The research team tested it across 7 domains: vision, biology, clinical trajectories, control, molecular dynamics, physical fields and weather.
来自 PhAI Labs、香港中文大学、复旦大学、斯坦福大学、牛津大学和普林斯顿大学的研究人员发布了 JEPA-Anything,这是一个用于构建世界模型的领域无关框架。它没有为每个领域设计新的预测模型,而是将一种共享的学习配方应用于截然不同的系统。它通过一种称为正交预测因子分解(Orthogonal Predictive Factorization, OPF)的方法扩展了联合嵌入预测架构(JEPAs)。研究团队在 7 个领域对其进行了测试:视觉、生物学、临床轨迹、控制、分子动力学、物理场和天气。
What problem does JEPA-Anything solve?
JEPA-Anything 解决了什么问题?
A standard JEPA, such as I-JEPA or V-JEPA 2, uses a context encoder, an EMA target encoder and one predictor. The predictor outputs one monolithic target embedding. The research team call this a capacity-allocation problem: high-variance structure dominates, and weaker modes get conflicting gradients.
标准的 JEPA,例如 I-JEPA 或 V-JEPA 2,使用上下文编码器、EMA 目标编码器和一个预测器。预测器输出一个单一的单体目标嵌入。研究团队将此称为容量分配问题:高方差结构占主导地位,而较弱的模式获得冲突的梯度。
How does Orthogonal Predictive Factorization work?
正交预测因子分解是如何工作的?
OPF splits the latent target of width d into K learned subspaces of width r, with d = K × r. Most experiments use K = 4. Each factor gets a dedicated predictor. The factor predictions are then recombined through the Moore-Penrose pseudoinverse of the projector matrix. The result is 1 complete latent state for decoding, planning or rollout.
OPF 将宽度为 d 的潜在目标拆分为 K 个宽度为 r 的学习子空间,满足 d = K × r。大多数实验使用 K = 4。每个因子都有一个专用的预测器。然后,通过投影矩阵的 Moore-Penrose 伪逆将因子预测重新组合。结果是 1 个完整的潜在状态,用于解码、规划或 rollout。
Three regularizers keep the factors useful:
三个正则化项保持因子的有用性:
- Orthogonality loss: keeps columns within each projector orthonormal and different projectors in non-overlapping subspaces.
- Factor-activity loss: a hinge on per-coordinate standard deviation so no factor goes dead.
- Encoder-variance loss: sends a direct anti-collapse signal to the online encoder.
- 正交损失:保持每个投影器内的列相互正交且归一化,并使不同的投影器位于不重叠的子空间中。
- 因子活性损失:对每坐标的标准偏差设置铰链约束,以防止任何因子失效。
- 编码器方差损失:向在线编码器发送直接的抗坍塌信号。
The OPF loss is simply added to each domain’s original training loss. Domain adapters handle tokenization and encoders; the core library exposes the shared core as OrthogonalFactorProjection.
OPF 损失简单地添加到每个领域的原始训练损失中。领域适配器处理标记化和编码器;核心库将共享核心暴露为 OrthogonalFactorProjection。
Orthogonality matters for stable synthesis. On CITRIS Interventional Pong, a capacity-matched unconstrained multi-head model had a condition number of 438.52. The orthogonal version reached 1.00005, with cross-factor overlap near zero.
正交性对于稳定合成至关重要。在 CITRIS Interventional Pong 上,一个容量匹配的无约束多头模型的条件数为 438.52。正交版本达到了 1.00005,跨因子重叠接近零。
What results does the research report?
研究报告了什么结果?
Group I, terminal readout: On single-cell data, zero-shot PBMC clustering (AvgBIO) rose to 0.7752 versus 0.7194 for Cell-JEPA. Norman perturbation Pearson rose from 0.787 to 0.814. For forecasting over 1,000 clinical events on UK Biobank data, mean PRAUC was 0.718 versus 0.711 for the matched standard JEPA.
第一组,终端读取:在单细胞数据上,零样本 PBMC 聚类(AvgBIO)从 Cell-JEPA 的 0.7194 上升到 0.7752。Norman 扰动 Pearson 相关系数从 0.787 上升到 0.814。对于在英国生物银行数据上预测超过 1,000 个临床事件,平均 PRAUC 为 0.718,而匹配的标准化 JEPA 为 0.711。
Group II, latent world dynamics: On Interventional Pong, single-intervention MSE fell 34.83%. Unseen combined interventions improved 12.90%, and 6-step free rollout improved 8.58%. JEPA-Anything improved reported metrics on all 10 matched dynamics tasks. Benchmarks include CausalWorld, DeepMind Control, PDEBench and WeatherBench2. On APEBench Burgers, 6-step rollout error dropped about 44.7%, improving in every seed. For 100-step molecular rollouts with a TrajCast-style backbone, it posted the lowest MAE and RMSD on water, quartz, paracetamol and benzene.
第二组,潜在世界动力学:在干预性 Pong 任务中,单次干预的均方误差(MSE)下降了 34.83%。未见过的组合干预提升了 12.90%,6 步自由展开(free rollout)提升了 8.58%。JEPA-Anything 在所有 10 个匹配的动力学任务上改善了报告的指标。基准测试包括 CausalWorld、DeepMind Control、PDEBench 和 WeatherBench2。在 APEBench Burgers 任务中,6 步展开误差降低了约 44.7%,在每个随机种子下均有改善。对于具有 TrajCast 风格主干网络的 100 步分子展开,它在 H2O、石英、对乙酰氨基酚和苯上取得了最低的 MAE 和 RMSD。
Planning results are mixed. With parameters matched within 0.3%, JEPA-Anything improved CEM return on Walker2d and HalfCheetah. Hopper favored standard JEPA.
规划结果喜忧参半。在参数匹配度在 0.3% 以内的情况下,JEPA-Anything 提高了 Walker2d 和 HalfCheetah 上的 CEM 回报。Hopper 则更倾向于标准 JEPA。
Group III, scientific analysis: Factor analysis nominated IL-18 plus CD73 blockade as a cancer intervention. Wet-lab tests supported it in co-cultures, patient-derived organoids, tumor fragments and mice. Latent orbital modes also recovered Kepler’s law with a fitted slope of −1.4991 against the theoretical −1.5.
第三组,科学分析:因子分析提名 IL-18 联合 CD73 阻断作为癌症干预手段。湿实验在共培养体系、患者来源类器官、肿瘤碎片和小鼠模型中支持了这一结论。潜在轨道模式也恢复了开普勒定律,拟合斜率为 -1.4991,与理论值 -1.5 相符。
How does JEPA-Anything compare with other world models?
JEPA-Anything 与其他世界模型相比如何?
| Feature | JEPA-Anything | V-JEPA 2 | DINO-WM | DreamerV3 | TD-MPC2 |
|---|---|---|---|---|---|
| Core idea | JEPA with K orthogonal predictive factors | Video JEPA plus action-conditioned V-JEPA 2-AC | World model on pretrained visual features | World model plus actor-critic trained in imagination | Decoder-free latent dynamics plus MPC |
| Target structure | Factorized, recombined via pseudoinverse | Single latent target | Single latent target | Categorical latent states | Single latent state |
| Domains shown | 7: vision, cells, clinical, control, molecules, PDEs, weather | Video understanding, robot manipulation | PointMaze, PushT, Wall, deformables | Diverse RL domains, fixed hyperparameters | 104 continuous-control tasks, 4 domains |
| Planning / rollout | Latent rollout, CEM planning | Planning from image goals | CEM planning | Policy from imagined rollouts | MPC planning |
| Public checkpoints | Per-domain research checkpoints on HF | Yes, 300M to 1B (V-JEPA 2) | PointMaze, PushT, Wall | Not listed in repo | 300+ checkpoints, up to 317M |
| License | Apache-2.0 | MIT (some files Apache-2.0) | MIT | MIT | MIT |
| 特性 | JEPA-Anything | V-JEPA 2 | DINO-WM | DreamerV3 | TD-MPC2 |
|---|---|---|---|---|---|
| 核心理念 | 带有 K 个正交预测因子的 JEPA | 视频 JEPA 加上动作条件化的 V-JEPA 2-AC | 基于预训练视觉特征的世界模型 | 想象力中训练的 Actor-Critic 世界模型 | 无解码器的潜在动力学加 MPC |
| 目标结构 | 通过伪逆分解并重组 | 单一潜在目标 | 单一潜在目标 | 分类潜在状态 | 单一潜在状态 |
| 展示领域 | 7 个:视觉、细胞、临床、控制、分子、偏微分方程、天气 | 视频理解、机器人操作 | PointMaze, PushT, Wall, 可变形物体 | 多样化的强化学习领域,固定超参数 | 104 个连续控制任务,4 个领域 |
| 规划/展开 | 潜在展开,CEM 规划 | 从图像目标进行规划 | CEM 规划 | 从想象展开中获取策略 | MPC 规划 |
| 公开检查点 | HF 上按领域的研究检查点 | 是,3 亿至 10 亿参数 (V-JEPA 2) | PointMaze, PushT, Wall | 仓库中未列出 | 300+ 检查点,最高 3.17 亿 |
| 许可证 | Apache-2.0 | MIT(部分文件为 Apache-2.0) | MIT | MIT | MIT |
Sources: each project’s GitHub README, linked in the header row. JEPA-Anything checkpoint status from its Hugging Face card. Verified October 5, 2026.
来源:每个项目的 GitHub README,链接位于表头行。JEPA-Anything 检查点状态来自其 Hugging Face 卡片。验证于 2026 年 10 月 5 日。
Key Takeaways
关键要点
- OPF splits 1 JEPA target into K orthogonal factors with dedicated predictors.
- It beat matched JEPA baselines on all 10 dynamics tasks.
- Interventional Pong single-intervention error fell 34.83%.
- Planning gains are environment-dependent; Hopper favored standard JEPA.
- Core code is Apache-2.0; research checkpoints sit on Hugging Face.
- OPF 将 1 个 JEPA 目标拆分为 K 个正交因子,并配备专用预测器。
- 它在所有 10 个动力学任务上都击败了匹配的 JEPA 基线。
- Interventional Pong 单次干预误差下降了 34.83%。
- 规划收益取决于环境;Hopper 偏好标准 JEPA。
- 核心代码采用 Apache-2.0 协议;研究检查点位于 Hugging Face。
Check out the Paper, GitHub Repo and Model Checkpoints. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
查看论文、GitHub 仓库和模型检查点。所有功劳归于本项目的研究者。此外,欢迎在 Twitter 上关注我们,别忘了加入我们拥有 15 万+成员的 ML SubReddit 并订阅我们的新闻通讯。等等!你在 Telegram 上吗?现在你也可以加入我们的 Telegram 群组。
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力