Code-as-World：将真实视频重写为可执行的 MuJoCo 物理程序
Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs
做视频理解或世界模型的同学值得细读：这套把视频转成可执行物理程序、再用验证世界做训练数据的方法，思路新颖且开源可复现，建议对照技术报告跑一遍。
MirroS released Code-as-World: a paradigm that represents physical worlds through executable world representations. The argument is narrow and testable: pixels are evidence of a physical scene, not its ontology. A video model can predict plausible frames without ever representing mass, contact, or gravity. So instead of pixels, latents, or captions, Code-as-World represents a scene as executable code — a scene.json that MuJoCo can run, that an agent can verify against the source video, and that anyone can edit and re-simulate. An agentic loop recovers those programs from real footage in up to five rounds. The verified worlds then become training data with exact physical labels, which real video does not carry. Trained on that supervision, Code-as-World-VL-9B scores 55.4 MRA on QuantiPhy-validation, above Gemini-3.1 Flash at 54.8 and roughly 15 points above the strongest open-weight baseline.
MirroS 发布了 Code-as-World：一种通过可执行的世界表示来代表物理世界的范式。其论点狭窄且可测试：像素是物理场景的证据，而非其本体。视频模型可以预测合理的帧，而无需表示质量、接触或重力。因此，Code-as-World 不使用像素、潜在变量或字幕，而是将场景表示为可执行代码——一个 scene.json，MuJoCo 可以运行它，代理可以对照源视频进行验证，任何人都可以编辑并重新模拟。一个代理循环从真实镜头中恢复这些程序，最多五轮。验证后的世界成为带有精确物理标签的训练数据，而真实视频并不携带这些标签。在这种监督下训练的 Code-as-World-VL-9B 在 QuantiPhy-validation 上得分为 55.4 MRA，高于 Gemini-3.1 Flash 的 54.8，并且比最强的开放权重基线高出约 15 分。
Is it deployable?
它是否可部署？
Yes, at the research and internal-prototype tier. MirroS shipped the GitHub repo and two checkpoints — Code-as-World-VL-4B and Code-as-World-VL-9B — under Apache 2.0, fine-tuned from Qwen3.5-4B and Qwen3.5-9B. Both are BF16 safetensors served by vLLM behind an OpenAI-compatible /v1 endpoint, with 16 sampled frames per video and --max-model-len 4608.
是的，在研究和内部原型层面。MirroS 发布了 GitHub 仓库和两个检查点——Code-as-World-VL-4B 和 Code-as-World-VL-9B——采用 Apache 2.0 许可，基于 Qwen3.5-4B 和 Qwen3.5-9B 微调。两者都是 BF16 safetensors，由 vLLM 在 OpenAI 兼容的 /v1 端点后提供服务，每个视频采样 16 帧，--max-model-len 为 4608。
The idea: pixels are evidence, not ontology
核心思想：像素是证据，而非本体
The MirroS technical report argues that video models, 3D reconstruction, and captions each recover part of a scene but none recovers its mechanism. Code-as-World represents a scene as an executable world representation (EWR), a triple p = (C, E, A):
MirroS 技术报告认为，视频模型、3D 重建和字幕各自恢复了场景的一部分，但都没有恢复其机制。Code-as-World 将场景表示为可执行的世界表示（EWR），即三元组 p = (C, E, A)：
- Composition: objects, geometry, metric dimensions, mass, friction, gravity. Floors and walls are static physical entities so they can support and collide.
- Evolution: initial states, forces, contacts, collisions, termination conditions, duration. Executing it expands composition into a full state trajectory.
- Appearance: camera, lighting, materials, background, frame rate, render config. Changing it never changes the physics.
- 组成：物体、几何、度量尺寸、质量、摩擦、重力。地板和墙壁是静态物理实体，因此它们可以支撑和碰撞。
- 演化：初始状态、力、接触、碰撞、终止条件、持续时间。执行它将组成扩展为完整的状态轨迹。
- 外观：相机、光照、材质、背景、帧率、渲染配置。改变它永远不会改变物理。
In the released implementation, that triple compiles into a scene.json executed in MuJoCo, with two interchangeable engines: an animation engine (kinematic poses) and a physics engine (forces and contacts).
在发布的实现中，该三元组编译为在 MuJoCo 中执行的 scene.json，具有两个可互换的引擎：动画引擎（运动学姿态）和物理引擎（力和接触）。
Agentic discovery instead of one-shot prediction
代理式发现而非一次性预测
Recovering an EWR from a video is an inverse problem, so the team frames it as abductive search. An agent runs propose → instantiate → execute → render → verify for up to K = 5 rounds. For video input, SAM 3 supplies instance masks and image-plane tracks, VGGT-Omega estimates depth and camera geometry, and SAM 3D generates per-object meshes. Candidate rollouts are projected back into the input view and compared at selected key frames on RGB, depth, masks, and trajectories. Frame-level discrepancies aggregate into structured feedback Δ that guides the next revision; when the budget runs out without acceptance, the hypothesis is rejected.
从视频中恢复EWR是一个逆问题，因此团队将其框架化为溯因搜索。代理执行提出→实例化→执行→渲染→验证，最多进行K=5轮。对于视频输入，SAM 3提供实例掩码和图像平面轨迹，VGGT-Omega估计深度和相机几何，SAM 3D生成每个对象的网格。候选滚动结果被投影回输入视图，并在选定的关键帧上比较RGB、深度、掩码和轨迹。帧级差异聚合成结构化反馈Δ，指导下一次修订；当预算耗尽而未接受时，假设被拒绝。
At a matched five-evaluation budget, the loop beats Best-of-5 independent sampling on Visual Alignment, Object IoU, Traj-ADE, and Accuracy@2%D — and the result repeats under the second execution engine. Candidate videos come from WISA-80K after motion-focused filtering; sim-to-real re-rendering uses Wan2.2-VACE plus an internal video model.
在匹配的五次评估预算下，该循环在视觉对齐、对象IoU、轨迹ADE和Accuracy@2%D上优于Best-of-5独立采样——并且在第二个执行引擎下重复了结果。候选视频来自WISA-80K经过运动聚焦过滤；模拟到现实的重新渲染使用Wan2.2-VACE加上内部视频模型。
Verified worlds as training supervision
验证世界作为训练监督
Phase 1 is supervised fine-tuning on 73,335 image-space QA pairs built from RefCOCO/+/g, RefCLEF and GOT-10K, covering extent, position, displacement, velocity and acceleration in raw pixels. Phase 2 applies GRPO to world-space VQA drawn from 1,585 text-driven and 988 video-driven executable worlds, rewarded on scale-normalized numerical accuracy plus unit and format terms. Training used eight NVIDIA H100 GPUs.
阶段1是在73,335个图像空间QA对上进行的监督微调，这些QA对来自RefCOCO/+/g、RefCLEF和GOT-10K，涵盖原始像素中的范围、位置、位移、速度和加速度。阶段2应用GRPO到从1,585个文本驱动和988个视频驱动的可执行世界中抽取的世界空间VQA，奖励基于尺度归一化的数值准确性以及单位和格式项。训练使用了八块NVIDIA H100 GPU。
On QuantiPhy-validation (159 items, MRA macro-averaged over 2S/2D/3S/3D): 4B = 50.6, 9B = 55.4, 27B reasoning = 58.6, against Gemini-3.1 Flash at 54.8, ChatGPT-5.1 at 48.4, and the strongest open-weight baseline Qwen3-VL-32B-Instruct at 40.2. The ablation is the more useful number: image-space-only scores 44.2 (4B) and 50.9 (9B); adding both world-space sources lifts them to 50.6 and 55.4. Pixel-level grounding improves too — the 9B goes 63.7 → 68.3 on RefCOCO and 20.1 → 26.6 on GOT-10K after world-space RL.
在QuantiPhy验证集（159项，MRA在2S/2D/3S/3D上宏平均）上：4B=50.6，9B=55.4，27B推理=58.6，对比Gemini-3.1 Flash的54.8，ChatGPT-5.1的48.4，以及最强的开源权重基线Qwen3-VL-32B-Instruct的40.2。消融实验更有用：仅图像空间得分为44.2（4B）和50.9（9B）；加入两个世界空间源后提升至50.6和55.4。像素级定位也得到改善——9B在RefCOCO上从63.7提升至68.3，在GOT-10K上从20.1提升至26.6，经过世界空间RL后。
Key Takeaways
关键要点
- Code-as-World turns a video into an editable scene.json that MuJoCo can execute and verify.
- Five-round propose→verify search beats Best-of-5 sampling at the same compute budget.
- Verified worlds supply exact physical labels that real video simply does not carry.
- 9B hits 55.4 MRA on QuantiPhy, above Gemini-3.1 Flash at 54.8; 4B and 9B are Apache 2.0.
- Rigid-body only, and the model never learns the discovery loop itself.
- 代码即世界将视频转换为可编辑的scene.json，MuJoCo可以执行并验证。
- 五轮提出→验证搜索在相同计算预算下优于Best-of-5采样。
- 验证世界提供了真实视频无法携带的精确物理标签。
- 9B在QuantiPhy上达到55.4 MRA，高于Gemini-3.1 Flash的54.8；4B和9B采用Apache 2.0许可证。
- 仅限刚体，且模型从未学习发现循环本身。
Check out the Technical report, MirroS blog, Project page, GitHub and Announcement on X. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
查看技术报告、MirroS博客、项目页面、GitHub和X上的公告。此外，欢迎在Twitter上关注我们，别忘了加入我们的15万+ML SubReddit社区并订阅我们的新闻通讯。等等！你在Telegram上吗？现在你也可以在Telegram上加入我们。
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
需要与我们合作推广你的GitHub仓库、Hugging Face页面、产品发布或网络研讨会等吗？请联系我们。
The post Meet ‘Code-as-World’: An Agentic Loop That Rewrites Real Videos Into Executable MuJoCo Physics Programs appeared first on MarkTechPost.
文章《认识‘代码即世界’：将真实视频重写为可执行MuJoCo物理程序的代理循环》首次出现在MarkTechPost上。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力