构建低延迟AI游戏伴侣与玩家同玩Skyrim的工程实践
构建低延迟AI伴侣与玩家同玩Skyrim的工程实践
做AI游戏伴侣或实时交互系统的同学必看，这篇给出了完整的架构取舍：本地行为图处理实时路径、云端大模型负责慢速个性演化，还附了可复现的工程细节，值得收藏。
The goal
目标
Simple: Let's build a super-charged next-level gaming companion that actually feels good.
简单来说：让我们打造一个超强、次世代的游戏伴侣，真正让人感觉良好。
There are already multiple frameworks that let LLMs control NPC dialogue. They are fantastic for role-playing and staying in character, but they have two recurring problems: weak world agency and latency. Good at talking but far less reliable at performing actions and terrible at complex instruction sets. You may have also noticed how popular AI NPC demos often cut between the player speaking and the AI replying, trying to mask latency. Can we do better?
已有多个框架能让LLM控制NPC对话。它们非常适合角色扮演和保持角色设定，但有两个反复出现的问题：世界行动能力弱和延迟高。擅长说话，但在执行动作方面远不可靠，且对复杂指令集处理糟糕。你可能也注意到，流行的AI NPC演示常常在玩家说话和AI回复之间切换，试图掩盖延迟。我们能做得更好吗？
I wanted a companion that:
我想要一个伴侣，它应该：
- Useful and instant. It should fight, fetch, loot, inspect, carry and give items etc etc, follow complex multi-step instructions without feeling buggy or experimental. This matters especially in VR, where navigating menus is cumbersome and immersion-breaking. it needs to be FAST fast, not just fast
- Alive and present. It should have a fun, endearing personality, not canned robotic pre-written responses. Remember shared experiences and change over time. The microphone stays active while a session is running: you do not summon Varkos through a dialogue menu, you talk to him. When immersion kicks in, it should feel like you are not playing alone.
- Local and private wherever practical. The elephant in the room is that cloud LLM calls can get quite pricy (especially with multi-thousand-token LLM calls) and the added latency can be an experience killer. And why turn a private single-player game into a metered and surveilled experience? Let's try to give as much control to the user as possible (bonus it's a fun technical challenge).
- 实用且即时。它应该能战斗、取物、搜刮、检查、携带和给予物品等等，遵循复杂的多步骤指令，而不显得有bug或实验性。这在VR中尤其重要，因为导航菜单繁琐且破坏沉浸感。它需要快，非常快，不仅仅是快。
- 鲜活且在场。它应该有有趣、讨人喜欢的个性，而不是预先编写的机械回复。记住共同经历并随时间变化。会话运行时麦克风保持激活：你不是通过对话菜单召唤Varkos，而是直接和他说话。当沉浸感袭来时，应该感觉你不是一个人在玩。
- 在可行的地方尽可能本地化和私密。房间里的大象是云LLM调用可能相当昂贵（尤其是数千token的LLM调用），而且增加的延迟可能破坏体验。为什么要把私人的单人游戏变成按量计费且被监视的体验？让我们尽量给用户更多控制权（额外好处是这是一个有趣的技术挑战）。
Basically: a single-player game where you are not playing alone.
基本上：一个单人游戏，但你并非独自游玩。
Complex commands
复杂指令
Varkos can handle commands that extend beyond one immediate action. Plans can wait for events, preserve targets between steps, monitor progress and repair or stop when world state changes. Nothing is pre-scripted.
Varkos能处理超出单一即时动作的指令。计划可以等待事件，在步骤之间保留目标，监控进度，并在世界状态变化时修复或停止。没有任何预脚本。
Let's see some examples
让我们看一些例子
Varkos receives a conditional instruction involving the next arrow. He registers the future trigger instead of acting immediately, waits for the correlated projectile impact and then continues the plan.
Varkos收到一个涉及下一支箭的条件指令。他注册未来触发器而不是立即行动，等待相关的投射物撞击，然后继续计划。
Long-form multi-step command
长形式多步骤指令
“I want you to wait here and I’m gonna go over there. Once you see the signal, the signal is going to be an arrow I fire up in the sky, I want you to pick up this potion and come and bring it to me. Okay?”
“我想让你在这里等着，我要去那边。一旦你看到信号，信号将是我向天空射出的箭，我想让你拿起这瓶药水，过来带给我。好吗？”
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
A deferred command follows a real projectile event in Skyrim.
在《天际》中，一个延迟指令跟随真实的投射物事件。
Item search
物品搜索
Varkos can search the grounded world state for a requested item, identify where it is and respond using what is actually present in the game.
Varkos 可以搜索基于游戏世界的状态，查找请求的物品，识别其位置，并根据游戏中实际存在的内容进行回应。
“Do you see the ceremonial sword anywhere?”
“你看到那把仪式剑了吗？”
Varkos picks up a different sword and brings it to us. We tell him that’s not the one, then he offers to be on the lookout.
Varkos 拿起另一把剑并带给我们。我们告诉他那不是那把，然后他主动提出会留意寻找。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
Finding an item through game state rather than inventing an answer.
通过游戏状态而非虚构答案来查找物品。
Hide-and-seek
捉迷藏
Hide-and-seek is not a single API call. It becomes a persistent goal with movement, waiting, monitoring and completion conditions.
捉迷藏不是单一的 API 调用。它成为一个持续的目标，包含移动、等待、监控和完成条件。
“Let’s play hide-and-seek again. You wait here and I’m gonna go hide, then count to ten and come and try to find me.”
“我们再玩一次捉迷藏吧。你在这里等着，我去藏起来，然后数到十，再过来找我。”
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
A game represented as a persistent plan rather than a line of dialogue.
游戏被表示为持续的计划，而非对话的台词。
Loot this chest and give me the potion
打开这个箱子，把药水给我
This combines a grounded container, a filtered loot step and an inventory transfer. Each physical result advances the next part of the plan.
这结合了基于世界的容器、过滤的拾取步骤和库存转移。每个物理结果推进计划的下一个部分。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
Loot, select and transfer while preserving the requested item.
拾取、选择并转移，同时保留所请求的物品。
Pick up all the items
拾取所有物品
“Pick up all the items and give them to me” becomes a bounded collection plan over real references. Varkos gathers them, returns and transfers them without pretending that one magical action means “all.”
“拾取所有物品并给我”成为一个基于真实引用的有界收集计划。Varkos 收集它们，返回并转移，而不假装一个魔法动作意味着“所有”。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
A collection plan operating on grounded world objects.
一个基于世界对象的收集计划。
Combat
战斗
Varkos receives grounded events from the game, can warn the player through a fast reflex path and uses native body control to act. Instruction plans can strategize (e.g. attack this, then retreat, etc.), and his emotional state can affect how and if he chooses to fight.
Varkos 接收来自游戏的基于世界的事件，可以通过快速反射路径警告玩家，并使用原生身体控制来行动。指令计划可以制定策略（例如，攻击这个，然后撤退等），他的情绪状态可以影响他是否以及如何选择战斗。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
Combat footage 1: dungeon combat.
战斗片段1：地牢战斗。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
Combat footage 2: perception, warning and physical action on the latency-sensitive path.
战斗片段2：感知、警告和物理行动在延迟敏感路径上。
Personality evolution
个性演化
Varkos is fully customizable. He does not have to be a demon dog, and the runtime does not have to control only a single character. What systems are applied and what they do, is up to open configuration.
Varkos 是完全可定制的。他不一定非得是恶魔犬，运行时也不一定只控制单个角色。应用哪些系统以及它们做什么，取决于开放的配置。
One part of my current build still fully depends on big model/cloud LLM calls: slow personality evolution. This work happens away from the real-time action path. As the player and Varkos travel together, important interactions become evidence for gradual changes to his personality.
我当前构建的一部分仍然完全依赖大型模型/云端 LLM 调用：缓慢的个性演化。这项工作发生在实时行动路径之外。随着玩家和 Varkos 一起旅行，重要的互动成为他个性逐渐变化的证据。
My demo Varkos begins as a demon reincarnated as a dog. He considers his canine instincts humiliating, his dog body a prison, and is mistrustful, proud and sarcastic. Through shared experiences he can become more and more domesticated, grow attached to the player and starts enjoying being a dog. Eventually he starts bringing over toys because he wants to play, running off to chase things and seeking affirmation from the player.
我的演示版 Varkos 开始时是一个转世为狗的恶魔。他认为自己的犬类本能是耻辱，狗的身体是监狱，并且多疑、骄傲、讽刺。通过共同的经历，他可以变得越来越驯化，对玩家产生依恋，并开始享受作为狗的生活。最终他开始叼来玩具因为想玩，跑去追东西，并寻求玩家的肯定。
Only the starting character traits are authored. The system changes both his explicit traits and his emotional homeostasis. How easily he becomes irritated, frightened, affectionate or playful, etc etc. He can overwrite parts of his vocabulary and code. Changes are versioned and reversible.
只有起始角色特质是作者编写的。系统会改变他的显性特质和情绪稳态。他变得容易烦躁、害怕、亲昵或爱玩等等的程度。他可以覆盖部分词汇和代码。更改是版本化的且可逆的。
I could make it more bounded, but I think there's something fun about some open world clankiness, so how he evolves is up in the air.
我可以让它更受限制，但我认为开放世界的一些粗糙感很有趣，所以他的演化方式是不确定的。
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
The demon slowly discovers that being a pup is not a bad life. (And he has learnt to love cabbage...)
恶魔慢慢发现做一只小狗并不是糟糕的生活。（而且他已经学会了爱卷心菜……）
Dog in and out of the game - Void mode
游戏内外的狗 - 虚空模式
My plan is to make this system a gaming companion that can follow you across multiple different games, not just Skyrim (Skyrim felt like a good starting point due to its massive modding community, VR support and big open world).
我的计划是让这个系统成为一个游戏伴侣，可以跟随你穿越多个不同的游戏，而不仅仅是《天际》（《天际》感觉是一个很好的起点，因为它有庞大的模组社区、VR 支持和大型开放世界）。
For this reason he exists outside the game too. When the game closes, he enters “void mode” and cannot see or feel anything. How he responds to that depends on his personality evolution.
因此，他也存在于游戏之外。当游戏关闭时，他进入“虚空模式”，无法看到或感受到任何东西。他对此的反应取决于他的个性演化。
Being mean to Varkos results in some pretty grim attitudes.
对 Varkos 刻薄会导致一些相当阴郁的态度。
WTF… SHUT IT DOWN!
搞什么……关掉它！
Your browser does not support embedded video.
您的浏览器不支持嵌入式视频。
Speech-only contact after the game world and his body are gone. (Needs sound.)
游戏世界和他的身体消失后，仅语音接触。（需要声音。）
This state also works as an in-between for different games. One moment Varkos could be fighting a dragon, then the world goes dark, then he appears beside you in Microsoft Flight Simulator. Maybe he would be shocked, need time to understand the new world and slowly learn what its machines and rules mean, or maybe he knows about it already and overjoyed tries to chase the sun.
这种状态也充当了不同游戏之间的过渡。前一秒瓦尔科斯可能还在与巨龙搏斗，下一秒世界陷入黑暗，然后他出现在你身边的《微软飞行模拟器》中。也许他会感到震惊，需要时间来理解这个新世界，慢慢了解它的机器和规则意味着什么；又或许他早已知道这一切，欣喜若狂地试图追逐太阳。
Let's talk technology now
现在我们来谈谈技术
Unfortunately I am bitter-lesson pilled. Big model is better. If we wanted a perfectly intelligent system then letting a council of hyper-intelligent LLMs control impulses, sensory processing, thinking and acting at sufficient refresh rate would be best.
不幸的是，我深受“苦涩教训”的影响。大模型更好。如果我们想要一个完美智能的系统，那么让一群超智能LLM以足够的刷新率控制冲动、感官处理、思考和行动将是最佳选择。
In some early experiments this worked insanely well, unfortunately today it is too slow and too expensive. I do believe this will be the approach of some vague future.
在一些早期实验中，这种方法效果惊人，但遗憾的是如今它太慢且成本太高。我确实相信这将是某个模糊未来的方法。
Until then however we need to hack our way in. Today's games have pretty cool "AI" (not in the llm sense, more in the behavioral graph one), games like Red Dead Redemption and Dwarf Fortress have tons of depth and they can run perfectly on 10 year old hardware.
然而，在那之前，我们需要另辟蹊径。如今的游戏拥有相当酷的“AI”（不是LLM意义上的，更多是行为图意义上的），像《荒野大镖客》和《矮人要塞》这样的游戏拥有大量深度，而且它们可以在10年前的硬件上完美运行。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力