跳到主内容
@wquguru
精选88r/LLMDevs（Reddit）技巧与观点

用Hindsight给OnCall Agent加长期记忆的工程复盘

I gave my on-call agent long-term memory with Hindsight. Here's what changed

原文
发到 X
推荐理由

Agent长期记忆落地的硬核工程复盘，包含完整的API调用逻辑、具体的参数调优细节及避坑指南，对做RAG和Agent的同学极具参考价值。

On this page

在此页面

I pasted a production-looking alert into my own incident agent: payments-api returning 502s, Redis reporting "Too many connections". The answer was correct and useless: check Redis, scale it, add monitoring. Anything a search engine could have told me.

我将一个看似生产环境的告警粘贴到了我自己的事件代理中：payments-api 返回 502 错误，Redis 报告“连接数过多”。得到的回答既正确又无用：检查 Redis、扩容、添加监控。搜索引擎本可以告诉我的一切。

Then I turned memory on. Same alert, different answer: this matches INC-001 from May 8, and the fix that worked was raising Redis max_connections from 50 to 200 and setting the payments pool limit to 120. MTTR was 22 minutes.

然后我开启了记忆功能。同样的告警，不同的回答：这与 5 月 8 日的 INC-001 匹配，有效的修复方案是将 Redis max_connections 从 50 提升到 200，并将 payments 池限制设置为 120。平均修复时间（MTTR）为 22 分钟。

That gap is the whole project.

这个差距正是整个项目的核心。

I built this because the pattern is familiar to anyone who has carried a pager. An alert fires, the on-call engineer half-remembers a similar incident from a few months ago, and then spends twenty minutes digging through old tickets and chat threads for the fix. The knowledge exists, but it lives in people's heads and in postmortems nobody rereads. I wanted an agent that treats every resolved incident as something to learn from, so the next engineer starts from what the last one already figured out.

我构建这个项目是因为这种模式对任何经历过值班轮转的人都很熟悉。告警触发后，值班工程师半回忆起几个月前发生的类似事件，然后花费二十分钟在旧工单和聊天记录中挖掘修复方案。知识确实存在，但它存在于人们的头脑中以及无人重读的故障复盘报告中。我希望有一个代理能将每个已解决的事件视为学习素材，以便下一位工程师能站在前人的肩膀上开始工作。

What OnCall Memory does

OnCall Memory 的功能

OnCall Memory is a small Streamlit app. You paste an alert and a log snippet, and it produces two diagnoses side by side. The left one has no memory. The right one is grounded in past incidents stored in Hindsight, an open-source agent memory system.

OnCall Memory 是一个小型的 Streamlit 应用。你粘贴一条告警和一段日志片段，它会并排生成两个诊断结果。左侧的诊断没有记忆能力。右侧的诊断则基于存储在 Hindsight（一个开源的代理记忆系统）中的历史事件。

The stack is deliberately boring: Streamlit for the UI, Groq (openai/gpt-oss-120b) for generation, and Hindsight as the only persistent store. There is no database of my own.

技术栈故意选得平淡无奇：Streamlit 用于 UI，Groq (openai/gpt-oss-120b) 用于生成内容，Hindsight 作为唯一的持久化存储。我没有使用自己的数据库。

How Hindsight is used

Hindsight 的使用方式

Hindsight exposes three operations, and I use all three. The Hindsight docs cover them in detail, but the core calls are this small:

Hindsight 提供了三个操作，我都使用了。Hindsight 文档中有详细说明，但核心调用如下：

代码 · 12 行
from hindsight_client import Hindsight
client = Hindsight(base_url=HINDSIGHT_URL)
# retain: store a resolved incident
client.retain(
    bank_id="oncall-v3",
    content="INC-001: payments-api Redis connection exhaustion. "
            "Fix: raised max_connections 50 -> 200, pool limit 120.",
)
# recall: find relevant history for a new alert
client.recall(bank_id="oncall-v3", query="payments-api 502 Too many connections")
# reflect: reason over that history
client.reflect(bank_id="oncall-v3", query="How should I fix this incident?")
  • retain runs in two places. A seed script loads 15 synthetic postmortems with stable IDs and dates (payments-api, auth-service, Kafka consumers, Kubernetes, Postgres). And a "Mark resolved" form saves the root cause and fix of a new incident.
  • recall runs before the memory-enabled answer. The app shows the top five deduplicated incidents with ID, date, service and fix.
  • reflect produces the reasoning the final answer is built from.
  • retain 在两个地方运行。一个种子脚本加载了 15 个带有稳定 ID 和日期的合成故障复盘报告（涉及 payments-api、auth-service、Kafka 消费者、Kubernetes、Postgres）。还有一个“标记为已解决”表单保存新事件的根因和修复方案。
  • recall 在启用记忆的答复之前运行。应用会显示前五条去重后的事件，包含 ID、日期、服务和修复方案。
  • reflect 生成最终答复所依据的推理过程。

Telling similar incidents apart

区分相似事件

My seed data has several Redis incidents on purpose. INC-001 is connection exhaustion, fixed by raising max_connections. INC-015 is a failover that left 80 workers pointing at a retired primary, fixed by cutting the DNS cache TTL from 120s to 10s. INC-013 is missing TTLs on cache keys leading to OOM.

我的种子数据特意包含了几个 Redis 相关事件。INC-001 是连接耗尽问题，通过提高 max_connections 解决。INC-015 是一次故障转移导致 80 个工作进程指向已退役的主节点，通过将 DNS 缓存 TTL 从 120 秒降至 10 秒解决。INC-013 是缓存键缺少 TTL 导致内存溢出（OOM）。

A keyword search for "Redis" returns all of them. For the "Too many connections" alert, recall put INC-001 on top. For a failover timeout alert, it surfaced INC-015. Same service, same technology, different fixes, and the right one came back.

关键词搜索 "Redis" 会返回所有相关条目。对于 "连接数过多" 告警，记得将 INC-001 置顶。对于故障转移超时告警，则浮现出 INC-015。相同的服务、相同的技术，不同的修复方案，而正确的那个被检索到了。

Making it learn

让它学习

After a resolved incident, I saved INC-016 (stale DNS cache on payments-api workers after a Redis failover) through the "Mark resolved" form. A few minutes later I pasted a new failover alert, and INC-016 came back as the first recalled memory.

在解决了一个事件后，我通过“标记为已解决”表单保存了 INC-016（Redis 故障转移后 payments-api 工作节点上的过期 DNS 缓存）。几分钟后，我粘贴了一条新的故障转移告警，INC-016 作为第一条被回忆起的记忆出现了。

What went wrong

出了什么问题

  • The reasoning model returned empty answers. gpt-oss-120b spends tokens thinking before it writes. With a small max_tokens, message.content came back empty and my app treated that as a failure. Raising max_completion_tokens to 2000 and setting reasoning_effort to low fixed it. My fallback model name also no longer existed on Groq, so the fallback never worked.
  • My own validator hid a correct answer. I added a check that rejects any incident ID the model cites that wasn't actually recalled. The first version was too strict and replaced a correct answer with "no relevant history". It now rejects only IDs absent from recall.
  • The save form can pollute memory. "Mark resolved" saves whatever is in the incident box. I saved one incident while the old alert was still in the box, and it distorted later recalls. I had to reseed into a fresh bank.
  • Recalled isn't the same as used. On the failover alert the answer cited INC-015, while INC-016, the one I had just saved, ranked first in recall. The memory is there, but the model leans on the closest older match.
  • The data is synthetic. The 15 seeded incidents are made up. It shows the mechanism, not results on real production history.
  • 推理模型返回了空答案。gpt-oss-120b 在输出前会消耗 token 进行思考。由于 max_tokens 设置较小，message.content 返回为空，我的应用将其视为失败。将 max_completion_tokens 提高到 2000 并将 reasoning_effort 设置为 low 解决了这个问题。此外，我的备用模型名称在 Groq 上也不再存在，因此备用机制从未生效。
  • 我自己的验证器隐藏了正确答案。我添加了一项检查，用于拒绝模型引用的任何实际上未被召回的事件 ID。第一版规则过于严格，将正确答案替换为了“无相关历史记录”。现在它仅拒绝那些未出现在召回结果中的 ID。
  • 保存表单可能会污染记忆。“标记为已解决”会保存事件框中的任何内容。我在旧告警仍留在框中时保存了一个事件，这扭曲了后续的召回结果。我不得不向一个新的存储库重新注入数据。
  • 被召回不等于被使用。在故障转移告警中，答案引用了 INC-015，而我刚刚保存的 INC-016 在召回排名中位列第一。记忆确实存在，但模型倾向于依赖最接近的较旧匹配项。
  • 数据是合成的。这 15 个种子事件都是虚构的。它展示了机制，而非真实生产历史中的结果。

Lessons

经验教训

  • Make the difference visible. A Memory ON/OFF toggle taught me more than any log, and it makes the value obvious to someone seeing it for the first time.
  • Store exact fixes, not summaries. "Raised max_connections from 50 to 200" is useful at 2 AM. "Fixed Redis" is not.
  • Don't let the model invent history. Check every cited incident ID against what was recalled.
  • Test the model layer separately. My memory worked long before my LLM calls did.
  • Validate what you save. A memory system is only as good as the postmortems you put in.
  • 让差异可见。一个 Memory ON/OFF 开关教给我的比任何日志都多，而且它对第一次看到它的人来说价值一目了然。
  • 存储确切的修复方案，而不是摘要。“将 max_connections 从 50 提升到 200”在凌晨 2 点很有用。“修复 Redis”则不然。
  • 不要让模型编造历史。根据召回结果检查每一个被引用的事件 ID。
  • 单独测试模型层。我的内存系统在 LLM 调用正常工作之前很久就已经能用了。
  • 验证你保存的内容。一个内存系统的价值取决于你放入其中的事后分析报告的质量。

Code is on GitHub. Video walkthrough: https://youtu.be/0vvGlgbJXCc

代码位于 GitHub。视频演示：https://youtu.be/0vvGlgbJXCc

Tagging Code.in.

标签 Code.in。

  • #ai
  • #llm
  • #python
  • #agents

Comments

评论

Join the discussion

加入讨论

No comments yet. Be the first to comment.

暂无评论。成为第一个发表评论的人。

更进一步：量化金融体系

看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力

进入量化体系 →
话题#Hindsight#Agent#工程实践#长短期记忆

相似阅读

关联信息，但可能不是同一事件