FLEET算法：将记忆引入搜索以优化奖励最大化任务
Adding memory to search instead of sampling in reward maximization tasks [R]
这是一篇提供完整可复现流程与参数权衡的硬核算法研究，FLEET 通过引入记忆机制显著提升了搜索效率，值得做推理优化的同学深入阅读并复现。
I am one of the authors of FLEET - an algorithm that enhances Best-of-N generation by attributing external rewards to particular tokens and then uses MCTS to adjust logits during the next run.
我是 FLEET 的作者之一——这是一种通过为特定 token 分配外部奖励，然后使用 MCTS 在下一轮运行中调整 logits 来增强 Best-of-N 生成的算法。
I find it rather funny that most of the tasks where repetitive sampling is widely used are based on reward maximization, yet it is not aware of that reward. Tuning the sampling parameters allows to make the process more efficient, but it is still a blind search. We propose a way to make generation aware of previous rewards with solutions on how to attribute reward to the completion and how to use this information.
我发现相当有趣的一点是，大多数广泛使用重复采样的任务都基于奖励最大化，但它们却并未意识到这一奖励。调整采样参数可以使过程更高效，但这仍然是一种盲目搜索。我们提出了一种使生成过程能够感知先前奖励的方法，并提供了如何将奖励分配给完成结果以及如何利用这些信息的具体解决方案。
In FLEET the technique from adaptive sampling methods is used that is to track logits for which entropy and varentropy are high thus showing the model's uncertainty about token optimality. We treat these states as branching points. The corresponding normalized hidden states are stored in the vector store and mapped to metadata entries with the history of rewards and transitions between "nodes". The retrieval and update of metadata is based on cosine similarity as for very high similarity KL divergence is low enough to preserve most of the meaningful tokens.
FLEET 采用了自适应采样方法中的技术，即追踪那些熵（entropy）和方差熵（varentropy）较高的 logits，从而显示模型对 token 最优性的不确定性。我们将这些状态视为分支点。相应的归一化隐藏状态被存储在向量数据库中，并映射到带有奖励历史和“节点”间转换记录的元数据条目中。元数据的检索和更新基于余弦相似度，因为对于高度相似的情况，KL 散度足够低，足以保留大部分有意义的 token。
Instead of actually selecting the tokens FLEET uses modified MCTS to rank top-k tokens + special exploration (or other tokens) set and penalize the suboptimal ones. Then decoding strategy is applied to modified logits.
FLEET 并不直接选择 token，而是使用修改后的 MCTS 对 top-k token + 特殊探索（或其他 token）集合进行排序，并对次优 token 施加惩罚。然后对修改后的 logits 应用解码策略。
It was tested on GSM8K and LiveCodeBench v6 easy split with Llama 3.2 3B, penalty set to effectively zero probability for the suboptimal tokens + greedy decoding:
它在 GSM8K 和 LiveCodeBench v6 easy split 上进行了测试，使用的是 Llama 3.2 3B 模型，将次优 token 的概率有效设为零，并采用贪婪解码：
- For GSM8K it solved just seven more tasks, but reached the sampling baseline with half the iterations.
- For LiveCodeBench it increased the score from 0.59 to 0.69 under the same budget and reached the baseline even faster, now with only 9 iterations against 32.
- 在 GSM8K 上，它多解决了七个任务，但仅用一半的迭代次数就达到了采样基线水平。
- 在 LiveCodeBench 上，它在相同的预算下将分数从 0.59 提高到 0.69，并且以更快的速度达到了基线水平，现在仅需 9 次迭代，而基线需要 32 次。
The sequential execution is not required, as it is not updated during the iteration itself it can simply be passed as a lookup table. The metadata store can be preserved as a prior for other tasks or to enrich SFT/RL.
不需要顺序执行，因为它在迭代过程中本身不会更新，可以直接作为查找表传递。元数据存储可以保留为先验知识用于其他任务，或用于丰富 SFT/RL。
Paper (preprint): https://arxiv.org/abs/2609.27657
论文（预印本）：https://arxiv.org/abs/2609.27657
Huggingface: https://huggingface.co/papers/2609.27657
Huggingface：https://huggingface.co/papers/2609.27657
Repository (experiments, examples and python package): https://github.com/Alexiush/fleet
代码库（实验、示例和 Python 包）：https://github.com/Alexiush/fleet
There are more details on changes made to MCTS, how to tune the search parameters for specific model and task as well as code for experiments and trajectories.
有关对 MCTS 所做的更改、如何针对特定模型和任务调整搜索参数以及实验和轨迹的代码，还有更多详细信息。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力