StarSkirmish Bench v0.2：GPT-6 Astra夺冠
Show HN: Gemini 4 Argon plays StarCraft
这是一份极具参考价值的硬核评测，揭示了长程推理与Agentic编码能力的独特性，建议关注模型在复杂代码生成与自我迭代方面的实际表现差异。
TL;DR
TL;DR（太长不看）
- StarSkirmish is a benchmark where LLMs write bots for StarCraft: Brood War
- Each LLM gets one hour to write a Protoss bot in C++, which is then evaluated against other LLMs' bots and established human written bots
- StarSkirmish tests long horizon reasoning and agentic coding as LLMs must incorporate learnings from practice games to improve their StarCraft bot
- StarSkirmish 是一个基准测试，LLM 在此为《星际争霸：母巢之战》编写机器人
- 每个 LLM 有一小时的时间用 C++ 编写一个神族机器人，随后将其与其他 LLM 编写的机器人以及成熟的人类编写机器人进行对比评估
- StarSkirmish 测试长程推理和代理式编码能力，因为 LLM 必须将从练习游戏中获得的学习成果融入其中，以改进其星际争霸机器人
StarSkirmish Bench v0.2
The StarSkirmish Bench evaluates how strong of a StarCraft bot can each LLM write given 1 hour of wall clock time. v0.2 updates StarSkirmish Bench v0.1 with flagship models from OpenAI, Anthropic, Google and Meta, each running in its lab's 1st party CLI harness.
StarSkirmish Bench 评估在 1 小时墙钟时间内，每个 LLM 能写出多强的星际争霸机器人。v0.2 版更新了 StarSkirmish Bench v0.1，加入了来自 OpenAI、Anthropic、Google 和 Meta 的旗舰模型，每个模型均在其实验室的第一方 CLI 工具链中运行。
An LLM’s score reflects its win rate against a roster of competitive human written bots and demo bots.
LLM 的得分反映了其在面对一系列竞争性人类编写机器人和演示机器人时的胜率。
StarSkirmish Bench v0.2
1 hour runs, in its 1st party CLI harness. Scores are average win rates (0–100) of each model's 5 runs against a roster of competitive human written bots.
1 小时运行，使用其第一方 CLI 工具链。得分为每个模型在 5 次运行中对阵一系列竞争性人类编写机器人的平均胜率（0–100）。
Gemini 4 Argon runs at high, the highest reasoning effort available to it; every other model runs at max.
Gemini 4 Argon 以“高”模式运行，这是它可用的最高推理强度；其他所有模型均以最大强度运行。
GPT-6 Astra is the top scoring StarSkirmish Bench v0.2 LLM, a clear step ahead of GPT-6.1 Sol in second. Gemini 4 Argon, Claude Opus 5.5 and Claude Fable 5.1 are functionally tied behind them, all scoring between 37 and 39.
GPT-6 Astra 是 StarSkirmish Bench v0.2 中得分最高的 LLM，明显领先于排名第二的 GPT-6.1 Sol。Gemini 4 Argon、Claude Opus 5.5 和 Claude Fable 5.1 紧随其后，三者实力相当，得分均在 37 到 39 之间。
StarSkirmish Bench vs. Cost
StarSkirmish Bench 得分与成本对比
StarSkirmish Bench score against the average API cost of one 1 hour run. Cost is on a log scale.
StarSkirmish Bench 得分与单次 1 小时运行的平均 API 成本对比。成本采用对数刻度。
Muse Wonder Full's pricing is based on Muse Wonder's API pricing.
Muse Wonder Full 的定价基于 Muse Wonder 的 API 定价。
GPT-6.1 Sol offers uniquely good value: it is the only LLM in the most attractive quadrant, scoring second at just over a tenth of GPT-6 Astra's cost per run.
GPT-6.1 Sol 提供了独特的极高性价比：它是唯一处于最具吸引力象限的 LLM，以仅略高于 GPT-6 Astra 单次运行成本十分之一的价格获得第二名。
No Signs of Saturating
未见饱和迹象
The strongest human written bots are still well ahead of every model. Against Stardust, PurpleWave and BananaBrain, the best model, GPT-6 Astra, wins only 7% of its games.
最强的人类编写机器人仍远远领先于所有模型。在面对 Stardust、PurpleWave 和 BananaBrain 时，表现最好的模型 GPT-6 Astra 仅赢得了 7% 的比赛。
Win rates vs the top human written bots
对阵顶级人类编写机器人的胜率
Average win rate of each model's 5 bots against top human written bots Stardust, PurpleWave, and BananaBrain.
每个模型的 5 个机器人对阵顶级人类编写机器人 Stardust、PurpleWave 和 BananaBrain 的平均胜率。
1 hour reasoning limits are a factor, but these matches are played on three 2 player maps. Real human written bot tournament conditions are significantly harder.
1 小时的推理限制是一个因素，但这些比赛是在三个双人对战地图上进行的。真实的人类编写机器人锦标赛条件要困难得多。
New AI Model Highlights
新 AI 模型亮点
Google and Meta's new flagships replace v0.1's Gemini 3.8 Flash and Muse Spark 1.3: Gemini 4 Argon places 3rd, and Muse Wonder Full 7th. GPT-6 Astra and Claude Opus 5.5 were covered in v0.1.
Google 和 Meta 的新旗舰模型取代了 v0.1 中的 Gemini 3.8 Flash 和 Muse Spark 1.3：Gemini 4 Argon 排名第三，Muse Wonder Full 排名第七。GPT-6 Astra 和 Claude Opus 5.5 已在 v0.1 中介绍过。
AI Model Behaviors
AI 模型行为
Each run's agent trace records how it spent its hour. The strongest predictor of a model's score is how much it practiced against the 7 strongest bots on the roster.
每次运行的智能体轨迹记录了它如何度过这一小时。预测模型得分的最强指标是它在阵容中最强的 7 个机器人身上花费了多少练习时间。
Win Rates vs. Practice Against Strong Bots
胜率与对抗强效机器人的练习次数对比
StarSkirmish Bench score against the number of practice games a model's median run played against the 7 strongest roster bots during the hour. The dashed line is a linear fit across the 9 models.
StarSkirmish Bench 得分与模型中位数运行在1小时内对抗7个最强阵容机器人所进行的练习游戏数量之间的关系。虚线表示跨越9个模型的线性拟合结果。
GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon start playing these bots early in the hour, while their own bots still lose to them. Muse Wonder Full, Claude Haiku 5.5 and GPT-6 Luna rarely or never do. Claude Fable 5.1 is the outlier: it scores 37 while practicing almost entirely against weaker bots.
GPT-6 Astra、GPT-6.1 Sol 和 Gemini 4 Argon 在小时初期就开始与这些机器人对战，而它们自己的机器人此时仍会输给对方。Muse Wonder Full、Claude Haiku 5.5 和 GPT-6 Luna 很少或从不这样做。Claude Fable 5.1 是异常值：它在几乎完全对抗较弱机器人的情况下得分为37分。
Run-to-Run Variance
运行间方差
Win rates of each model's 5 runs. Each dot is one run's final bot; the line spans the lowest to the highest run, the bar marks their mean (the model's Bench score) and the band one standard deviation (σ).
每个模型的5次运行的胜率。每个点代表一次运行的最终机器人；线条跨度从最低到最高运行，条形标记其平均值（即该模型的 Bench 得分），带状区域表示一个标准差（σ）。
Results also vary between runs of the same model. Gemini 4 Argon varies far more than the other labs' flagships: its best and worst runs are 18 points apart, against 9 for GPT-6 Astra and 8 for Claude Opus 5.5.
同一模型的不同运行之间结果也存在差异。Gemini 4 Argon 的波动远大于其他实验室的旗舰模型：其最佳和最差运行相差18分，而 GPT-6 Astra 为9分，Claude Opus 5.5 为8分。
Implementation Details
实现细节
Objective
目标
Each LLM writes a Protoss bot in C++ against BWAPI 4.4.0, played on OpenBW. Games are Protoss versus Protoss on three ladder maps: Heartbreak Ridge, Benzene and Destination. A game ends when one side's buildings are all destroyed. Games that reach the 60 minute cap (86,400 frames) are decided as on the BASIL ladder: the higher kills + razings score wins.
每个大语言模型使用 C++ 编写一个针对 BWAPI 4.4.0 的 Protoss 机器人，并在 OpenBW 上运行。比赛为 Protoss 对 Protoss，在三张天梯地图上进行：Heartbreak Ridge、Benzene 和 Destination。当一方所有建筑被摧毁时，比赛结束。达到60分钟上限（86,400帧）的比赛按照 BASIL 天梯规则判定：击杀数+摧毁建筑数更高的一方获胜。
The LLM starts from a skeleton bot that only mines, with the source code of the three demo bots and two standard bot libraries: FAP, a combat simulator, and BWEM, a map analyzer. Whatever is in its bot folder when the hour ends is its final bot, so it has to keep the bot compiling.
大语言模型从一个仅具备采矿功能的骨架机器人开始，并拥有三个演示机器人的源代码以及两个标准机器人库：FAP（战斗模拟器）和 BWEM（地图分析器）。一小时结束时其机器人文件夹中的内容即为最终机器人，因此它必须保持机器人代码可编译。
The LLM has tools to compile its bot, play practice games, and read each game's summary, transcript and its bot's own log. Practice opponents are grouped into tiers: D (the three demo bots), C and B (three mid-ranked human written bots each), A (BananaBrain, Locutus) and S (Stardust, PurpleWave).
大语言模型拥有编译机器人、进行练习游戏以及读取每场比赛摘要、转录记录和机器人自身日志的工具。练习对手按等级分组：D级（三个演示机器人）、C级和B级（各三个中级人类编写的机器人）、A级（BananaBrain、Locutus）和S级（Stardust、PurpleWave）。
Tournament Setup
锦标赛设置
LLM bots
大语言模型机器人
Opponents
对手
Games
比赛
Games / pair
每对组合的比赛场次
Each model's 5 final bots play each of the 13 roster bots, the 10 competitive human written bots and the 3 demo bots, 6 times on each map, on fresh seeds. Unlike v0.1, the LLM bots don't play each other.
每个模型的5个最终机器人分别与13个阵容机器人、10个竞技性人类编写的机器人以及3个演示机器人对战，每张地图各进行6次，使用全新的随机种子。与 v0.1 不同，大语言模型机器人之间不进行对战。
Harness and Reasoning
运行环境与推理能力
StarSkirmish Bench v0.1 ran every model in the same harness at 'high' reasoning. v0.2 runs each model in its lab's 1st party CLI harness at the highest reasoning setting it offers.
StarSkirmish Bench v0.1 在“高”推理设置下，使用相同的运行环境运行所有模型。v0.2 则在各自实验室提供的官方 CLI 运行环境中，以所能提供的最高推理设置运行每个模型。
Harness
运行环境
v0.1 ran every model in one neutral 3rd party harness; v0.2 runs each lab's models in its 1st party CLI harness. Both versions give each model 1 hour.
v0.1 在一个中立的第三方工具包中运行所有模型；v0.2 在每家实验室的自有 CLI 工具包中运行其模型。两个版本均给予每个模型 1 小时的时间。
v0.1
v0.2
Inspect ReAct deepagent
- Claude Opus 5.5
- GPT-6 Astra, Sol, Luna
- Gemini 3.8 Flash
- Muse Spark 1.3
Claude CodeClaude Opus 5.5, Fable 5.1, Sonnet 5.5, Haiku 5.5
CodexGPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna
Antigravity CLIGemini 4 Argon
Muse CodeMuse Wonder Full
CLI versions: Claude Code 2.1.296, Codex 0.162.1, Antigravity CLI 1.3.5 and Muse Code 1.4.7.
CLI 版本：Claude Code 2.1.296、Codex 0.162.1、Antigravity CLI 1.3.5 和 Muse Code 1.4.7。
Reasoning Effort
推理力度 (Reasoning Effort)
v0.1 ran every model at high reasoning effort; v0.2 runs them at max.
v0.1 以高推理力度运行所有模型；v0.2 则以最大力度运行。
v0.1
LowMediumHighXHighMax
低中高高X高最高
High
高
v0.2
LowMediumHighXHighMax
低中高高X高最高
Max
最高
Gemini 4 Argon runs at high, the highest effort supported.
Gemini 4 Argon 以高力度运行，这是其支持的最高力度。
Every v0.2 harness compacts the model's context at about 250k tokens. When a conversation nears that point, the CLI condenses it and the model carries on. Codex, OpenAI's CLI, for example, gives the GPT models its default 272k token window and triggers compression once 90% filled.
每个 v0.2 工具包都在约 25 万 token 处压缩模型的上下文。当对话接近该点时，CLI 会将其浓缩，模型继续运行。例如，OpenAI 的 CLI Codex 为 GPT 模型提供默认的 27.2 万 token 窗口，并在填充至 90% 时触发压缩。
The CLIs' web search and web fetch tools are turned off, and the agent has no other internet access. The CLI is installed before the clock starts, and the hour counts from when the agent starts. There is no submit tool: the harness collects the bot when the time runs out.
CLI 的网络搜索和网络抓取工具已关闭，且智能体没有其他互联网访问权限。CLI 在计时开始前安装，一小时从智能体启动时开始计算。没有提交工具：当时间耗尽时，工具包会收集机器人的输出。
Scoring
评分
v0.2 replaces v0.1's Elo ratings with plain win rates. A bot's score is its win rate over its tournament games, and a model's StarSkirmish Bench score is the average of its 5 bots' scores, from 0 (every game lost) to 100 (every game won).
v0.2 用简单的胜率取代了 v0.1 中的 Elo 等级分。机器人的分数是其锦标赛比赛的胜率，而模型的 StarSkirmish Bench 分数是其 5 个机器人分数的平均值，范围从 0（每场皆输）到 100（每场皆赢）。
Because every bot plays the same fixed roster, its games don't depend on the other LLM bots in the field. They all run in parallel, and a new model's bots can be scored without replaying anyone else's. A fixed roster also keeps the bar the same over time, so scores from different model generations are more directly comparable than Elo ratings fitted within one field.
由于每个机器人都使用相同的固定阵容进行比赛，其比赛结果不依赖于场上的其他 LLM 机器人。所有比赛并行运行，新模型的机器人可以在不重放任何其他模型比赛的情况下进行评分。固定的阵容也确保了评判标准随时间保持一致，因此不同代际模型的分数比在同一场域内拟合的 Elo 等级分更具直接可比性。
Benchmark Correlations
基准相关性
StarSkirmish no longer tracks the most widely reported evals. It barely moves with the Artificial Analysis Intelligence Index (ρ 0.26), where GPT-6 Astra, the top model here, only ties for 4th of these 9 models. It has also drifted away from Terminal-Bench 4.0, which it tracked closely in v0.1. Claude Sonnet 5.5 and Muse Wonder Full both rank near the top of Terminal-Bench 4.0 but near the bottom here.
StarSkirmish 不再追踪最广泛报道的评估指标。它与 Artificial Analysis Intelligence Index（相关系数 ρ 0.26）的相关性极低，其中 GPT-6 Astra 作为此处的顶级模型，在这 9 个模型中仅并列第 4。它也偏离了 Terminal-Bench 4.0，后者在 v0.1 中与 StarSkirmish 高度相关。Claude Sonnet 5.5 和 Muse Wonder Full 在 Terminal-Bench 4.0 中排名靠前，但在此处排名靠后。
Benchmark Correlation
基准相关性
Each model's StarSkirmish Bench score against its Vals IOI and WeirdML v3 scores, with the Spearman ρ between them. The dashed line is the best straight-line fit.
各模型在 StarSkirmish Bench 上的得分与其 Vals IOI 和 WeirdML v3 得分的对比，以及它们之间的斯皮尔曼相关系数 ρ。虚线为最佳直线拟合。
WeirdML v3 scores 7 of the 9 models: Gemini 4 Argon and Muse Wonder Full have not been included at time of writing.
WeirdML v3 对 9 个模型中的 7 个进行了评分：Gemini 4 Argon 和 Muse Wonder Full 截至撰写时未被纳入。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力