GLM-5.2 成为最强开源模型，逼近前沿

GLM-5.2 arrived last week. It boasts excellent benchmarks and looks strong. Benchmarks here are a de facto ceiling of how good it is, not a point estimate. Essentially all other aspects of an open model like this, beyond speed and price, will almost always be worse than the numbers suggest. Still, impressive. It is definitely a large step up from GLM-5.1, and likely the strongest open model. GLM-5.2 is still substantially behind the absolute frontier, although plausibly on the cost-benefit Pareto frontier. It seems closer to the frontier than previous efforts, including probably closer than DeepSeek R1 was during the DeepSeek moment. This is the new ‘peak close behind’ moment. Its existence is a substantial updates to push back some of the ‘where are all the updates’ updates in the opposite direction over time. Purely in terms of core tasks that GLM-5.2 is capable of doing, and ignoring missing features and its inferior generalization, and ignoring that it is distilled from Claude, and ignoring the Mythos class of models, and marking purely from date of public release, you can make a case GLM-5.2 is somewhere between 4 months and 7 months behind the frontier, at a lower price. That does not mean it is all that useful in practice. Finding its niche is tricky unless you inherently value openness. It is not cheap enough, or better enough than cheaper alternatives, for the true bulk tasks, nor strong enough for the strongest tasks. There are various practical difficulties, including lack of vision. This post gives GLM-5.2 the full capabilities post treatment. But first, a word for our favorite Congressional candidate, whose election is tomorrow. Alex Bores For Congress In NY-12 In the strongest terms, this blog enthusiastically endorses democrat Alex Bores in his congressional primary in my home district, NY-12. Alex Bores has been a champion of sensible AI regulation in the New York Assembly, including championing the RAISE Act, and fighting to keep its provisions intact against strong opposition, risking great political capital. He understands and I believe primarily cares about AI existential risk. He does discuss other AI issues as well, as this is good politics and the other issues he discusses are real concerns, but what matters is the frontier. If he is elected to Congress, he will be a champion of sensible federal AI frontier model regulation. Having a champion in Congress willing to stake their political capital and time is vital to getting things done. He will also bring the knowledge and technical chops necessary to move this forward. This election is also an opportunity to send a message. OpenAI and a16z’s Leading the Future declared Alex Bores their primary target. Him losing is a potential chilling effect for other candidates and could help cower others into not ‘taking on’ OpenAI or advocating for AI regulation. Him winning (this is a very safe district, whoever wins the primary will win the general election) would do the opposite, and indicate that we can stand against such matters. If you live in the district and will be voting tomorrow, or otherwise could potentially assist, and want to chat with someone about this, you can fill out this form. Ok, that’s over with. On to GLM-5.2. Signs of Life Teortaxes: hey @TheZvi , if I may GLM is the strongest Chinese lab (at this specific moment) and this really is a frontier model. It is ≈Opus 4.7 in almost all text-only ways. Is reduces the gap more than R1 did at its time. Do pay attention, we don’t want to repeat the same mistakes do we. Teortaxes (DeepSeek 推特铁粉 2023 – ∞): GLM is the first time I see a Chinese agent capable of actually doing the /goal thing. It CAN work for hours, it can just keep obsessively optimizing. I get that Xiaomi/Kimi/Qwen/MInimax nominally have it too. But it has never felt so solid. one nitpick: permission hell in Zcode amendment, you can just go YOLO actually but the default “edit automatically” mode is too restrictive, eg it can’t use puppeteer [his ‘oh shit’ moment was it doing well on CritPt where it matched Opus 4.8 and trailing only high effort settings on top frontier models.] Teortaxes suggesting GLM-5.2 might be something, and he’s reasonably restrained with such suggestions, so I did a reaction thread and investigated. What did we find? The Benchmarks The benchmarks are remarkably close to frontier level. Artificial Analysis v4.1 has GLM-5.2 at a damn impressive (for open models) 51, behind only Fable (60), Opus 4.8 (56), GPT-5.5 (55) and Opus 4.7 (54), and tied with GPT-5.4. They have it at 95 in the speed index, the same as GLM-5.1, just behind DeepSeek v4. Gemini Flash 3.5 is faster at 116, but all the clearly better models are at least somewhat slower, GPT-5.5-xhigh gets 63 and Opus 4.8 scores 58. Cost is lower than the big closed models, but as I understand it relatively high for open models, partly because it is a very token hungry model. API cost is $1.40/$0.26/$4.40 for input, cached input and output. Their subscription plans go from $10 to $160 per month, with discounts for a year commitment. That leaves GLM-5.2 in an awkward spot, where other open models can do easy things a lot cheaper, and for hard things you usually want to hire the best. How do you know you are in its sweet spot, if one exists, unless you want the strongest open model? If you want the strongest open model, the choice seems clear right now. It gets +4 on AA-Omniscience, behind several other open models and well outside the top tier. There are a number of other AA scores I’d have been curious about, where they still haven’t scored GLM-5.2. LiveBench has GLM-5.2 between Opus 4.5 and Opus 4.6. Vals.ai has GLM-5.2 in 5th behind Fable, Opus 4.8 and 4.7 and GPT-5.5, as the clear best open model. FrontierSWE has it in 3rd only one notch behind Opus 4.8 and one notch ahead of GPT-5.5. Everyone is well behind Fable. The Jake Boggs Capability index has it on par with Sonnet 4.6, which is still ahead of everyone except OpenAI and Anthropic. On PosttrainBench is is actually #1 slightly ahead of Opus 4.8. Fable and GPT-5.5 really struggle here, I don’t know why. It has the second highest score on Vending-Bench 2, which was surprising. We need to be more curious about what makes models score highly here. It gets #8 on EQ-Bench for longform creative writing. It landed at #25 on Arena for text, although there are a lot of duplicate variants ahead of it. On the agent leaderboard it is #10, behind Fable, and variations fo Opus 4.6-4.8 and GPT-5.4 and GPT-5.5. It scored badly on You’re Absolutely Right, the anti-sycophancy test. All of that tells a consistent story. On traditional benchmarks one might be targeting, performance is impressive, on average around Opus 4.7. The less targetable the benchmark, the worse the performance, but still an excellent showing and the best open model. The pattern feels somewhat benchmaxxed, but not excessively. Håvard Ihle: New clear best open model on WeirdML [#16 overall behind variations of GPT-5.2 to 5.4, Fable and Claude Opus 4.6-4.8 plus a few Geminis]. GLM improving faster than I expected. Updates me towards expecting a Chinese Mythos level model in less than a year, but still very unsure. GLM-5.2 Is Distilled From Claude Some of the evidence: It has a strong prior that it is Claude, which presumably is from distillation. It identifies as Claude often and has the distinct ‘Claude voice.’ It also uses a Claude harness, although I think that mostly doesn’t cause such behavior. It would surprise me greatly if GLM-5.2 was not heavily distilled from Claude Opus. That does not invalidate the model, but it does mean two things. Distilled models tend to generalize poorly. They overperform on benchmarks and benchmark-like tasks, and on the most common tasks, and underperform on less common tasks. Distillation causes you to underestimate the gap in capabilities, especially now that top models are potentially unavailable for distillation. Positive Responses On to t