17个AI打7700手扑克：决策模型与廉价LLM表现媲美前沿模型

I built this benchmark to answer one practical question: if you need an AI to make a sequence of decisions under uncertainty, should you use one of the new “decision” models, a cheap chat LLM, an expensive frontier model, a model running on your own GPU, or a plain script? Poker is a good place to ask. Every decision has a price in chips, and luck can be measured and partly removed.

The short version

• Cheap reasoning LLMs played as well as the frontier. DeepSeek V4.1 Flash, Muse Spark Contributor and GPT-6 Luna finished level with or above GPT-6.1 Sol and Claude Opus 5.5. Head to head, Luna and Muse drew with both. Luna costs $0.71 per thousand hands; Opus costs $22.
• A 30-line Python script beat the decision models. equity_tag estimates its chance to win with a quick simulation and bets when it is ahead. It beat Jev 1.13, the best decision model, by 2.7 big blinds per hand (95% interval 1.0 to 4.6) over 100 duplicate deals. It also beat Solar by 5.3, and led pplx by 1.3, right at the edge of significance.
• The script held Opus to a draw. Over 70 duplicate deals, Opus was ahead by 0.5 big blinds per hand, well inside the noise. GPT-6.1 Sol did beat it, by 2.4 (0.2 to 4.7).
• Decision models are cheap, fast and deterministic, and they play strange poker. Jev almost never raises before the flop, then bets six times the pot after it. pplx folds hands that are ahead on the river. Solar goes all-in in more than half its hands.
• Bielik, the Polish open-weights model I ran locally, finished last, below the random bot. It never once folded to a bet.

The best hands

The table above replays real hands from the benchmark. In the first one, a model whose whole job is to make decisions looks at two aces, the best starting hand in the game, faces a small raise from a bot that picks its actions at random, says it has a 48% chance to win, and folds. The bar under the table shows what it could not see: with both hands known, its aces were an 84% favourite.

That is not a cherry-picked glitch. Over 1,400 hands, Solar Decide folded a premium hand (AA, KK, QQ or AK) in 4 of the 13 spots where it held one. The random bot in the same league did it 3 times in 18.

Luck, and how to take it out

Poker has enormous variance. A good player can lose for a thousand hands because the other player got better cards. Most AI poker results you see online are a few hundred hands of raw chip counts, and that is mostly a measurement of luck.

Each match here is heads-up no-limit Texas hold’em: two players, blinds of 50 and 100 chips, 100 big blinds each at the start of every hand. Stacks reset after each hand, so nobody busts and every hand counts the same. Three things keep luck out of the ranking:

• Duplicate deals. Every deal is played twice with the seats swapped. If A gets aces and B gets kings in the first hand, B gets the aces in the second. What remains is the difference in how they played the same cards.
• Shared deals. Every pairing in a league uses the same list of deals, so everyone faces identical cards.
• All-in adjustment. When both players are all-in before the river, the hand is scored by each player’s equity at that moment, not by the cards that happened to come. This alone cut the spread of results by a factor of 2.7.

Below is a real match, counted three ways. Start with the first step, the way most benchmarks count, and watch the confidence band. Then switch to duplicate deals and the all-in adjustment. The match and the cards do not change; only the noise does. The slider replays the match deal by deal: early on, the band is wide enough to crown either player.

Even with all three tricks, one duplicate deal between two players swings by about 9 big blinds, so pinning a single pairing to within one big blind per hand takes about 320 deals. That is why the cheap models play many opponents and the expensive ones play the opponents that matter.

All the AI players see exactly the same text: their cards, the board, the pot, the stacks, the betting so far, a small tracker panel with the opponent’s statistics, and a menu of legal actions (fold, check, call, five bet sizes from a third of the pot to one and a half pots, all-in). They answer with one option and an estimate of their chance to win. Chat models get the menu as a JSON schema; decision models get it through OpenRouter’s Decisions API, which returns a probability for each option.

The rating is the average result per hand in big blinds, fitted over every match at once, so two players who never met still get comparable numbers through common opponents. Intervals come from a bootstrap over deals.

The decision models

Decision models are a new kind of endpoint: instead of free text, you send a state and a list of options and get back a choice with probabilities. They are very cheap (5 to 10 cents per thousand decisions) and fast (Jev answers in 0.3 seconds). I tested four: Jev 1.13, pplx-decider 27B, Solar Decide and Mercury Decide.

Because they return a probability for every option, you can look inside a decision. The aces fold from the replays above was not a close call that went wrong: Solar spread its belief across the whole menu, and fold won with a quarter of it.

Jev was the best of them and plays the strangest poker in the league. It raises before the flop in 3% of hands, calls almost everything, and then, after the flop, its median bet is six times the pot.

A six-pot bet wins against players who fold too much. Against a script that simply calls when its equity justifies it, the same bets lose whole stacks. The second replay above is one of them.

pplx has the opposite problem. In 7 of the 35 river spots where it had at least 90% equity, it folded. It also underestimates its own hand: asked for its chance to win, it reports on average 19 points less than a simulation gives. Jev underestimates by 17 points, Solar by 17, Mercury by 15. The chat LLMs are off by 4 to 9 points; the scripts by under 1.

Solar is not a poker player in any useful sense. It goes all-in in 56% of its hands and folds premium hands a third of the time.

The cheap LLMs

The five cheap reasoning LLMs play something that looks like textbook poker: they enter many pots, raise more often than they call, and size their bets between half a pot and a full pot. They are also the best at judging their own hands among the cheap players: their win-probability estimates correlate 0.72 to 0.83 with the real equity, against about 0.6 for Jev and pplx.

The clearest way to see the difference is to put a bet in front of each player and look at what it does with weak and strong hands. A sound player folds the weak ones and raises the strong ones. The calling station never folds, by design. Bielik never folds either, and nobody designed it that way.

They are not consistent, though. Asked about the same spot twice, GLM picks the same action only 55% of the time and MiMo 65%; the best, DeepSeek, manages 82%. Decision models are deterministic: Jev and pplx always give the same answer. Renaming the suits, which changes nothing about the hand, still flips about one answer in ten for both. The frontier models are the steadiest language models here: Opus gave the same answer to a repeated question 40 times out of 40 and kept it after the suits were renamed 38 times; Sol managed 39 and 40.

The frontier round

Frontier models are expensive enough that I could not afford a full league. Instead, GPT-6.1 Sol and Claude Opus 5.5 (both with their default reasoning) each played the five opponents whose ratings were already pinned down: the script, Jev, pplx, Luna and Muse Spark Contributor. They also played each other. Then I gave both a second, longer match against the script.