Rippling 实测 15 款 AI 模型：最便宜与最贵打平

Rippling Tested 15 AI Models on Real Payroll Data. The Cheapest One Tied the Most Expensive One.

Rippling’s President and CPO Matt MacInnis published something most B2B companies have and almost none share: a real test of AI models doing real work inside a real production system.

Not a leaderboard. Not made-up tasks. About 2,100 graded attempts per model, across 15 models, on actual personnel, payroll, and financial records. Questions like headcount by department and tenure distribution. And actions like “increase base salaries of everyone who meets this condition by 10%,” onboard a new hire through the checklist, schedule a termination and route its approvals, enter payment amounts into a pay run from a spreadsheet.

Every attempt either passed Rippling’s production correctness checks or failed. An attempt that never finished counted as a failure. That is a much harder grader than most published benchmarks use.

Three numbers run through the whole study, and they’re the only vocabulary you need:

• Pass rate. How often the model did the job correctly. No partial credit.
• Cost. What it cost to run the full set of tests once, in dollars.
• The slowest 10%. How long a task took on the bad days, not the average day. This is the number your customer actually feels.

The 15 models collapse to three real choices

Almost every model in the study is beaten outright by another one. What’s left is a three-way decision:

• Opus 4.6 if being right matters most, and it’s nearly a tie. 91.0% pass rate, $1,453, 154 seconds on the slowest 10%. The next model down, GPT-5.5 med, scored 89.5% for $1,435. That’s an $18 difference in cost, which is nothing, and a 1.5-point difference in accuracy that sits inside the margin of error on a test this size. Add the five months Rippling spent tuning its setup specifically for Opus 4.6, worth a point or two by MacInnis’s own estimate, and the two are a wash. What actually separates them is speed, and that goes to Opus 4.6.
• GPT-5.5 low if speed matters most. 88.8%, $1,308, 130 seconds on the slowest 10%. Cheaper and faster than Opus 4.6, giving up 2.2 points of accuracy. Worth noting that OpenAI’s own two settings of the same model land on either side of Opus 4.6, so the setting mattered more than the brand.
• GLM 5.2 if speed doesn’t matter at all. 88.7%, $621, 243 seconds on the slowest 10%. Same accuracy as GPT-5.5 low within a tenth of a point, half the price, and much slower. Every overnight job, backfill, and data enrichment run belongs on something like this.

The expensive models don’t make that list. Fable 5 matched GPT-5.5 low’s accuracy at 3.3x the price and twice the wait. Opus 5 was beaten on both price and speed by Grok 4.5, which scored 0.2 points behind it for 68% less.

Seven takeaways for B2B founders:

#1. Quality flattened at the top. Price did not.

Seven models landed between 88.5% and 89.5%. That is a one-point spread. Add the leader at 91.0% and the whole competitive group is 2.5 points wide.

Here’s what those models cost:

GLM 5.2 and Fable 5 are one tenth of a point apart in accuracy. One costs seven times the other.

If your process is “use whatever the big lab just shipped,” you’re paying a 3x to 7x premium for a difference your customers cannot detect.

#2. The newest model was not the best model

Opus 4.6 beat both of the newer Anthropic models on accuracy, cost 42% less than Opus 5, and cost a third of Fable 5. Fable 5 was the most expensive model tested and finished fifth.

Version numbers are a lab’s internal accounting, not a promise about your business. Nobody at Rippling would have known this without running the test.

Then MacInnis proved it a second time. He re-ran the same 2,100 attempts on Grok 4.6 and posted the result: accuracy dropped from 87.3% to 85.9%, and typical response time nearly doubled, from 71 seconds to 131. The newer model was worse and slower at the same work.

One caution on reading those two numbers. The 87.3% he quotes for Grok 4.5 doesn’t match the 89.1% in his published table, because it came from a different run on a different day. Only compare results measured in the same run. A model that “improved two points” against a number you measured last quarter has told you nothing.

So when a new model ships, the default move is not to upgrade. It’s to re-run your own tests. Sometimes the answer is stay.

#3. Your own tuning is worth about as much as a whole model generation

Rippling spent five months tuning its instructions and tools around Opus 4.6. MacInnis estimates that work is worth a point or two of accuracy. The entire spread across the leading group is 2.5 points.

So the tuning is worth roughly the gap between the best model in the study and the cheapest one. Every other model ran with no tuning at all, which means their scores are floors, not ceilings.

Two things follow from that. First, your test suite and your instructions are a real asset, and they compound. Second, they’re also a switching cost you’re building against yourself, which is why you want the work to sit outside any one vendor’s model rather than baked into it.

Rippling deliberately kept its setup plain: one model, no routing logic, and two general-purpose tools that do all the work. If a bunch of custom plumbing were making the decisions, they’d be measuring the plumbing instead of the model.

#4. Cheap models are not cheap because they do less work

Models are billed by the volume of text they read and write, measured in tokens. Grok 4.5 used 601,000 per task. Opus 5 used 599,000. Effectively the same amount of work. Grok cost $791. Opus 5 cost $2,509.

The whole difference is the price list. Grok charges $2 per million read and $6 per million written. Opus 5 charges $5 and $25. Fable 5 charges $10 and $50.

The cheaper models actually did more back-and-forth, not less. They aren’t cutting corners. They’re just priced differently.

Which means the metric to watch is dollars, not efficiency. With prices and usage both moving constantly, the invoice is the only number that stays honest.

#5. Model choice is a margin decision, not an engineering one

Run it against your own P&L. If AI usage is your biggest variable cost, a 3x swing in that bill moves gross margin by double digits. Same product, same accuracy, same customer experience.

That makes this the cheapest margin improvement available to most AI-native B2B companies. No new features, no new headcount, no roadmap change. It’s a setting.

It also means someone in finance should own the number and review it on the same cadence as any other cost line.

#6. Speed and price are separate purchases

Two models with the same accuracy:

• GPT-5.5 low: 88.8%, $1,308, and fast
• GLM 5.2: 88.7%, $621, and slow

One is roughly 2.7x quicker. The other is half the price. Nothing is both.

And for anything a customer is waiting on, the number to watch is the slowest 10%, not the average. The average is what founders quote in board decks. The slowest 10% is what a customer hits on the day they’re already annoyed.

GLM 5.2 saves $687 against GPT-5.5 low at identical accuracy, and turns a 130-second worst case into a 243-second one. In a live support chat, four minutes of silence is an abandoned session and a ticket you now handle twice. That costs more than the $687.

So the cheap-model argument flips depending on the job. For work nobody is waiting on, price wins and GLM takes it outright. For live customer work, speed wins first and price second, and the winners are models nobody was calling a bargain. Opus 4.6 is the example: top accuracy, respectable speed, middle of the pack on price. It looks expensive on a cost chart and correct on a speed chart.

Most B2B companies pick one model and run everything through it. The split worth making: