用AI代理跑50万封邮件的供应商交付性对比与迁移方案

We’ve run vendor bake-offs at SaaStr for as long as we’ve bought software. Everyone has.

They used to look about the same every time. A few demos. A two-week trial. Someone on the team sends a few hundred test emails, checks the dashboards, and builds a spreadsheet. The decision then mostly comes down to which rep was most responsive and what the team already knew how to use.

A few hundred sends, judged on each vendor’s own reporting, by people with other jobs, doesn’t tell you much.

This week we compared email deliverability across three vendors and more than 500,000 emails. 10K, our AI VP of Marketing, ran the whole thing. It caught every issue, ran the multivariate diagnostics, and wrote a recommendation clear enough that we acted on it the same day.

What a Bake-Off Used to Cost Us

A proper deliverability comparison takes a lot of people time:

• Building matched audience groups so the comparison is fair
• Sending identical content through each vendor at the same time
• Pulling delivery, open, click, bounce, complaint and unsubscribe data out of three dashboards that each define those terms differently
• Checking whether “delivered” actually means delivered
• Filtering security-scanner clicks out of human clicks
• Repeating all of it at enough volume to trust the result

At our size, no human team was going to do that across 500,000 emails. We’d have done a few hundred test sends, picked a winner, and lived with it for three years.

Same Campaign, Matched Groups: 50.5% Opens vs. 21.0%

The cleanest comparison was a split 10K set up on our AI Day campaign: two matched groups of about 2,300 people each, identical content, sent the same morning. One went through our incumbent sender and the other through a challenger.

The challenger got 2.4x the opens and 2.9x the clicks. With the same audience, email and send time, the gap comes from how each vendor delivered the email.

10K listed the limits before we could get excited. It’s one campaign at about 2,300 per side. Opens are noisy. The raw click counts include security scanners. Clicks moved in the same direction as opens, which made the gap hard to explain away.

For our sponsors, this mattered a lot. On days when everything was working, every newsletter send through the incumbent had been reaching them with less than half the engagement.

The Incumbent Reported “Completed” on 80,708 Failed Sends

On one Daily send, the incumbent’s flow showed “Completed.” 10K queried the step-level records underneath that label and found 80,708 attempts and 80,708 errors. Zero emails went out. The vendor’s status page showed green the whole time.

10K then ran four diagnostic sends, changing one variable each time:

• A different sender
• Tracking turned off
• Different subscription settings
• Minimal plain-text content

All four failed the same way. That ruled out our content, configuration and list, and put the failure on the vendor’s transport layer.

In a two-week trial, someone on our team would have seen “Completed” on the dashboard and moved on.

10K Scored the Vendors on 7 Criteria

10K wouldn’t pick a winner on opens. Its scorecard:

• Completed delivery, confirmed at the step level
• Inbox placement
• Spam complaints
• Unsubscribes
• Human clicks, with scanner traffic filtered out
• Sponsor reporting: whether we can prove to a sponsor that their placement went out and got engagement
• Operator effort: how much human time each vendor takes in consent reconciliation, audience prep, schedule checks and reporting

We had never put operator effort in a bake-off spreadsheet. It turned out to be one of the biggest differences between the vendors.

The Recommendation Fit on One Page

10K’s plan:

• Move core newsletter sends to the winning vendor now, ramping over days. Roughly 20K, then 40K, then full volume, stepping up only if bounces, complaints and unsubscribes hold.
• Confirm unsubscribe sync in both directions before scaling. Unsubscribes from the new sender write back to our CRM consent records, and CRM opt-outs suppress in the new sender. A gap here turns a sending switch into a compliance problem.
• Map every other workflow that depends on the incumbent before canceling anything.
• Report to sponsors from the new provider’s own logs.
• Set acceptance criteria before the incumbent sends anything sponsor-backed again: a root cause, an accounting of every outstanding attempt, and an operator-only test confirmed at the step level.
• Don’t renew.

I wanted to move all 80,000+ subscribers over immediately. 10K pushed back: a clean 5,000-person test doesn’t show a sender can carry the full list, and moving everyone at once would put deliverability at risk. We went with the ramp.

A Quarter-Long Project Took a Few Days

Every technique here is standard practice for a good deliverability team: matched splits, step-level verification, isolating one variable at a time, scoring on more than opens.

We have three humans at SaaStr and no deliverability team. A test at this depth would have meant hiring a consultant for a quarter. 10K ran it in a few days alongside its other work.

That changes how we buy software:

• We can keep vendors in a running comparison instead of re-evaluating every three years.
• We test at production volume. 500,000 emails surfaces problems a few hundred test sends never will.
• We check vendor reporting against the underlying records. Status labels, dashboards and status pages all got cross-checked.
• Switching costs less. The agent that ran the test can stand up the alternative and run the migration ramp.

Every Core Vendor in Our Stack Is Now in a Running Bake-Off

If you sell B2B software, your customers’ agents can now run the evaluation your sales team used to steer. They’ll test at real volume, query your step-level data, and score you on how much of their team’s time you take. Incumbency protects you far less when a challenger can be tested against you every week.

If you’re buying, have your agent run your next bake-off, and give it the scorecard up front: completed delivery confirmed at the source, human engagement, and operator effort.

We’ve always done vendor bake-offs at SaaStr. This week was the first one where I trusted the result completely.