ML模型生产测试的4种策略

4 strategies to test ML models in production:

(a popular ML interview question; bookmark this)

Despite rigorously testing an ML model locally, it could still be a terrible idea to instantly replace the previous model with the new model.

This is because it is difficult to replicate the exact production environment and conditions locally, and justify success with val/test accuracies.

A more reliable strategy is to test the model in production (yes, on real-world incoming data).

While this might sound risky, ML teams do it all the time, and it isn’t that complicated.

Note: > Legacy model: The existing model. > Candidate model: The new model.

The visual below depicts 4 common techniques to do so.

1) A/B testing

Distribute the incoming requests non-uniformly between the legacy model and the candidate model. This limits the exposure of the candidate model to avoid any potential risks.

So, say, 10% requests go to the candidate model, and the rest are still served by the legacy model.

2) Canary testing

A/B testing typically affects all users since it randomly distributes “traffic” to either model (irrespective of the user).

In canary testing, the candidate model is exposed to a small subset of users in production and gradually rolled out to more users if its metrics signal success.

3) Interleaved testing

This involves mixing the predictions of multiple models in the response.

For instance, in Amazon's recommendation engine, some recommendations can come from the legacy model, while some can be produced by the candidate model.

Alongside, we can log the downstream success metrics (click-rate, watch-time, reported-as-not-useful-recommendation, etc.) for comparison later.

4) Shadow testing

All of the above techniques affect some (or all) users.

Shadow testing (or dark launches) lets us test a new model in a production environment without affecting the user experience.

The candidate model is deployed alongside the existing legacy model and serves requests like the legacy model.

However, the output is not sent back to the user. Instead, the output is logged for later use to benchmark its performance against the legacy model.

We explicitly deploy the candidate model instead of testing offline because the exact production environment can be difficult to replicate offline.

Shadow testing offers risk-free testing of the candidate model in a production environment.

But one caveat is that you can’t measure user-facing metrics in shadow testing.

Since the candidate model’s predictions are never shown to users, you don’t get real engagement data, like clicks, watch time, or conversions.

And this is exactly how top ML teams at Netflix, Amazon, and Google roll out new models safely.

They never flip the switch all at once, but rather first test in production, observe, compare, and then promote the model to 100% traffic.

Of course, alongside all this, you would also measure latency, throughput, resource usage, and downstream success metrics.

A model that’s 2% more accurate but 3× slower isn't desired from a user experience standpoint.

That said, all this is downstream of one thing though. Production testing only means something if the offline evaluation underneath it was honest.

If the test set gets touched during model selection, it leaks into your choices, and the score is inflated before production testing even starts.

I wrote an article that covers several misconceptions engineers have about using train, validation, and test sets. It also covers the way they're actually meant to be used.

Read it below.