AI工程师必知的11种LLM评估方法

11 LLM evaluation methods AI engineers must know:

(bookmark this)

Two eval metrics can rank the same two models in opposite orders, and neither one is wrong.

A model that paraphrases the reference can score near zero on BLEU and near the top on BERTScore for the exact same output.

Neither metric is wrong because one is measuring wording and the other is capturing meaning.

This is why LLM evaluation is fragmented into several methods, depicted in the visual below and grouped by what each one assumes:

> Reference-based (ground truth exists):

• BLEU - ROUGE - BERTScore

> Judge-based (no ground truth):

• G-Eval - LLM-as-Judge - LLM juries

> Human and deterministic:

• Human eval - DAG

> Built for agents:

• Trajectory accuracy - Multi-turn eval

> Run as a gate:

• Safety eval

To use them in practice, most of these metrics are already implemented in Opik, which is open source (20k+ stars) and runs them over traced production data. You can start using them in a few lines of code.

GitHub repo: https://github.com/comet-ml/opik

(don’t forget to star it ⭐️)

That said, metrics only point at the failing case.

The rest of the work is still done manually, like inspecting the trace to see where the span went wrong, editing a prompt or a tool description, re-running, and checking that the fix did not break anything else.

My co-founder wrote a walkthrough (with code) that automates this loop using Opik.

It explains the full lifecycle where a failing trace gets diagnosed, the fix runs against the exact input that failed, and that input stays in the eval set as a regression case so it does not recur.

Read it below.