Concepts
Compare
Compare diffs two evaluation runs — a baseline (A) and a candidate (B) — test case by test case, so you can catch regressions before they ship.
What gets compared
- Overall pass rate for each run, and the delta between them.
- Pass/fail status per test case, in both runs.
- The score delta for each metric, per test case.
Reading the deltas
A green up-arrow means the candidate run improved on that metric; a red down-arrow means it regressed — and the arrow direction is metric-aware. Because Hallucination and Toxicity are "lower is better," a decreasing score on those two shows as an improvement (green), not a regression, even though the number went down.
Typical use
Run your dataset against the current production prompt (baseline), then again after editing the prompt or swapping models (candidate). Compare tells you exactly which test cases got better, which got worse, and on which specific metric — much faster than re-reading every output by hand.
When test cases don't line up
Comparison is keyed on testCaseId, not row order. If A and B ran different datasets — or the same dataset after a test case was added or removed — a case that only exists in one run shows a dash (—) on the other side instead of a false regression.
What Compare can't tell you
- It won't tell you why a metric moved — for that, open both runs' detail pages and read the actual vs. expected output for the specific test case that regressed.
- A flat 0.00 delta on every metric usually means the two runs produced byte-identical output for that case (the real scorer is deterministic — same text always scores the same), not that the comparison is broken.
- It only compares metric scores and pass/fail — cost, latency, and token usage live in Traces and Analytics instead.