Concepts

Evaluations

An evaluation run scores a model's outputs against a dataset using a fixed set of metrics.

Anatomy of a run

Dataset — the golden test cases being run.
Model — which model produced the outputs being scored.
Metrics — the scoring criteria applied to every result.
Results — one per test case: actual output, per-metric scores, pass/fail.
Pass rate — the share of test cases that passed every metric.

Pass rate and status

A run's overall status is passed when its pass rate is 60% or higher, and failed otherwise. This threshold is separate from the per-metric thresholds — it's a run-level summary judgment, useful for a quick scan on the Dashboard and Evaluations list.

Where runs come from

Five runs across the three datasets are seeded at startup, covering a mix of models. On top of that, the Evaluations page has a real "New run" form — pick a dataset and a model, optionally paste a prompt template, and it computes an actual new run on demand using the real scoring engine described in Metrics.

Pick a dataset and a model.
Optionally paste a prompt template with {{question}} and {{context}} — see Prompts for how rendering works.
Submit — every test case in the dataset gets scored and you land on the new run's detail page.

Not a real model call

There's still no live LLM behind this — the "actual output" per test case is a simulated completion that varies in quality run to run. What's real is the interpolation and the scoring: the same output text, scored twice, always produces the same metric scores.

Next: comparing runs

Once you have two runs against the same dataset — say, before and after a prompt change — head to Compare to see exactly what moved.

Reading the run list

Duration is real elapsed time for runs you trigger — usually single-digit milliseconds, since scoring is local NLP, not a network call. Seeded runs carry a historical duration that simulates what a real model-backed pipeline would take.
Test cases equals the dataset's size at the moment the run happened — adding a case to a dataset afterward doesn't retroactively change older runs.
A run's model field is a label you choose when triggering it; nothing enforces that the "actual output" text was really produced by that model.

One run vs. many small ones

There's no cost or rate limit here, so it's cheap to over-trigger runs while iterating on a prompt template. In a real pipeline this maps to CI: run the full dataset on every prompt or model change, keep the history, and use Compare instead of eyeballing pass rates across a growing list.