Concepts
Datasets
A dataset is a named collection of golden test cases — the ground truth an evaluation run is scored against.
Test case fields
| Field | Description |
|---|---|
| input | The question or prompt sent to the model. |
| expectedOutput | The reference answer used to judge correctness. |
| context | Retrieved passages the answer should be grounded in (for RAG-style metrics). |
| tags | Free-form labels for filtering, e.g. billing, auth, enterprise. |
Creating and editing
From a dataset's detail page, use "Add test case" to append a new row — it's immediately available to any evaluation run against that dataset. Rows can be removed with the trash icon.
How datasets relate to evaluations
- An evaluation run always targets exactly one dataset.
- Every test case in the dataset gets its own result row in the run.
- A dataset can be reused across many runs — different models, different prompt versions.
What makes a good test case
- Expected output should be specific. A vague reference answer ("something about refunds") makes every similarity-based metric noisy — see Metrics for why exact phrasing matters to the real scorer.
- Context should contain the actual grounding. Faithfulness and Hallucination are only meaningful if
contextgenuinely contains what a correct answer would draw from — an emptycontextarray falls back to comparing against the expected output instead. - Tags are worth the extra thirty seconds. They're how a large dataset stays navigable once it's grown past a dozen cases, and how you'd eventually filter "just the billing cases" out of a broader set.
Growing a dataset over time
The most common way real teams grow a dataset isn't writing cases up front — it's pulling them from production. A trace that looked wrong, or got a "bad" rating in Annotations, is a strong candidate for a new test case: take the real input, write down what the answer should have been, and the next evaluation run will catch it if it regresses again.