Annotations

Annotations are human judgments attached to a trace or an evaluation result — a thumbs up/down rating plus an optional note.

Fields

• targetType — trace or evaluation_result.
• targetId — which trace or result the annotation applies to.
• rating — good or bad.
• note — free-text context for why.
• annotator — who left the annotation.

Why it matters

Automated metrics catch a lot, but not everything — tone, subtle factual drift, or "technically correct but unhelpful" answers often need a human eye. The annotation feed is where that review lives: pick a trace, rate it, leave a note for whoever looks at it next.

In the preview

The Annotations page lets you rate any trace directly. Evaluation-result annotations are seeded (you can see a mix of good and bad ratings on past runs) but aren't creatable from the UI in this preview — the underlying API supports it, see API Reference.

Writing a note worth reading later

A rating alone tells you that something was wrong; the note is what tells the next person (or your future self) what to actually fix. "Bad" with no note just means someone has to re-derive the problem from scratch by re-reading the output.

• Say what's specifically wrong — tone, missing detail, wrong policy cited — not just "bad."
• If it points at a fixable prompt issue, say so — that's a direct lead into a new Prompts version.
• If it points at a gap in the golden dataset, that's a lead into adding a new test case.

Annotations as a feedback loop

The most useful pattern isn't rating in isolation — it's closing the loop: a "bad" trace becomes a new test case with the correct expected output, which becomes part of the next evaluation run, which either confirms the fix or catches a regression. Metrics catch what you already thought to test for; annotations are how new things get added to that set.