devx-aidocs
HomePreview
Example

Catching a regression before it ships

A walkthrough of the workflow devx-ai is built around: change a prompt, re-run the evaluation, and confirm nothing got worse before shipping it.

Scenario

You've tightened the "Support Answer Assistant" prompt to be more concise. Before pushing it to production, you want to confirm it didn't quietly break any answers that were working fine before.

Steps

  1. Open Evaluations and note the most recent passing run against the "Customer Support QA" dataset — this is your baseline.
  2. Open Prompts → Support Answer Assistant, add a new version with the tightened wording, tagged staging.
  3. Run a fresh evaluation of "Customer Support QA" using the new prompt version (in a real deployment, this would be an evaluation pipeline call using the new template).
  4. Open Compare, set the baseline run as A and the new run as B.
  5. Scan the per-test-case delta table for any red (regressed) cells — pay close attention to Faithfulness and Hallucination, which are the metrics most sensitive to a prompt getting too terse.
What you're looking for
A regression is a test case that passed on A but failed on B, or a metric that moved in the wrong direction even if the pass/fail status didn't change — a shrinking margin today is a failure waiting to happen next week.

If you find a regression

Don't promote the new prompt version to production yet. Open the specific failing test case in the run detail page, read the actual output next to the expected one, and adjust the prompt again — then repeat the compare.