Setting up an alert on pass rate

A walkthrough of adding a threshold rule so a pass-rate drop shows up automatically instead of requiring someone to notice it on the Dashboard.

Scenario

You want to be notified whenever an evaluation run's pass rate drops below 80% — tighter than the seeded 70% rule — for a specific dataset you're actively hardening.

Steps

1. Open Alerts to see the existing rules and understand the current thresholds.
2. Create a new rule via the API (the preview UI supports viewing and toggling rules; creating one uses the same endpoint the UI would call):

curl -X POST https://preview.devx-ai.xyz/api/alerts/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support QA pass rate below 80%",
    "metric": "pass_rate",
    "comparator": "lt",
    "threshold": 0.8,
    "scope": "dataset:ds_001"
  }'

1. The new rule appears in the Rules table, enabled by default.
2. Toggle any rule off directly from the Alerts page — the switch calls PATCH /api/alerts/rules/:ruleId immediately.
3. Once a run breaches the threshold, it shows up in the triggered-events feed with a link back to the run.