Alerts
Alert rules define a threshold on a metric; when a run or trace breaches it, an event is recorded in the triggered-events feed.
Rule fields
| Field | Description |
|---|---|
| metric | pass_rate, latency_ms, cost_usd, or error_rate. |
| comparator | lt (less than) or gt (greater than). |
| threshold | The numeric value that trips the rule. |
| scope | What the rule applies to, e.g. global. |
| enabled | Whether the rule is currently active. |
Seeded rules
The preview ships with four rules: pass rate below 70%, latency above 3 seconds, trace cost above $0.01, and error rate above 10% (disabled by default). Toggling a rule's status on the Alerts page calls the API immediately — it's not just a visual toggle.
Triggered events
Each event records which rule fired, the actual value that breached the threshold, a human-readable message, and a link back to the offending evaluation run or trace.
Designing a threshold that's actually useful
- Too tight and every run trips it — the alert stops meaning anything and gets ignored. Too loose and it never fires until something is already badly broken.
- A reasonable starting point: look at your last several runs' actual pass rate/latency/cost in Analytics, and set the threshold a bit worse than your typical range — tight enough to catch a real regression, loose enough to survive normal variance.
- Disabling a rule (rather than deleting it) keeps its definition around for when you want it back — that's why
enabledis a separate field from the rule's existence.
Alerts don't replace Compare
An alert tells you that a run or trace crossed a line. It doesn't tell you which specific test case caused it or what changed — for that, pair a triggered alert with Compare against the last known-good run.