Documentation
Introduction to devx-ai
devx-ai is a workspace for evaluating, tracing, and testing LLM applications — golden datasets, metric-driven evaluations, regression-catching run comparisons, versioned prompts, production tracing, human review, and threshold alerts, all in one place.
Why devx-ai
Most teams building LLM features end up stitching together a handful of separate tools — a spreadsheet of test prompts, a notebook for scoring outputs, a logging dashboard for production traces, a Slack channel for "hey does this response look right." devx-ai puts all of that behind one sidebar and one data model, following the same workflow teams already use to harden a model-backed feature before shipping it.
What's inside
- Datasets — golden test cases: input, expected output, and retrieval context.
- Evaluations — run a dataset against a model and score it with metrics.
- Compare — diff two evaluation runs to catch regressions before they ship.
- Prompts — version prompt templates across draft, staging, and production.
- Traces — inspect individual LLM calls: prompt, completion, latency, cost, spans.
- Annotations — a human review queue for labeling traces and eval results.
- Alerts — threshold monitors on pass rate, latency, cost, and error rate.
- Analytics — usage and cost breakdowns by model, route, and time.
- Playground — test prompts against models and compare outputs interactively.
This is a demo workspace
Everything in the preview runs on seeded, in-memory mock data — there's no database and no authentication. In-app actions (adding a test case, running the playground, toggling an alert rule) persist only for the life of the running server. See Getting Started for how to explore it.
Where to go next
- Getting Started — a tour of the app, section by section.
- Evaluations — how scoring and pass/fail work.
- API Reference — every endpoint the app exposes.