devx-ai is a demo LLM tooling workspace, built to explore what a unified evaluation, observability, and prompt-ops surface could look like — the kind of product space occupied by tools like Confident AI, LangSmith, Langfuse, Braintrust, Arize, and Helicone. Rather than bolting evals, tracing, and a playground together as separate tools, devx-ai puts them behind one sidebar and one data model.
It covers the full loop teams actually use day to day: build a golden dataset, run it against a model with a set of metrics, compare two runs to catch regressions, version the prompt that produced the output, review flagged traces by hand, get alerted when a threshold slips, and keep an eye on usage and cost.
The evaluation metrics — Answer Relevancy, Faithfulness, Hallucination, Contextual Precision, Contextual Recall, and Toxicity — mirror the naming, default 0.5 threshold, and pass/fail direction used by DeepEval, Confident AI's open-source LLM evaluation framework. Metrics like hallucination and toxicity measure "amount of a bad thing," so they pass when the score is at or below the threshold, while relevancy and faithfulness pass at or above it — same as DeepEval. When you trigger a run yourself, those scores are computed for real: token-overlap and Jaro-Winkler string distance via the natural NLP library, and profanity detection via leo-profanity — no external API calls.
Everything you see in the preview — datasets, evaluation runs, traces, prompts, annotations, and alerts — is seeded, in-memory mock data. There's no database and no authentication: the app is meant to be explored freely, and the in-app actions (adding a test case, running the playground, toggling an alert rule) persist only for the life of the running server.
The preview is open — click through every section without creating an account.