Concepts

Metrics

Metrics score a single test case result on a 0–1 scale. Naming, default thresholds, and pass/fail direction mirror DeepEval, Confident AI's open-source LLM evaluation framework.

Available metrics

Metric	Measures	Default threshold	Passes when
Answer Relevancy	How relevant the output is to the input.	0.5	score ≥ 0.5
Faithfulness	Whether the output is factually consistent with the retrieved context.	0.5	score ≥ 0.5
Hallucination	Degree of fabricated information not present in the context.	0.5	score ≤ 0.5
Contextual Precision	Whether relevant context nodes rank above irrelevant ones.	0.5	score ≥ 0.5
Contextual Recall	How much of the expected output the retrieved context captures.	0.5	score ≥ 0.5
Toxicity	Whether the output contains toxic or harmful language.	0.5	score ≤ 0.5

Direction matters

Most metrics pass when the score is at or above the threshold — higher is better. But "amount of a bad thing" metrics, Hallucination and Toxicity, are inverted: a good output scores low, so they pass when the score is at or below the threshold. This is the same convention DeepEval uses.

Reading a result

Every result on an evaluation run's detail page shows a score per metric, a pass/fail badge, and a short reason string explaining whether the threshold was met. A test case passes overall only if every metric attached to the run passes.

How scores are actually computed

Runs you trigger yourself (see Evaluations) are scored with real, deterministic NLP — not a random number:

Answer Relevancy, Faithfulness, Contextual Precision/Recall — token-overlap (Dice coefficient over significant tokens) blended with Jaro-Winkler string distance, via the natural NLP library.
Hallucination — the share of the output's significant tokens that don't appear anywhere in the retrieved context (a real unsupported-content ratio).
Toxicity — the share of tokens flagged by leo-profanity's profanity dictionary.

No API key or external call is involved — everything runs locally in the same request that creates the run. Seeded/historical runs (present before you trigger any of your own) keep their original fixture values so the Dashboard has data to show on first load.

The similarity formula

Answer Relevancy, Faithfulness, and both Contextual metrics all reduce to the same similarity(a, b) function, blended two ways:

similarity(a, b) =
  0.65 × diceCoefficient(significantTokens(a), significantTokens(b))
+ 0.35 × jaroWinklerDistance(a, b)

significantTokens lowercases, tokenizes, and drops stopwords and tokens ≤ 2 characters — so "the", "a", "is" don't inflate overlap.
The Dice coefficient rewards exact shared vocabulary; Jaro-Winkler rewards near-miss spelling and short paraphrases that don't share many tokens. Blending both catches more real cases than either alone.

Limitations of heuristic scoring

This is real, deterministic text similarity — not semantic understanding. Two answers that are correct but phrased with completely different vocabulary can score lower than they "should," and a wrong answer that happens to reuse a lot of the expected phrasing can score higher than it should. Production DeepEval typically uses an LLM as the judge for relevancy/faithfulness; devx-ai substitutes fast, free, local NLP so the preview needs no API key. Treat scores as a strong signal, not ground truth.