API Reference

Evaluations

Runs, comparisons, and the real scoring engine.

GET/api/evaluations

List all runs, newest first. results is stripped to keep the payload small — fetch a single run for full detail.

{
  "runs": [
    {
      "id": "run_005",
      "name": "Support QA — gpt-4o regression check",
      "datasetId": "ds_001",
      "model": "gpt-4o",
      "status": "passed",
      "metricIds": ["answer-relevancy", "faithfulness", "hallucination", "contextual-precision"],
      "passRate": 0.75,
      "createdAt": "2026-07-15T22:22:00.000Z",
      "durationMs": 21810,
      "resultCount": 8
    }
  ]
}

GET/api/evaluations/:runId

Get a run with full per-test-case results, plus its parent dataset.

{
  "run": {
    "id": "run_005",
    "results": [
      {
        "id": "res_5001",
        "testCaseId": "case_001",
        "input": "What is your refund policy?",
        "expectedOutput": "Refunds are available within 30 days of purchase with a valid receipt.",
        "actualOutput": "Refunds are available within 30 days of purchase with a valid receipt.",
        "passed": true,
        "scores": [
          { "metricId": "answer-relevancy", "score": 0.91, "passed": true, "reason": "Output meets the answer relevancy threshold of 0.5." }
        ]
      }
    ]
  },
  "dataset": { "id": "ds_001", "name": "Customer Support QA" }
}

404 if runId doesn't match any run.

POST/api/evaluations/run

Computes a brand-new run for real — every test case in the dataset is scored by the actual NLP engine in natural + leo-profanity, not a stub. This is the only write endpoint under Evaluations.

Body field	Type	Required
datasetId	string	yes
model	string	no — defaults to "gpt-4o" (label only, no real model call)
promptTemplate	string	no — Mustache template; blank uses raw test case input

curl -X POST https://preview.devx-ai.xyz/api/evaluations/run \
  -H "Content-Type: application/json" \
  -d '{
    "datasetId": "ds_001",
    "model": "claude-sonnet-5",
    "promptTemplate": "Context: {{context}}\nQuestion: {{question}}\nAnswer concisely."
  }'

Returns 201 with the full new run, structured identically to GET /api/evaluations/:runId. 400 if datasetId is missing/unknown or the dataset has no test cases.

What's real, what's simulated

The prompt rendering and every metric score are computed for real from whatever text comes out. There's no live model call behind it though — the "actual output" per test case is a simulated completion that varies in quality run to run, so scores are genuinely different each time you call this endpoint, not staged.

GET/api/evaluations/compare

Diff two runs by shared testCaseId.

Query param	Required
a	yes — baseline run id
b	yes — candidate run id

curl "https://preview.devx-ai.xyz/api/evaluations/compare?a=run_001&b=run_005"

{
  "comparison": {
    "runAId": "run_001",
    "runBId": "run_005",
    "runAPassRate": 1,
    "runBPassRate": 0.75,
    "passRateDelta": -0.25,
    "testCases": [
      {
        "testCaseId": "case_004",
        "input": "Do you support single sign-on (SSO)?",
        "runAPassed": true,
        "runBPassed": true,
        "scoreDeltas": [
          { "metricId": "answer-relevancy", "runAScore": 0.73, "runBScore": 0.65, "delta": -0.08 }
        ]
      }
    ]
  }
}

A test case only appears once even if it exists in both runs; if it's only in one run, the other side's fields are null. 400 if a or b is missing; 404 if either run id doesn't exist.

Data model

Field	Type	Notes
status	"passed" \| "failed"	passed when passRate ≥ 0.6
metricIds	MetricId[]	The four metrics scored on every result — see Metrics
passRate	number (0–1)	Share of results where every metric passed
durationMs	number	Real elapsed time for POST /run; historical for seeded runs