Verdict
Verdict’s server lets you manage rubrics, evaluate and compare subjects, run human-review/calibration workflows, and use the public hub.
Validate or create rubrics (criteria, weights, scales, anchors, judge types, aggregation, gates, grades, pass threshold) and list stored rubrics.
Evaluate a subject against a stored or inline rubric over text, JSON, files, URLs (optionally with screenshots), images, or YouTube transcripts; get score, grade, pass/fail, per-criterion rationales, and pending human tasks.
Compare/rank multiple subjects against one rubric.
Fetch evaluations, submit human scores, list human tasks, and re-aggregate scores without re-running judges.
Check a rubric against a human-labeled gold set and produce calibration reports.
Search/read published hub cards and fork them into local rubrics.
Uses Cloudflare Workers AI models such as Clef and Clef-flash as decision judges, sending rubric questions and collecting model answers for comparison.
Uses OpenAI's Decisions API models, such as gpt-6-luna, as decision judges to answer typed rubric questions with probabilities.
Accepts YouTube URLs as inputs and extracts the video transcript for rubric judging.
Verdict — one rubric, every decision model
Write the rubric once. Run it on Jev, Clef, OpenAI's Decisions API, Gemini's and any other decision model, and see where they disagree on your own cases.
Decision models answer typed questions (yes or no, one of these, a level on a scale) with probabilities instead of prose, in a fraction of a second. Every vendor publishes benchmarks; none of them is about your cases, and each vendor wants its questions in its own shape. Verdict turns a rubric — criteria, scales, levels — into the questions each model takes, sends the same questions to all of them, and puts the answers side by side.
npm install -g @verdict-ai/cli
verdict run -r examples/banking77/intent.yaml -d examples/banking77/cases.jsonl --models jev,clef,clef-flash,openrouter,openai,geminiTransfer intent × cases.jsonl: 80 case(s), 6 model(s), the same questions to each
model answered by agreed with labels failed asked a human input tokens calls time
jev typesafe/jev-1.14.0 66/80 (83%) 0 0 44,560 80 2.1 s
clef cloudflare/clef 70/80 (88%) 0 0 31,259 80 6.9 s
clef-flash cloudflare/clef-flash 73/80 (91%) 0 0 31,259 80 4.5 s
openrouter openrouter/typesafe/jev-1.13-20260917 67/80 (84%) 0 0 44,560 80 3.9 s
openai openai/gpt-6-luna 69/80 (86%) 0 0 25,254 80 7.1 s
gemini gemini/gemini-decision-1 68/80 (85%) 0 0 25,364 80 7.1 s
the models gave one answer on 70 of 80 question(s) they all answered; they differ on 10:
…A real run (2026-11-12) of eighty customer messages from a public dataset, on six models: examples/banking77 has the rubric, the cases, the whole output and how to read it. It is not a ranking: eighty cases separate the models by a handful of answers. The ten cases where they differ are the ones worth reading, and on your cases they will be different ones.
Try it
Node 22 or newer. No key is needed for the first run:
npm install -g @verdict-ai/cli
git clone https://github.com/Ustay-coder/verdict && cd verdict/examples/banking77
verdict run -r intent.yaml -d cases.jsonl --models mock # a stand-in model: shows the flow, not real answersWith a key for any one service, name its model instead (TYPESAFE_API_KEY for jev; CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN for clef; OPENROUTER_API_KEY; OPENAI_API_KEY). A model that cannot answer keeps its row and says why.
Then your own cases:
verdict rubric init rubric.yaml # a starter rubric to edit
verdict run -r rubric.yaml -d cases.jsonl --models jev,clef # cases.jsonl: one {"id", "inputs", "expected"} per line
verdict diff --base run_… --head run_… # any two runs, case by caseThe rubric file is not edited: --models re-points its decision judges, one run per model, and every run is kept. Getting started walks through it; Decision models explains the table.
Related MCP server: Hunch MCP
Models and inputs
Name it | Model | Service | Checked against the live service | Pictures and PDFs |
| Jev | TypeSafe | Yes | No, text only |
| Clef | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| Clef-flash | Cloudflare Workers AI | Yes | Pictures, up to 4 |
| For example | OpenRouter | Yes | No |
| OpenAI's Decisions API, for example | OpenAI | Yes | Pictures |
| Gemini Decision 1 | Google (Gemini API) | Yes | Pictures and PDFs |
| Any endpoint that takes the same request | Yours | You check it | No |
Input | With | What the judge receives |
Text |
| the text |
File | a | its text; a PDF itself for a model that reads PDFs, pictures of its pages for one that takes pictures |
JSON |
| the document, checkable against a schema |
Web page | a URL | the page's visible text; with |
YouTube | a YouTube URL | the transcript |
Image | a | the picture, for models that take pictures |
Rules (benchmark), people (human) and prompted models (llm) can judge a criterion too. Keys, context sizes and what each scale becomes: Models and inputs.
Checked against the live services: Jev, Clef, Clef-flash, OpenRouter and OpenAI's Decisions API — what each was sent, what it answered and when is in docs/PROVIDERS.md.
Once you have chosen a model
The same rubric keeps working after the comparison:
Version it.
verdict version freezegives a rubric a content-addressed id;verdict diffshows what a change to the rubric did to each case, and a GitHub Action comments that on pull requests. Versions, forks and CIUse it from any machine. A Verdict server (in this repository, self-hosted) keeps your frozen versions and says which one each environment runs:
verdict push,verdict promote, thenverdict run -r <name> --env prodanywhere. The server never receives a case or an answer. Versions, forks and CIBring in people. Low-confidence answers can go to a human reviewer, and
verdict checkcompares a rubric with what people already decided. Human reviewCall it from code. A REST API that runs on Node or Cloudflare Workers, a TypeScript SDK, and an MCP server for agents. REST · SDK · MCP
Judge more than text. PDFs, web pages, YouTube transcripts, JSON and images; rules, people and prompted models as judges beside decision models; weighted scores, gates and grades. Everything Verdict does
Packages
npm install -g @verdict-ai/cli # the `verdict` command
npm install @verdict-ai/sdk # typed REST client
npm install @verdict-ai/core # the engine, embedded
npm install @verdict-ai/api @verdict-ai/db # self-hosted service and persistence
npx @verdict-ai/mcp # stdio tools for agentsDocumentation
docs.verdict-ai.dev: guides, the rubric reference, configuration, deployment and a cookbook of complete rubrics. What changed in each release: CHANGELOG.md.
Contributing
Found a bug, or a model that answers in a shape Verdict does not read? Open an issue with the command and what it printed. To build and test the code, see CONTRIBUTING.md; to report a security problem, SECURITY.md.
License
MIT. The cases in examples/banking77 are from BANKING77 by PolyAI, under CC BY 4.0.
Available Tools
14 toolscalibration_reportCalibration reportC
Compare decision confidence against human consensus on stored evaluations.
Inputs: rubricId (required), target, buckets.
check_against_humansCheck a rubric against human labelsA
Run a rubric over a gold set (subjects people already judged) and report how often Verdict reached the same verdict: overall pass/fail agreement, per-criterion agreement, how many subjects the judge handed to a human (low-confidence escalation), and every disagreement with a preview. rubricPath is a local YAML/JSON rubric file, re-read on every call; gold is a local JSON array or JSON-lines file of {id?, name, inputs, expected?: {criterion: label}, human?: {passed?, criteria?}, labelSource?, split?}; choice criteria are labelled with an option id, and unreadable labels are rejected before any judge runs. Local mode only.
Inputs: gold (required), rubric, rubricPath.
compareCompare / rankC
Evaluate several subjects against one rubric and return a ranking (leaderboard).
Inputs: rubric (required), subjects (required), idempotencyKey.
create_rubricCreate rubricB
Store a rubric and get back its id for reuse. Criteria need id/name; optional weight, scale, anchors (= rubric levels for decision models), judge ({type: decision|human|benchmark|llm|panel}; default decision/typesafe). Aggregation supports weighted_mean, weighted_sum, mean, median, min, max, geometric_mean, formula; gates; grades; passThreshold.
Inputs: rubric (required).
evaluateEvaluateA
Score a subject against a rubric. rubric is a stored rubric id or an inline rubric. Inputs can be text, json, file (path or base64), url (set screenshot: true to let vision judges such as Clef see the page), image (PNG/JPEG/WebP, path or base64), or youtube. Returns the total score, grade, pass/fail, per-criterion scores with rationales, and any pending human tasks.
Inputs: inputs (required), rubric (required), name, metadata, +1 more.
fork_rubricFork a hub cardA
Copy a published card into this server's rubrics (a new rubric id; the card is not changed) so evaluate can use it. The rubric's metadata records where it came from and its licence.
Inputs: ref (required), hub, name.
get_evaluationGet evaluationA
Fetch an evaluation by id (status, scores, rationales, pending human tasks).
Inputs: id (required).
hub_getRead a hub cardB
Read a published card: its question and answers, threshold, licence, lineage, author-reported agreement numbers, versions and some public cases. ref is hub:owner/slug or hub:owner/slug@rv_….
Inputs: ref (required), hub.
hub_searchSearch the hubA
Search published single-question judgment cards (rubrics with public test cases). Returns one result per card at its newest version, with a hub:owner/slug@rv_… ref for hub_get or fork_rubric.
Inputs: q, hub, type, limit, +3 more.
list_human_tasksList human tasksD
Criteria waiting on human raters.
Inputs: rater.
list_rubricsList rubricsB
List stored rubrics.
Inputs: no named parameters.
reaggregateRe-aggregateA
Recompute the total with a different aggregation (method, formula, weights via gates/grades/passThreshold) without re-running judges. Useful for what-if analysis.
Inputs: aggregation (required), evaluationId (required), persist.
submit_human_scoreSubmit human scoreB
Submit a human rater's score for a criterion that is waiting on a human. Score is a number on the criterion scale or an ordinal/binary label.
Inputs: score (required), raterId (required), criterionId (required), evaluationId (required), +1 more.
validate_rubricValidate rubricA
Validate a rubric (JSON object or YAML string) and return it with defaults filled in. Use before evaluate to catch mistakes.
Inputs: rubric (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 14 tools
Most tools target clearly distinct actions: rubric CRUD/validation, evaluation, multi-subject ranking, human task handling, hub browsing, and calibration. The only real overlap is between check_against_humans (run rubric over a local gold set) and calibration_report (compare decision confidence against human consensus on stored evaluations), which could be confused at a glance despite differing inputs.
There is a solid verb_noun core (get_evaluation, submit_human_score, validate_rubric, create_rubric, list_rubrics, fork_rubric), but it is mixed with bare verbs (evaluate, compare, reaggregate) and noun-first names (calibration_report, hub_search, hub_get). Readable overall, but no single predictable pattern.
Fourteen tools is well within the ideal range and each group (rubric management, evaluation, human tasks, calibration, hub) earns its place in a coherent workflow. Nothing feels padded or redundant at the surface level.
Rubric lifecycle is only partial: create/list/validate exist but there is no get-by-id, update, or delete for stored rubrics, and the hub surface supports search/get/fork but no publish. Evaluation and human-task workflows look complete, so agents can work around the gaps but with friction.
Maintenance
Related MCP Connectors
Governed decision engine: yes/no, score, choice and rank answers about cases, from past outcomes.
Typed AI decisions for agents: classify, route, screen evidence, score and batch-label text.
Score academic and professional writing against published criteria with three reference models.
Create and evaluate versioned decision models with reproducible results and declared assumptions.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables agents to evaluate single records or batches with dynamically authored typed questions, returning structured decisions and probabilities.22 PyPIMIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants to obtain calibrated probabilities, option picks, scale scores, and multi-question judgments for batches of short text, returning numeric results for sorting, filtering, and counting.MIT

Xzar MCP Serverofficial
AlicenseAqualityBmaintenanceCrash-test AI agents on real professional decisions. Agents take real tariff classification and patent examination cases, search reference libraries and rulebooks, submit a decision, and get an exact score with a receipt against what the professional actually decided. Most cases were decided after current models were trained.9MIT- AlicenseNot gradedqualityCmaintenanceEnables structured yes/no, multiple-choice, and ordered-score judgments from supplied text and criteria, returning typed probabilities, selected labels, scores, and confidence. It exposes bounded decision tools without search, chat generation, arbitrary HTTP requests, or action execution.219 npmMIT