Skip to main content
Glama

Verdict — one rubric, every decision model

Write the rubric once. Run it on Jev, Clef, OpenAI's Decisions API, Gemini's and any other decision model, and see where they disagree on your own cases.

Decision models answer typed questions (yes or no, one of these, a level on a scale) with probabilities instead of prose, in a fraction of a second. Every vendor publishes benchmarks; none of them is about your cases, and each vendor wants its questions in its own shape. Verdict turns a rubric — criteria, scales, levels — into the questions each model takes, sends the same questions to all of them, and puts the answers side by side.

npm install -g @verdict-ai/cli
verdict run -r examples/banking77/intent.yaml -d examples/banking77/cases.jsonl --models jev,clef,clef-flash,openrouter,openai,gemini
Transfer intent × cases.jsonl: 80 case(s), 6 model(s), the same questions to each
  model        answered by                             agreed with labels   failed   asked a human   input tokens   calls   time
  jev          typesafe/jev-1.14.0                     66/80 (83%)          0        0               44,560         80      2.1 s
  clef         cloudflare/clef                         70/80 (88%)          0        0               31,259         80      6.9 s
  clef-flash   cloudflare/clef-flash                   73/80 (91%)          0        0               31,259         80      4.5 s
  openrouter   openrouter/typesafe/jev-1.13-20260917   67/80 (84%)          0        0               44,560         80      3.9 s
  openai       openai/gpt-6-luna                       69/80 (86%)          0        0               25,254         80      7.1 s
  gemini       gemini/gemini-decision-1                68/80 (85%)          0        0               25,364         80      7.1 s
the models gave one answer on 70 of 80 question(s) they all answered; they differ on 10:
  …

A real run (2026-11-12) of eighty customer messages from a public dataset, on six models: examples/banking77 has the rubric, the cases, the whole output and how to read it. It is not a ranking: eighty cases separate the models by a handful of answers. The ten cases where they differ are the ones worth reading, and on your cases they will be different ones.

Try it

Node 22 or newer. No key is needed for the first run:

npm install -g @verdict-ai/cli
git clone https://github.com/Ustay-coder/verdict && cd verdict/examples/banking77
verdict run -r intent.yaml -d cases.jsonl --models mock           # a stand-in model: shows the flow, not real answers

With a key for any one service, name its model instead (TYPESAFE_API_KEY for jev; CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN for clef; OPENROUTER_API_KEY; OPENAI_API_KEY). A model that cannot answer keeps its row and says why.

Then your own cases:

verdict rubric init rubric.yaml                                   # a starter rubric to edit
verdict run -r rubric.yaml -d cases.jsonl --models jev,clef       # cases.jsonl: one {"id", "inputs", "expected"} per line
verdict diff --base run_… --head run_…                            # any two runs, case by case

The rubric file is not edited: --models re-points its decision judges, one run per model, and every run is kept. Getting started walks through it; Decision models explains the table.

Related MCP server: Hunch MCP

Models and inputs

Name it

Model

Service

Checked against the live service

Pictures and PDFs

jev

Jev

TypeSafe

Yes

No, text only

clef

Clef

Cloudflare Workers AI

Yes

Pictures, up to 4

clef-flash

Clef-flash

Cloudflare Workers AI

Yes

Pictures, up to 4

openrouter:<model>

For example typesafe/jev-1.13

OpenRouter

Yes

No

openai:<model>

OpenAI's Decisions API, for example gpt-6-luna

OpenAI

Yes

Pictures

gemini

Gemini Decision 1

Google (Gemini API)

Yes

Pictures and PDFs

systemone:<model>

Any endpoint that takes the same request

Yours

You check it

No

Input

With verdict eval -i

What the judge receives

Text

text:…, a text file, or -

the text

File

a .pdf, .html, .md, .txt or .json path

its text; a PDF itself for a model that reads PDFs, pictures of its pages for one that takes pictures

JSON

json:{…}

the document, checkable against a schema

Web page

a URL

the page's visible text; with --screenshot, a picture too

YouTube

a YouTube URL

the transcript

Image

a .png, .jpg or .webp path

the picture, for models that take pictures

Rules (benchmark), people (human) and prompted models (llm) can judge a criterion too. Keys, context sizes and what each scale becomes: Models and inputs.

Checked against the live services: Jev, Clef, Clef-flash, OpenRouter and OpenAI's Decisions API — what each was sent, what it answered and when is in docs/PROVIDERS.md.

Once you have chosen a model

The same rubric keeps working after the comparison:

  • Version it. verdict version freeze gives a rubric a content-addressed id; verdict diff shows what a change to the rubric did to each case, and a GitHub Action comments that on pull requests. Versions, forks and CI

  • Use it from any machine. A Verdict server (in this repository, self-hosted) keeps your frozen versions and says which one each environment runs: verdict push, verdict promote, then verdict run -r <name> --env prod anywhere. The server never receives a case or an answer. Versions, forks and CI

  • Bring in people. Low-confidence answers can go to a human reviewer, and verdict check compares a rubric with what people already decided. Human review

  • Call it from code. A REST API that runs on Node or Cloudflare Workers, a TypeScript SDK, and an MCP server for agents. REST · SDK · MCP

  • Judge more than text. PDFs, web pages, YouTube transcripts, JSON and images; rules, people and prompted models as judges beside decision models; weighted scores, gates and grades. Everything Verdict does

Packages

npm install -g @verdict-ai/cli                 # the `verdict` command
npm install @verdict-ai/sdk                    # typed REST client
npm install @verdict-ai/core                   # the engine, embedded
npm install @verdict-ai/api @verdict-ai/db     # self-hosted service and persistence
npx @verdict-ai/mcp                            # stdio tools for agents

Documentation

docs.verdict-ai.dev: guides, the rubric reference, configuration, deployment and a cookbook of complete rubrics. What changed in each release: CHANGELOG.md.

Contributing

Found a bug, or a model that answers in a shape Verdict does not read? Open an issue with the command and what it printed. To build and test the code, see CONTRIBUTING.md; to report a security problem, SECURITY.md.

License

MIT. The cases in examples/banking77 are from BANKING77 by PolyAI, under CC BY 4.0.

Available Tools

14 tools
calibration_reportCalibration reportC

Compare decision confidence against human consensus on stored evaluations.

Inputs: rubricId (required), target, buckets.

check_against_humansCheck a rubric against human labelsA

Run a rubric over a gold set (subjects people already judged) and report how often Verdict reached the same verdict: overall pass/fail agreement, per-criterion agreement, how many subjects the judge handed to a human (low-confidence escalation), and every disagreement with a preview. rubricPath is a local YAML/JSON rubric file, re-read on every call; gold is a local JSON array or JSON-lines file of {id?, name, inputs, expected?: {criterion: label}, human?: {passed?, criteria?}, labelSource?, split?}; choice criteria are labelled with an option id, and unreadable labels are rejected before any judge runs. Local mode only.

Inputs: gold (required), rubric, rubricPath.

compareCompare / rankC

Evaluate several subjects against one rubric and return a ranking (leaderboard).

Inputs: rubric (required), subjects (required), idempotencyKey.

create_rubricCreate rubricB

Store a rubric and get back its id for reuse. Criteria need id/name; optional weight, scale, anchors (= rubric levels for decision models), judge ({type: decision|human|benchmark|llm|panel}; default decision/typesafe). Aggregation supports weighted_mean, weighted_sum, mean, median, min, max, geometric_mean, formula; gates; grades; passThreshold.

Inputs: rubric (required).

evaluateEvaluateA

Score a subject against a rubric. rubric is a stored rubric id or an inline rubric. Inputs can be text, json, file (path or base64), url (set screenshot: true to let vision judges such as Clef see the page), image (PNG/JPEG/WebP, path or base64), or youtube. Returns the total score, grade, pass/fail, per-criterion scores with rationales, and any pending human tasks.

Inputs: inputs (required), rubric (required), name, metadata, +1 more.

fork_rubricFork a hub cardA

Copy a published card into this server's rubrics (a new rubric id; the card is not changed) so evaluate can use it. The rubric's metadata records where it came from and its licence.

Inputs: ref (required), hub, name.

get_evaluationGet evaluationA

Fetch an evaluation by id (status, scores, rationales, pending human tasks).

Inputs: id (required).

hub_getRead a hub cardB

Read a published card: its question and answers, threshold, licence, lineage, author-reported agreement numbers, versions and some public cases. ref is hub:owner/slug or hub:owner/slug@rv_….

Inputs: ref (required), hub.

list_human_tasksList human tasksD

Criteria waiting on human raters.

Inputs: rater.

list_rubricsList rubricsB

List stored rubrics.

Inputs: no named parameters.

reaggregateRe-aggregateA

Recompute the total with a different aggregation (method, formula, weights via gates/grades/passThreshold) without re-running judges. Useful for what-if analysis.

Inputs: aggregation (required), evaluationId (required), persist.

submit_human_scoreSubmit human scoreB

Submit a human rater's score for a criterion that is waiting on a human. Score is a number on the criterion scale or an ordinal/binary label.

Inputs: score (required), raterId (required), criterionId (required), evaluationId (required), +1 more.

validate_rubricValidate rubricA

Validate a rubric (JSON object or YAML string) and return it with defaults filled in. Use before evaluate to catch mistakes.

Inputs: rubric (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3/5.0

    Scored across 14 tools

    Disambiguation4/5

    Most tools target clearly distinct actions: rubric CRUD/validation, evaluation, multi-subject ranking, human task handling, hub browsing, and calibration. The only real overlap is between check_against_humans (run rubric over a local gold set) and calibration_report (compare decision confidence against human consensus on stored evaluations), which could be confused at a glance despite differing inputs.

    Naming Consistency3/5

    There is a solid verb_noun core (get_evaluation, submit_human_score, validate_rubric, create_rubric, list_rubrics, fork_rubric), but it is mixed with bare verbs (evaluate, compare, reaggregate) and noun-first names (calibration_report, hub_search, hub_get). Readable overall, but no single predictable pattern.

    Tool Count5/5

    Fourteen tools is well within the ideal range and each group (rubric management, evaluation, human tasks, calibration, hub) earns its place in a coherent workflow. Nothing feels padded or redundant at the surface level.

    Completeness3/5

    Rubric lifecycle is only partial: create/list/validate exist but there is no get-by-id, update, or delete for stored rubrics, and the hub surface supports search/get/fork but no publish. Evaluation and human-task workflows look complete, so agents can work around the gaps but with friction.

    Maintenance

    ActivityMaintained
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to evaluate single records or batches with dynamically authored typed questions, returning structured decisions and probabilities.
      22 PyPI
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables AI assistants to obtain calibrated probabilities, option picks, scale scores, and multi-question judgments for batches of short text, returning numeric results for sorting, filtering, and counting.
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Crash-test AI agents on real professional decisions. Agents take real tariff classification and patent examination cases, search reference libraries and rulebooks, submit a decision, and get an exact score with a receipt against what the professional actually decided. Most cases were decided after current models were trained.
      9
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables structured yes/no, multiple-choice, and ordered-score judgments from supplied text and criteria, returning typed probabilities, selected labels, scores, and confidence. It exposes bounded decision tools without search, chat generation, arbitrary HTTP requests, or action execution.
      219 npm
      MIT