Verdict
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| OPENAI_API_KEY | No | API key for OpenAI's Decisions API, used when running `openai:<model>` models. | |
| TYPESAFE_API_KEY | No | API key for TypeSafe, used when running the `jev` model. | |
| OPENROUTER_API_KEY | No | API key for OpenRouter, used when running `openrouter:<model>` models. | |
| CLOUDFLARE_API_TOKEN | No | Cloudflare API token, required together with CLOUDFLARE_ACCOUNT_ID when running the `clef` or `clef-flash` models (Cloudflare Workers AI). | |
| CLOUDFLARE_ACCOUNT_ID | No | Cloudflare account ID, required together with CLOUDFLARE_API_TOKEN when running the `clef` or `clef-flash` models (Cloudflare Workers AI). |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Verdict scores documents, websites, YouTube videos, JSON and text against customizable rubrics. Judges: decision models (Jev / Clef / OpenAI Decisions — typed questions, probabilities, one call per rubric), humans, deterministic benchmarks, optional generative LLM for rationales. Create or inline a rubric, then call evaluate.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| validate_rubricA | Validate a rubric (JSON object or YAML string) and return it with defaults filled in. Use before evaluate to catch mistakes. |
| create_rubricB | Store a rubric and get back its id for reuse. Criteria need id/name; optional weight, scale, anchors (= rubric levels for decision models), judge ({type: decision|human|benchmark|llm|panel}; default decision/typesafe). Aggregation supports weighted_mean, weighted_sum, mean, median, min, max, geometric_mean, formula; gates; grades; passThreshold. |
| list_rubricsB | List stored rubrics. |
| calibration_reportC | Compare decision confidence against human consensus on stored evaluations. |
| evaluateA | Score a subject against a rubric. |
| compareC | Evaluate several subjects against one rubric and return a ranking (leaderboard). |
| get_evaluationA | Fetch an evaluation by id (status, scores, rationales, pending human tasks). |
| submit_human_scoreB | Submit a human rater's score for a criterion that is waiting on a human. Score is a number on the criterion scale or an ordinal/binary label. |
| reaggregateA | Recompute the total with a different aggregation (method, formula, weights via gates/grades/passThreshold) without re-running judges. Useful for what-if analysis. |
| list_human_tasksD | Criteria waiting on human raters. |
| check_against_humansA | Run a rubric over a gold set (subjects people already judged) and report how often Verdict reached the same verdict: overall pass/fail agreement, per-criterion agreement, how many subjects the judge handed to a human (low-confidence escalation), and every disagreement with a preview. |
| hub_searchA | Search published single-question judgment cards (rubrics with public test cases). Returns one result per card at its newest version, with a |
| hub_getB | Read a published card: its question and answers, threshold, licence, lineage, author-reported agreement numbers, versions and some public cases. |
| fork_rubricA | Copy a published card into this server's rubrics (a new rubric id; the card is not changed) so |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 14 tools
Most tools target clearly distinct actions: rubric CRUD/validation, evaluation, multi-subject ranking, human task handling, hub browsing, and calibration. The only real overlap is between check_against_humans (run rubric over a local gold set) and calibration_report (compare decision confidence against human consensus on stored evaluations), which could be confused at a glance despite differing inputs.
There is a solid verb_noun core (get_evaluation, submit_human_score, validate_rubric, create_rubric, list_rubrics, fork_rubric), but it is mixed with bare verbs (evaluate, compare, reaggregate) and noun-first names (calibration_report, hub_search, hub_get). Readable overall, but no single predictable pattern.
Fourteen tools is well within the ideal range and each group (rubric management, evaluation, human tasks, calibration, hub) earns its place in a coherent workflow. Nothing feels padded or redundant at the surface level.
Rubric lifecycle is only partial: create/list/validate exist but there is no get-by-id, update, or delete for stored rubrics, and the hub surface supports search/get/fork but no publish. Evaluation and human-task workflows look complete, so agents can work around the gaps but with friction.