Skip to main content
Glama

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
OPENAI_API_KEYNoAPI key for OpenAI's Decisions API, used when running `openai:<model>` models.
TYPESAFE_API_KEYNoAPI key for TypeSafe, used when running the `jev` model.
OPENROUTER_API_KEYNoAPI key for OpenRouter, used when running `openrouter:<model>` models.
CLOUDFLARE_API_TOKENNoCloudflare API token, required together with CLOUDFLARE_ACCOUNT_ID when running the `clef` or `clef-flash` models (Cloudflare Workers AI).
CLOUDFLARE_ACCOUNT_IDNoCloudflare account ID, required together with CLOUDFLARE_API_TOKEN when running the `clef` or `clef-flash` models (Cloudflare Workers AI).

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

Verdict scores documents, websites, YouTube videos, JSON and text against customizable rubrics. Judges: decision models (Jev / Clef / OpenAI Decisions — typed questions, probabilities, one call per rubric), humans, deterministic benchmarks, optional generative LLM for rationales. Create or inline a rubric, then call evaluate.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": true
}

Tools

Functions exposed to the LLM to take actions

NameDescription
validate_rubricA

Validate a rubric (JSON object or YAML string) and return it with defaults filled in. Use before evaluate to catch mistakes.

create_rubricB

Store a rubric and get back its id for reuse. Criteria need id/name; optional weight, scale, anchors (= rubric levels for decision models), judge ({type: decision|human|benchmark|llm|panel}; default decision/typesafe). Aggregation supports weighted_mean, weighted_sum, mean, median, min, max, geometric_mean, formula; gates; grades; passThreshold.

list_rubricsB

List stored rubrics.

calibration_reportC

Compare decision confidence against human consensus on stored evaluations.

evaluateA

Score a subject against a rubric. rubric is a stored rubric id or an inline rubric. Inputs can be text, json, file (path or base64), url (set screenshot: true to let vision judges such as Clef see the page), image (PNG/JPEG/WebP, path or base64), or youtube. Returns the total score, grade, pass/fail, per-criterion scores with rationales, and any pending human tasks.

compareC

Evaluate several subjects against one rubric and return a ranking (leaderboard).

get_evaluationA

Fetch an evaluation by id (status, scores, rationales, pending human tasks).

submit_human_scoreB

Submit a human rater's score for a criterion that is waiting on a human. Score is a number on the criterion scale or an ordinal/binary label.

reaggregateA

Recompute the total with a different aggregation (method, formula, weights via gates/grades/passThreshold) without re-running judges. Useful for what-if analysis.

list_human_tasksD

Criteria waiting on human raters.

check_against_humansA

Run a rubric over a gold set (subjects people already judged) and report how often Verdict reached the same verdict: overall pass/fail agreement, per-criterion agreement, how many subjects the judge handed to a human (low-confidence escalation), and every disagreement with a preview. rubricPath is a local YAML/JSON rubric file, re-read on every call; gold is a local JSON array or JSON-lines file of {id?, name, inputs, expected?: {criterion: label}, human?: {passed?, criteria?}, labelSource?, split?}; choice criteria are labelled with an option id, and unreadable labels are rejected before any judge runs. Local mode only.

hub_searchA

Search published single-question judgment cards (rubrics with public test cases). Returns one result per card at its newest version, with a hub:owner/slug@rv_… ref for hub_get or fork_rubric.

hub_getB

Read a published card: its question and answers, threshold, licence, lineage, author-reported agreement numbers, versions and some public cases. ref is hub:owner/slug or hub:owner/slug@rv_….

fork_rubricA

Copy a published card into this server's rubrics (a new rubric id; the card is not changed) so evaluate can use it. The rubric's metadata records where it came from and its licence.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

B3/5.0

Scored across 14 tools

Disambiguation4/5

Most tools target clearly distinct actions: rubric CRUD/validation, evaluation, multi-subject ranking, human task handling, hub browsing, and calibration. The only real overlap is between check_against_humans (run rubric over a local gold set) and calibration_report (compare decision confidence against human consensus on stored evaluations), which could be confused at a glance despite differing inputs.

Naming Consistency3/5

There is a solid verb_noun core (get_evaluation, submit_human_score, validate_rubric, create_rubric, list_rubrics, fork_rubric), but it is mixed with bare verbs (evaluate, compare, reaggregate) and noun-first names (calibration_report, hub_search, hub_get). Readable overall, but no single predictable pattern.

Tool Count5/5

Fourteen tools is well within the ideal range and each group (rubric management, evaluation, human tasks, calibration, hub) earns its place in a coherent workflow. Nothing feels padded or redundant at the surface level.

Completeness3/5

Rubric lifecycle is only partial: create/list/validate exist but there is no get-by-id, update, or delete for stored rubrics, and the hub surface supports search/get/fork but no publish. Evaluation and human-task workflows look complete, so agents can work around the gaps but with friction.

Maintenance

ActivityMaintained
ResponsivenessOutdated data