jev-mcp-python
This server provides MCP judgment tools that let coding agents evaluate evidence, claims, and options using TypeSafe's Jev model, turning probabilities into auto/review/escalate actions.
Verify claims against cited evidence, returning verified/contradicted/unsupported verdicts.
Screen fetched or pasted content for prompt injection and relevance before it enters agent context.
Review patches against a request for correctness, spec match, test gaps, and blast radius.
Gate completion by reviewing a patch and verifying completion claims in one call.
Compare two passages to detect same facts, contradictions, or unrelated facts, optionally per-aspect.
Find the best-matching file, note, or line for a natural-language query.
Rerank candidates by relevance, returning a full sorted list with scores.
Classify items into a shared catalog of classes, flagging low-confidence results for review.
Decide between bounded alternatives with priorities, requirements, and escape hatches like ask_user/investigate/none.
Score severity or risk on a custom 2–10 level scale.
Extract structured fields (prices, dates, versions, IDs) using regex candidates with Jev picking the true match.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@jev-mcp-pythondecide if this refund request should be auto-approved, reviewed, or escalated"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP server that gives your coding agent eleven judgment tools backed by TypeSafe's Jev model. The agent hands a tool some evidence and a question it can enumerate: is this claim supported, is this page safe to read, which of these files answers the question, did this patch finish the task. Jev answers with probabilities, usually in under a second. Policy turns them into one of three actions: auto (proceed), review (check it another way), or escalate (stop).
Use it for checks that have a fixed set of answers. When the step needs new text, code, or options you cannot list, the agent should write it itself.
Quick start
You need Python 3.12+, uv, a POSIX system (Linux or macOS), and a TypeSafe API key from console.typesafe.ai. The package is not on PyPI yet, so you run it from a clone.
git clone https://github.com/PyModel/jev-judge-mcp
cd jev-judge-mcp
uv sync --extra typesafe
uv run jev-mcp-python setup # verify your key, then store it
uv run jev-mcp-python install # add the server to your agents
uv run jev-mcp-python doctor # check the configuration, offlineRestart your agent. The tools show up as jev_verify, jev_gate, and so on (mcp__jev__* in Claude Code).
setup reads the key from TYPESAFE_API_KEY, or asks for it at a hidden prompt. It never takes the key as an argument, so the key stays out of your shell history. It makes one live call to check the key and writes nothing if TypeSafe rejects it. A good key goes to ~/.config/jev-mcp/key, readable only by you. The server uses that file whenever TYPESAFE_API_KEY is unset, so agents you start without exporting the key still work. When the variable is set, it wins.
install finds the agents on your machine, shows what it will change, and asks before writing. It supports Claude Code, Claude Desktop, Codex (CLI and the ChatGPT app), Cursor, OpenCode, Pi, omp, and Pythinker.
uv run jev-mcp-python install --dry-run # show the plan, write nothing
uv run jev-mcp-python install -a claude-code # one agent (repeatable)
uv run jev-mcp-python install --remove # undo what install wroteTerminal agents get a reference to TYPESAFE_API_KEY, never the key itself. Desktop apps don't see your shell's environment. Claude Desktop is skipped unless you pass --desktop-key, which writes the key into that app's config file. The installer warns if that file ends up readable by other users. Pi also needs its MCP adapter first: pi install npm:pi-mcp-adapter.
Related MCP server: jev-mcp
What to use it for
Ask your agent in plain words. It picks the tool, or you can name it.
You want to | Tool | You get |
Check that the agent's "done" matches the diff and the test log |
| one ship decision over the patch and each completion claim |
Check claims in a summary or PR description against the sources |
| verified, contradicted, or unsupported for each claim |
Screen a fetched web page for prompt injection before reading it |
| pass, review, block, or skip |
Review a patch against the request |
| correctness, spec match, test gaps, blast radius |
Spot drift between docs and code, or a changelog and a diff |
| same fact, contradiction, or different facts |
Find the file or note that answers a question |
| the best match, plus whether anything matches at all |
Rank search hits or grep results |
| a relevance score for every candidate, sorted |
Route tickets or label many items at once |
| one class per item from your catalog |
Pick one option, or decide whether to keep waiting on a slow command |
| your option, or |
Grade severity or risk on your own scale |
| a position on your 2 to 10 levels, with the distribution; threshold it in code, since positions between levels are weakly calibrated |
Pull a version, date, or price out of a document |
| a value copied from a match of your regex, or null |
For example, "use jev_verify to check your summary against the changelog" returns one row per claim:
{
"claim": "The setup command accepts the API key as a command-line argument.",
"verdict": "contradicted",
"probabilities": { "supports": 0, "contradicts": 1, "says_nothing": 0 },
"confidence": 1,
"action": "auto",
"supporting_evidence": "setup.py"
}Jev sees only what the agent passes in the call, so the agent has to include the evidence. docs/skills/jev-mcp/SKILL.md is a skill you can give your agent: it covers which tool fits which step and what to do with each action. Allow rules for Claude Code are printed by doctor, and opt-in setups for Claude Code, Codex, and Pi are in the harness samples.
Configuration
The server reads environment variables only. It does not load a .env file.
Variable | Default | What it does |
| unset | TypeSafe key; takes priority over the stored key |
|
| where |
|
|
|
|
| Jev model to ask |
| off | replay identical requests from disk at no API cost; leave it off when answers must be fresh, and delete the directory to clear it |
|
| where the cache lives |
|
|
|
|
| logs go to stderr |
<uvx> is the absolute path of uvx. <spec> is your clone's absolute path plus [typesafe], for example /home/me/jev-judge-mcp[typesafe].
Claude Code (~/.claude.json), omp (~/.omp/agent/mcp.json), Cursor (~/.cursor/mcp.json), and Pi (~/.pi/agent/mcp.json) use the same shape. Claude Code and omp also add "type": "stdio".
{
"mcpServers": {
"jev": {
"command": "<uvx>",
"args": ["--from", "<spec>", "jev-mcp-python"],
"env": {"TYPESAFE_API_KEY": "${TYPESAFE_API_KEY}"}
}
}
}Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json) uses the same shape with the key itself in env. Pythinker (~/.pythinker-code/mcp.json) uses it without env.
Codex CLI and the ChatGPT app share ~/.codex/config.toml:
[mcp_servers.jev]
command = "<uvx>"
args = ["--from", "<spec>", "jev-mcp-python"]
env_vars = ["TYPESAFE_API_KEY"]OpenCode (~/.config/opencode/opencode.json):
{
"mcp": {
"jev": {
"type": "local",
"command": ["<uvx>", "--from", "<spec>", "jev-mcp-python"],
"environment": {"TYPESAFE_API_KEY": "{env:TYPESAFE_API_KEY}"}
}
}
}Measured results
Two paid studies, both descriptive, with small samples and no significance test.
On 150 questions with Pi (opencode-go/deepseek-v4.1-flash), forcing a Jev call added 10.4 s median wall time per task. Letting the agent choose left Jev uncalled on all 150. Jev itself answered in 465 ms median over 157 calls. Accuracy was not measured. Details: evals/reports/bench150.md.
arm | median wall s | p95 | called Jev | agent spend |
A direct | 3.06 | 6.17 | 0/150 | $0.0928 |
B automatic | 2.91 | 8.09 | 0/150 | $0.0955 |
C forced | 13.95 | 28.67 | 150/150 | $0.2904 |
The agent outcome study ran on 2026-09-23 with jev-1.13.0: three tasks, three repeats per arm, with and without Jev. Both agents solved the same pairs either way and picked the right decision on every run. Both were slower with Jev. One Pi pair is excluded because its with-Jev run never called Jev. Details, raw records, and the chart script: docs/evals/.
agent | solved without / with Jev | median time to correct, without / with | extra wall time with Jev (paired median) | spend |
Claude Code ( | 6/9 / 6/9 | 14.6 s / 18.3 s | +4.6 s | $1.4953 |
Pi ( | 6/8 / 6/8 | 49.4 s / 127.8 s | +85.8 s | $0.0006 |
About this project
This is a Python rewrite of the TypeScript @jkudish/jev-mcp 0.5.0. The ten reference tools match it on the wire, checked by recorded parity fixtures; jev_score is an addition. Vocabulary is in docs/CONTEXT.md, decisions in docs/adr/, and security notes in SECURITY.md. Windows is not supported; the server exits at startup on a non-POSIX platform.
uv sync --all-extras
make ci # lint, types, unit, property, contract, parity, security, build, smoke
make eval # offline scorer checksmake eval-live, make security-live, and JEV_AB_LIVE=1 make ab call paid services and stay off CI. Contribution notes are in CONTRIBUTING.md.
MIT license.
Available Tools
10 toolsjev_classifyClassify items against a shared label setA
Assign each item to one class from a shared catalog with TypeSafe Jev, in one batched request: the class catalog is sent once and every item becomes an independent Choice question. Returns per item: the chosen class, the full distribution, confidence, winner-to-runner-up margin, and an auto-versus-review decision. Auto requires both a high top probability (default 0.85) and a clear margin (default 0.50); everything else is flagged for review. Include a manual_review class in the catalog if you want an explicit escape hatch; the tool never invents one.
| Name | Required | Description | Default |
|---|---|---|---|
| items | Yes | Items to classify. Text is truncated at 2000 characters; send bounded excerpts, not whole documents. | |
| classes | Yes | Shared class catalog. Strong descriptions carry the decision: a precise definition, what belongs, what does not, precedence over overlapping classes, and a short example. | |
| context | No | Shared context available to every item's judgment: policies, catalogs, anything stable. | |
| purpose | No | What this classification is for; shared across all items. | |
| auto_accept | No | Minimum top probability for auto. Default 0.85. | |
| minimum_margin | No | Minimum winner-to-runner-up gap for auto. Default 0.5. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It transparently discloses the return fields (chosen class, distribution, confidence, margin, auto-versus-review decision), explains the auto thresholds (0.85 probability and 0.50 margin) and that everything else is flagged for review, and clarifies that a manual_review class must be explicitly included—the tool never invents one. This gives agents a clear picture of behavior without annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single paragraph with logically ordered information: purpose, output, auto criteria, and a key behavioral note. It is information-dense without being verbose, and front-loads the core purpose before diving into details. Slightly long but appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 6 parameters, no output schema, and no annotations, so the description must explain both usage and returns. It covers the return fields, auto decision logic, and the manual_review caveat. It does not address error handling, rate limits, or permission requirements, but for a classification tool these are minor. The description is sufficient for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are already fully documented. The description adds context about the batching and the independent Choice question mechanism, but does not add new semantic meaning beyond the schema's per-parameter descriptions. It reinforces that class descriptions are critical, but that is already stated in the schema. Thus, it meets the baseline but does not exceed it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool assigns each item to one class from a shared catalog in a single batched request. It uses a specific verb ('assign... to one class'), names the resource (items and class catalog), and distinguishes itself from siblings like decide or rerank by focusing on classification with a shared label set. The process and output are described in concrete terms.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides detailed process information but does not explicitly guide when to use this tool versus alternatives. There is no mention of sibling tools or conditions that would favor a different tool (e.g., single decision, ranking, extraction). It only implicitly suggests use for batched classification, but lacks exclusions or alternative routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_compareCompare two passages for factual agreementA
Judge the relation between two passages with TypeSafe Jev: same_fact, contradicts, or different_facts, with the full probability distribution, confidence, and an auto-versus-review decision. Optionally supply aspects (price, date, method, …) and each gets an independent per-aspect judgment in the same single request. Use for source reconciliation, changelog-vs-code drift, or merge sanity checks. The request supplies no evidence beyond the two passages, so a same_fact verdict means they agree with each other, not that they are true.
| Name | Required | Description | Default |
|---|---|---|---|
| aspects | No | Named aspects to judge independently (e.g. 'price', 'launch date'). Each tests one property. | |
| purpose | No | What this comparison is for; helps disambiguate overlap. | |
| passage_a | Yes | First passage. Rejected above 20,000 characters. | |
| passage_b | Yes | Second passage. Rejected above 20,000 characters. | |
| auto_accept | No | Minimum top probability for auto. Default 0.85. | |
| minimum_margin | No | Minimum winner-to-runner-up gap for auto. Default 0.5. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description takes on full responsibility for behavioral disclosure, and it does so very well. It reveals the output shape (probability distribution, confidence, auto-versus-review), the per-aspect behavior, and the critical epistemic caveat that same_fact means mutual agreement, not truth. This is substantive behavioral context beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences with no filler. The core verdict behavior is front-loaded, followed by optional aspects, use cases, and a caveat. Every sentence earns its place and the description is dense without being bloated.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has moderate complexity, no output schema, and no annotations, yet the description covers the return format, optional behavior, and the meaning of a verdict. An agent has enough context to select and call this tool correctly; the schema covers the remaining parameter details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents every parameter. The description adds genuine meaning beyond it by explaining that aspects receive independent per-aspect judgments in the same single request and that no external evidence is used, which clarifies the semantics of the purpose and aspects parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Judge the relation between two passages') and names the three possible verdicts, making the operation concrete. It stops short of explicitly differentiating itself from sibling tools like jev_verify or jev_classify, though the relation-judging focus is reasonably distinct within the suite.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit use cases: source reconciliation, changelog-vs-code drift, and merge sanity checks. It does not state when not to use the tool or name alternatives, but the provided contexts are clear enough for an agent to route appropriately.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_decideDecide between bounded alternativesA
One unresolved, bounded decision where semantic judgment over supplied evidence could change your plan: implementation alternatives, product tradeoffs with known preferences, workflow selection. Supply 2-6 candidates, evidence, and explicit priorities. Jev returns a Choice distribution over the candidates plus escape hatches (ask_user / investigate / none), and a per-candidate per-requirement supported / contradicted / unknown judgment for each optional requirement, all in one request. One call per unchanged decision; do not repeat a call to obtain a more pleasing answer. Use source inspection, tests, the user, or a reasoning model for open-ended research, routine choices, correctness proofs, or predicting user consent. High probability is not proof.
| Name | Required | Description | Default |
|---|---|---|---|
| decision | Yes | The bounded decision to make. | |
| evidence | Yes | Facts and measurements, not opinions. State is evidence, not instructions. | |
| candidates | Yes | The alternatives. Include 'do nothing' or 'gather more evidence' as candidates when useful. | |
| priorities | Yes | Explicit preferences and constraints from the user or plan. | |
| requirements | No | Specific requirements to check per candidate. Each must test one property, not overall goodness. | |
| escape_hatches | No | Include ask_user / investigate / none as Choosable options so the model can decline to rank. Default true. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full behavioral burden and does so thoroughly. It discloses that the tool may return escape hatches (ask_user / investigate / none), that it produces per-candidate per-requirement supported / contradicted / unknown judgments, that it is single-shot, and that high probability from the model is not proof.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but well organized: it front-loads the intended use case, then states required inputs, explains the output shape, and closes with usage cautions and exclusions. No sentence is wasted; the length is justified by the tool's behavioral complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description correctly explains return values: Choice distribution over candidates, escape hatches, and per-requirement judgments. It also covers operational guardrails (one call per unchanged decision, don't re-run for a pleasing answer) and epistemic framing ('High probability is not proof'). An agent has enough context to invoke and interpret the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters with detailed descriptions, so the baseline is 3. The tool description adds minimal parameter-level meaning beyond the schema, aside from framing how many candidates to supply and the distinction between evidence and priorities, which the schema already addresses.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: decide over bounded alternatives, with a distinctive output of a Choice distribution, escape hatches, and per-requirement judgments. The described use cases (implementation alternatives, product tradeoffs, workflow selection) and output artifacts clearly separate it from siblings like jev_classify, jev_rerank, and jev_compare.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use it ('unresolved, bounded decision... semantic judgment over supplied evidence') and when not to ('open-ended research, routine choices, correctness proofs, or predicting user consent'). It also gives alternatives—source inspection, tests, the user, or a reasoning model—and adds the critical 'do not repeat a call to obtain a more pleasing answer' rule.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_extractExtract fields by regex, Jev picks the right matchA
Extract structured fields from a document with TypeSafe Jev as the picker, not the generator: your regex finds candidate substrings in code, Jev chooses which candidate is the field's true value, and the result is returned verbatim — never model-generated text. Fields with zero regex matches never reach the model (not_found); if no field has matches, no API call is made. Ambiguous picks are flagged for review. Use for prices, dates, version numbers, IDs, and anything with a recognizable shape; keep documents bounded.
| Name | Required | Description | Default |
|---|---|---|---|
| fields | Yes | Fields to extract. Up to 32 per call, all judged in one request. | |
| purpose | No | What the extraction is for; shared across fields. | |
| document | Yes | The document to extract from. Rejected above 50,000 characters. | |
| auto_accept | No | Minimum top probability for auto. Default 0.85. | |
| minimum_margin | No | Minimum winner-to-runner-up gap for auto. Default 0.5. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It thoroughly discloses that results are returned verbatim, that fields with zero matches become 'not_found', that no API call is made if no field has matches, and that ambiguous picks are flagged for review. This is unusually rich behavioral context for a tool without annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: the first explains the core behavior, the second covers failure modes, the third covers ambiguity handling, and the last gives practical usage guidance. The description is front-loaded and contains no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description plus full schema coverage gives an agent enough to call the tool correctly. It explains the input semantics, failure behavior, and output philosophy. It does not specify the exact response JSON structure, but since there is no output schema, slightly more precision about the returned shape would be helpful; overall this is still strong.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents each parameter. The description goes beyond that by explaining how the parameters interact: the regex finds candidate substrings, Jev picks the true value using the field description, and zero-match fields never reach the model. It also explains that ambiguity causes review, which adds meaning to auto_accept and minimum_margin.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Extract'), a specific resource ('structured fields from a document'), and the core mechanism ('TypeSafe Jev as the picker, not the generator'). It also differentiates from sibling tools by emphasizing verbatim, non-model-generated results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear use cases: 'prices, dates, version numbers, IDs, and anything with a recognizable shape.' It also advises to 'keep documents bounded.' It does not explicitly name which sibling to use instead for other operations, but the extraction-versus-classification/decision framing makes the intended context fairly clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_findSemantic search over candidatesA
Rank candidates against a plain-language query with TypeSafe Jev — no embeddings needed. One Choice scores every candidate id by how well it answers the query, plus a Noul checks whether any candidate addresses the query at all (so a confident 'top hit' cannot masquerade as an answer). Pattern: docs.typesafe.ai/cookbooks/semantic_find. Use for 'which file/note/line covers X' across up to 250 candidates.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | What you are looking for, in natural language. | |
| top_k | No | How many ranked candidates to return. Default 5. | |
| candidates | Yes | Candidates to search. Up to 250 in one call; texts are truncated at 2000 chars. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden. It adds meaningful behavioral detail: 'no embeddings needed', 'One Choice scores every candidate', and a 'Noul' check preventing an unsupported 'top hit'. It does not cover output format or error behavior, but for a read-style search tool the core operation is well disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is two information-dense sentences plus a link and a use case. The opening sentence front-loads the main function. Some jargon ('One Choice', 'Noul', 'TypeSafe Jev') could be clearer, but nothing is extraneous.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is moderately complex with a nested candidates array and no output schema. The description gives use context but does not explain return shape, how the 'Noul' result appears, or when to prefer jev_screen/jev_verify over jev_find. These gaps matter given the absence of an output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description mentions plain-language query and candidate count, but these largely repeat schema content. No additional parameter-level insight (e.g. top_k behavior, id conventions) is added beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ('Rank candidates against a plain-language query') and adds a concrete use case ('which file/note/line covers X'). It is clearly a search/ranking tool, but it does not explicitly contrast itself with siblings jev_screen or jev_verify, so it misses the top score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides a clear trigger phrase ('Use for which file/note/line covers X') and a capacity limit (up to 250 candidates). It does not explicitly say when not to use it or reference alternatives among the named siblings, so it falls short of full guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_gateGate completion: review a patch and verify claimsA
Review a proposed patch and verify completion claims against supplied evidence in one TypeSafe Jev call. Auto only when the patch review is accepted and every claim is verified at or above auto_accept. Unsupported claims require review; confident contradictions, unknown confidence, or low confidence escalate. The request and claims are assertions to check, never proof; put supporting diff excerpts and test logs in evidence. Evidence is capped at 16 items and 200,000 characters in aggregate. Does not run tests or apply changes. Use jev_review for a patch without claims, jev_verify for claims without a patch review.
| Name | Required | Description | Default |
|---|---|---|---|
| diff | Yes | Proposed patch, file excerpt, or change summary. Truncated at 50000 chars. | |
| tests | No | Reported test output for the patch review. Truncated at the same cap. | |
| claims | Yes | Completion claims to check against evidence, each truncated at 2000 chars. Up to 16 per call. | |
| request | Yes | What the user asked for; this is not evidence of completion. | |
| evidence | Yes | ||
| review_at | No | Score, safe_to_apply, or per-claim confidence below this escalates. Must be <= auto_accept. Default min(0.5, auto_accept). | |
| auto_accept | No | Review and per-claim confidence at or above this may stand automatically. Default 0.8. | |
| composite_floor | No | Weighted composite at or above this is required for auto. Default 0.7. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even with no annotations, the description discloses key behaviors: 'Does not run tests or apply changes,' evidence caps, claim thresholds, and that request/claims are not proof. This is strong behavioral transparency for a complex tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact yet information-dense. Every sentence adds distinct value: purpose, auto-accept policy, escalation behavior, evidence guidance, safety disclaimer, and sibling routing. No filler or redundant repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 8 parameters and no output schema, the description covers purpose, safety, thresholds, evidence limits, and alternatives well. A slight gap is the lack of explicit statement about return values or output shape, but the behavioral descriptions largely compensate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is high (88%), so the baseline is 3. The description adds extra meaning by advising that evidence should contain diff excerpts and test logs payll and clarifying that request/claims are assertions, not proof. This goes beyond the schema's generic parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('review a proposed patch and verify completion claims against supplied evidence') and clearly distinguishes this from sibling tools. It names jev_review and jev_verify as alternatives, making the tool's scope unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use this tool versus alternatives: 'Use jev_review for a patch without claims, jev_verify for claims without a patch review.' It also describes auto-accept and escalation conditions, giving clear decision context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_rerankScore every candidate's relevance and return them sortedA
Rerank candidates against a query with TypeSafe Jev: one independent relevance probability per candidate, all in a single request, then sorted by score. Unlike jev_find (which picks one best answer), rerank scores every candidate so the full ordering survives. TypeSafe's rerank cookbook reports that on the CLERC benchmark this pattern lifted top-1 from 5% to 18% and top-10 from 38% to 62% (docs.typesafe.ai/cookbooks). Use for retrieval ordering, dedup triage, or feed ranking across up to 250 candidates.
| Name | Required | Description | Default |
|---|---|---|---|
| query | Yes | What relevance is measured against, in natural language. | |
| top_k | No | How many ranked candidates to return. Default: all. | |
| candidates | Yes | Candidates to search. Up to 250 in one call; texts are truncated at 2000 chars. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that each candidate receives one independent relevance probability, that all are processed in a single request, and that results are sorted by score. It also mentions the 250-candidate limit. While it doesn't explicitly state the return format or confirm non-mutation, the description gives sufficient behavioral insight for an agent to understand what happens when the tool is invoked.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded: it states the core action first, then differentiates from a sibling, provides a performance benchmark, and lists use cases. Each sentence adds value, though the benchmark detail, while useful, makes it slightly longer than strictly necessary. Still, it remains focused and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no output schema and no annotations, the description is fairly complete. It covers the operation, the use cases, and the limits. The only notable omission is an explicit statement of the return structure (e.g., an array of candidates with scores), but that is largely implied by the description and the tool's purpose. Given the richness of the description, it is sufficient for an agent to call the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 100% coverage, so the description does not need to explain parameters. It does add a minor note about the 250-candidate cap, which matches the schema's maxItems, but it does not enrich the semantic understanding beyond what the schema already provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: rerank candidates against a query, scoring each independently and sorting by score. It explicitly differentiates from jev_find by noting that find picks one best answer while rerank scores every candidate, so the full ordering survives. This is a specific verb + resource with clear scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use guidance, naming jev_find as an alternative and explaining the distinction. It also lists concrete use cases: retrieval ordering, dedup triage, and feed ranking. It even mentions a benchmark to suggest when this pattern is beneficial, giving the agent a clear decision heuristic.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_reviewReview a proposed patchA
Score a proposed diff against the request with TypeSafe Jev before the task is called done. Returns 0..2 rubric scores for correctness, spec match, test gap, and blast radius (the last two lower the weighted composite), a safe_to_apply probability, and an auto | review | escalate action. Auto requires safe_to_apply and min score confidence at auto_accept and the composite at composite_floor; truncated or malformed input never returns auto. Does not apply the patch or run tests. Use jev_gate to also verify completion claims against evidence in the same call.
| Name | Required | Description | Default |
|---|---|---|---|
| diff | Yes | Proposed patch, file excerpt, or change summary. Truncated at 50000 chars. | |
| tests | No | Reported test output, if any. Truncated at the same cap. | |
| request | Yes | What the user asked for; this frames the review, it is not proof of anything. | |
| review_at | No | Min score confidence or safe_to_apply below this escalates. Must be <= auto_accept. Default min(0.5, auto_accept). | |
| auto_accept | No | safe_to_apply and min score confidence at or above this may stand automatically. Default 0.8. | |
| composite_floor | No | Weighted composite at or above this is required for auto. Default 0.7. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states the tool is non-mutating ('Does not apply the patch or run tests'), explains the auto/escalate decision logic, and discloses truncation behavior ('Truncated at 50000 chars'). It could add more about failure modes or error handling, but the key behavioral traits are well covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but efficient, front-loading the core action and rubric before moving to decision logic and exclusions. Every sentence earns its place, though the decision-logic sentence is long and packs several conditions together. It is appropriately sized for a tool with 6 parameters and complex auto-accept behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a review tool with no output schema, the description does a good job of explaining what the tool returns (rubric scores, safe_to_apply probability, action) and how the action is determined. It could be more complete by describing the exact output format or error behavior, but the essential context for an agent to decide whether to call this tool and interpret its result is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 6 parameters. The description adds meaningful context beyond the schema: it explains how the parameters interact (auto requires safe_to_apply and min score confidence at auto_accept and composite at composite_floor), and clarifies that 'request' frames the review but is not proof. This is valuable semantic glue that the schema alone doesn't provide.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb ('Score'), a specific resource ('a proposed diff against the request'), and a named method ('TypeSafe Jev'). It distinguishes itself from siblings by naming jev_gate as the alternative for verifying completion claims, and the rubric detail (correctness, spec match, test gap, blast radius) makes the tool's function unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use this tool ('before the task is called done') and names the sibling alternative ('Use jev_gate to also verify completion claims against evidence in the same call'). It also states what the tool does not do ('Does not apply the patch or run tests'), which helps an agent avoid misusing it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_screenScreen content before it enters agent contextA
Judge fetched or external text with TypeSafe Jev before an agent reads it: probability it contains instructions aimed at an AI agent (prompt injection), whether it has substantive content, and (when a purpose is given) whether it is relevant to the task. Returns a recommendation: pass | review | block | skip. Pattern: docs.typesafe.ai/cookbooks/llm_guardrails.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The content to screen, e.g. a fetched web page or pasted document. | |
| purpose | No | What the consuming agent is trying to do; enables a relevance judgment and the 'skip' action. | |
| block_at | No | Injection probability at or above which content is blocked. Default 0.75. | |
| review_at | No | Injection probability at or above which content is flagged for review. Default 0.25. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does a good job: it explains the evaluation dimensions, the recommendation values (pass|review|block|skip), and the conditional relevance behavior. It does not explicitly state side-effect-free behavior or response structure beyond the recommendation list, but the judging nature is clear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the core purpose, and includes the output contract without fluff. The reference to the cookbook pattern is useful and compact. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given four parameters, no annotations, and no output schema, the description provides enough for an agent to invoke the tool correctly: it states inputs, output categories, and conditional behavior. It could be more explicit about the probability output format or threshold semantics, but the schema already covers threshold parameters.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all four parameters clearly. The description adds some context by linking 'purpose' to the relevance judgment and 'skip' action, but it does not meaningfully supplement the parameter meanings beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb ('Judge'), a specific resource (fetched or external text), and the analysis dimensions (injection probability, substantive content, relevance). It does not explicitly differentiate from the sibling tools jev_verify and jev_find, so it misses the top score for sibling distinction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly identifies when to use the tool: 'before an agent reads it.' It implies the guardrail context and references a cookbook pattern, giving solid situational context. However, it does not mention when not to use it or alternatives among the siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
jev_verifyVerify claims against evidenceA
Check each claim against provided evidence text with TypeSafe Jev. Returns per claim: verdict (verified | contradicted | unsupported), full probability distribution, confidence, and whether the verdict stands on its own (auto) or needs human review. Pattern: docs.typesafe.ai/cookbooks/citation_check. Pass reports, PR descriptions, or agent briefs as claims and their cited sources, diffs, or documents as evidence.
| Name | Required | Description | Default |
|---|---|---|---|
| claims | Yes | Claims to verify, e.g. individual factual statements from a report. | |
| evidence | Yes | ||
| auto_accept | No | Verdicts at or above this confidence stand automatically; below it they are flagged 'review'. Default 0.8. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly explains what the tool returns per claim—verdict, probability distribution, confidence, and auto/review status—and implies a confidence-threshold behavior through the output. It does not mention side effects or rate limits, but the verification behavior is a read-only-style computation and the output behavior is detailed enough for an agent to anticipate the result.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences and every one earns its place: core action, return format, and practical usage mapping. The most important information is front-loaded, and the writing is compact without sacrificing the behavioral detail an agent needs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with two required parameters, one optional threshold, and no output schema, the description covers both what the agent should pass and what it should expect back. The evidence parameter's ability to accept multiple items and map claims to evidence is handled partly by the schema and partly by the description, leaving only minor gaps around exact output formatting.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents all three parameters, so the description does not need to repeat their mechanics. The description adds useful mapping examples ('reports, PR descriptions, or agent briefs' as claims; 'cited sources, diffs, or documents' as evidence), but it does not add meaning to auto_accept beyond the schema, giving it only modest added value at this coverage level.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action ('Check each claim against provided evidence text') and a distinct resource ('TypeSafe Jev'), making the tool's core function unmistakable. It does not explicitly distinguish this from siblings jev_screen and jev_find, but the verification purpose and return categories are specific enough for an agent to separate it from those names.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives concrete usage context: pass reports, PR descriptions, or agent briefs as claims, and cited sources, diffs, or documents as evidence. It does not explicitly state when not to use this tool or name alternatives, but the input examples provide clear practical guidance for selecting appropriate content.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
10 tool updates
v0.1.0- First observed
jev_classify - First observed
jev_compare - First observed
jev_decide - First observed
jev_extract - First observed
jev_find - First observed
jev_gate - First observed
jev_rerank - First observed
jev_review - First observed
jev_screen - First observed
jev_verify
TDQS
Scored across 10 tools
Each tool has a distinct verb and purpose: verify claims, screen for injection, find best match, classify, decide, rerank, compare, extract, review, and gate. Even similar tools like jev_find and jev_rerank are clearly differentiated (one picks a single best, the other orders all), and jev_review vs jev_gate are explicitly separated by the presence of claim verification.
All tools follow a consistent 'jev_' prefix with a lowercase descriptive verb (verify, screen, find, classify, decide, rerank, compare, extract, review, gate). The naming pattern is uniform and predictable, making it easy for an agent to infer function from the name.
10 tools is well within the ideal 3-15 range. Each tool covers a distinct capability within the Jev evaluation domain, and none feel redundant or unnecessary. The count is appropriately scoped for the server's purpose.
The tool set provides comprehensive coverage of evaluation tasks: claim verification, prompt-injection screening, semantic search, classification, decision-making, reranking, pairwise comparison, structured extraction, patch review, and gated review with claim verification. Cross-references between tools (e.g., using jev_gate for patches with claims, jev_review without) indicate a well-thought-out and complete surface with no obvious gaps for the stated domain.
Maintenance
Related MCP Connectors
Sentiment, toxicity, entity extraction, PII, translation, summary, QA, fraud scoring, safety audit.
The system of record for AI agent authority: playbooks, routed policy questions, reusable rules.
- DatagoatOAuthio.datagoat
Governed decision engine: yes/no, score, choice and rank answers about cases, from past outcomes.
1 Tribeunal turns a question into a jury's verdict. An agent opens a case, a jury of humans and AI agents is seated, evidence is weighed and votes are cast, and the tally becomes a ruling the agent can long-poll for and act on. 39 tools, all annotated, plus eight Agent Skills that carry the procedure: how big a jury a decision needs, when a verdict actually lands, how to act on it. Arbitration mode bars the case owner from voting or closing early and enforces a quorum, closing with a verdict.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables agents to perform typed judgments—classify, score, check, match, and screen—over closed answer sets with confidence scores, without text generation.74MIT
- AlicenseAqualityBmaintenanceEnables typed, calibrated judgment calls through classify, score, check, and batched ask tools, each returning full probability distributions for programmatic decisions.5981 npm7MIT
- AlicenseNot gradedqualityCmaintenanceEnables agents to evaluate single records or batches with dynamically authored typed questions, returning structured decisions and probabilities.MIT
- AlicenseAqualityCmaintenanceProvides agents with fast, typed, calibrated decision tools for classification, scoring, yes/no checks, and gating risky tool calls.5981 npmMIT