Skip to main content
Glama
PyModel
by PyModel

jev-mcp-python

jev-mcp-python

Python 3.12+, MIT, MCP, v0.1.0, stdio ci

jev-mcp-python is a Python MCP server that exposes TypeSafe's Jev model as ten judgment tools. Its wire behavior is frozen against a TypeScript reference server, 0.5.0, through recorded parity fixtures. Jev answers a typed question with a probability distribution. Policy in this server turns a validated answer into auto, review, or escalate.

Wire behavior is pinned to the TypeScript reference through recorded parity fixtures. Vocabulary is in CONTEXT.md. Decisions are in docs/adr/.

Install

Python 3.12 or newer, and uv.

uvx --from git+https://github.com/PyModel/jev-mcp jev-mcp-python

From a checkout:

uv sync --all-extras
uv run jev-mcp-python

The default transport is stdio. Logs go to stderr. Settings come from the environment. The server does not read a .env file.

export TYPESAFE_API_KEY=...
export JEV_PROVIDER=typesafe
export JEV_MCP_MODEL=jev-1.13.0

JEV_PROVIDER=auto uses the first configured credential, in this order: typesafe, openrouter, cloudflare, compatible. The Vercel provider is not supported. HTTP is experimental (JEV_MCP_TRANSPORT=streamable-http, bound to 127.0.0.1:8000 unless you change the host and port).

Related MCP server: jev-mcp

Tools

Published in the reference order:

  • jev_verify checks each claim against the evidence you supply.

  • jev_screen judges fetched text before an agent reads it.

  • jev_find ranks candidates against a plain-language query.

  • jev_classify assigns each item to one class from a shared catalog.

  • jev_decide chooses among bounded alternatives, and can decline.

  • jev_rerank scores every candidate's relevance and returns them sorted.

  • jev_compare judges whether two passages state the same fact, contradict, or state different facts.

  • jev_extract lets your regex find the matches, then asks Jev to pick among them.

  • jev_review scores a proposed patch against the request.

  • jev_gate reviews a patch and verifies completion claims in one call.

Bench

Numbers below are copied from evals/reports/bench150.md. Accuracy is not measured: 0 items were labeled. Speed and time use the 43 pairs where both arms reached the model. 107 with-Jev runs never reached the local model. The report counts those as connection errors, and does not count them as Jev misses. Of the 43 runs that reached the model, 42 got a Jev answer (98%).

measure

before (no Jev)

after (with Jev)

accuracy

not measured (0 labeled items)

not measured (0 labeled items)

speed (items/min)

3.38

0.79

time consumed (s)

762.71

3266.66

Jev connection

42/43 (98%)

pairs where both arms ran

43

43

pairs recorded

150

150

Stop: all 150 pairs recorded.

Speed, items per minute

Time consumed, total wall time

The page with the same charts is evals/reports/bench150.html. Raw transcripts are not in this repo.

Agent pilot

The P8 pilot ran on 2026-09-21 with Claude Code CLI 2.1.278, model claude-sonnet-5, effort medium. One run per task and arm. Tasks were t1-bulk-boundary, t2-slugify, and t3-durations. No B or C run called a Jev tool. The report says the table cannot tell the arms apart, and that the cost gap is run noise. Full table and per-run rows: evals/reports/p8-pilot.md.

measure

A: no Jev

B: this server

C: TypeScript server 0.5.0

runs (failed)

3 (0)

3 (0)

3 (0)

correct tasks

3/3

3/3

3/3

cost per correct task

$0.0336

$0.0362

$0.0326

total cost

$0.1008

$0.1085

$0.0977

Jev tool calls

0

0

0

wall time, s (sum)

30.9

33.7

30.6

All 9 runs finished. Total spent: $0.3070 of the $25.00 cap.

Develop

uv sync --all-extras
make ci

make ci runs lint, typecheck, unit, property, policy coverage, contract, parity, security, build, and the smoke test. make eval is the offline scorer check. make eval-live, make security-live, and JEV_AB_LIVE=1 make ab call paid services and stay off CI.

Contribution notes are in CONTRIBUTING.md.

License: MIT.

Available Tools

10 tools
jev_classifyClassify items against a shared label setA

Assign each item to one class from a shared catalog with TypeSafe Jev, in one batched request: the class catalog is sent once and every item becomes an independent Choice question. Returns per item: the chosen class, the full distribution, confidence, winner-to-runner-up margin, and an auto-versus-review decision. Auto requires both a high top probability (default 0.85) and a clear margin (default 0.50); everything else is flagged for review. Include a manual_review class in the catalog if you want an explicit escape hatch; the tool never invents one.

ParametersJSON Schema
NameRequiredDescriptionDefault
itemsYesItems to classify. Text is truncated at 2000 characters; send bounded excerpts, not whole documents.
classesYesShared class catalog. Strong descriptions carry the decision: a precise definition, what belongs, what does not, precedence over overlapping classes, and a short example.
contextNoShared context available to every item's judgment: policies, catalogs, anything stable.
purposeNoWhat this classification is for; shared across all items.
auto_acceptNoMinimum top probability for auto. Default 0.85.
minimum_marginNoMinimum winner-to-runner-up gap for auto. Default 0.5.

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It transparently discloses the return fields (chosen class, distribution, confidence, margin, auto-versus-review decision), explains the auto thresholds (0.85 probability and 0.50 margin) and that everything else is flagged for review, and clarifies that a manual_review class must be explicitly included—the tool never invents one. This gives agents a clear picture of behavior without annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single paragraph with logically ordered information: purpose, output, auto criteria, and a key behavioral note. It is information-dense without being verbose, and front-loads the core purpose before diving into details. Slightly long but appropriate for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 6 parameters, no output schema, and no annotations, so the description must explain both usage and returns. It covers the return fields, auto decision logic, and the manual_review caveat. It does not address error handling, rate limits, or permission requirements, but for a classification tool these are minor. The description is sufficient for an agent to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are already fully documented. The description adds context about the batching and the independent Choice question mechanism, but does not add new semantic meaning beyond the schema's per-parameter descriptions. It reinforces that class descriptions are critical, but that is already stated in the schema. Thus, it meets the baseline but does not exceed it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool assigns each item to one class from a shared catalog in a single batched request. It uses a specific verb ('assign... to one class'), names the resource (items and class catalog), and distinguishes itself from siblings like decide or rerank by focusing on classification with a shared label set. The process and output are described in concrete terms.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides detailed process information but does not explicitly guide when to use this tool versus alternatives. There is no mention of sibling tools or conditions that would favor a different tool (e.g., single decision, ranking, extraction). It only implicitly suggests use for batched classification, but lacks exclusions or alternative routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_compareCompare two passages for factual agreementA

Judge the relation between two passages with TypeSafe Jev: same_fact, contradicts, or different_facts, with the full probability distribution, confidence, and an auto-versus-review decision. Optionally supply aspects (price, date, method, …) and each gets an independent per-aspect judgment in the same single request. Use for source reconciliation, changelog-vs-code drift, or merge sanity checks. The request supplies no evidence beyond the two passages, so a same_fact verdict means they agree with each other, not that they are true.

ParametersJSON Schema
NameRequiredDescriptionDefault
aspectsNoNamed aspects to judge independently (e.g. 'price', 'launch date'). Each tests one property.
purposeNoWhat this comparison is for; helps disambiguate overlap.
passage_aYesFirst passage. Rejected above 20,000 characters.
passage_bYesSecond passage. Rejected above 20,000 characters.
auto_acceptNoMinimum top probability for auto. Default 0.85.
minimum_marginNoMinimum winner-to-runner-up gap for auto. Default 0.5.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description takes on full responsibility for behavioral disclosure, and it does so very well. It reveals the output shape (probability distribution, confidence, auto-versus-review), the per-aspect behavior, and the critical epistemic caveat that same_fact means mutual agreement, not truth. This is substantive behavioral context beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences with no filler. The core verdict behavior is front-loaded, followed by optional aspects, use cases, and a caveat. Every sentence earns its place and the description is dense without being bloated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has moderate complexity, no output schema, and no annotations, yet the description covers the return format, optional behavior, and the meaning of a verdict. An agent has enough context to select and call this tool correctly; the schema covers the remaining parameter details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents every parameter. The description adds genuine meaning beyond it by explaining that aspects receive independent per-aspect judgments in the same single request and that no external evidence is used, which clarifies the semantics of the purpose and aspects parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource ('Judge the relation between two passages') and names the three possible verdicts, making the operation concrete. It stops short of explicitly differentiating itself from sibling tools like jev_verify or jev_classify, though the relation-judging focus is reasonably distinct within the suite.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit use cases: source reconciliation, changelog-vs-code drift, and merge sanity checks. It does not state when not to use the tool or name alternatives, but the provided contexts are clear enough for an agent to route appropriately.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_decideDecide between bounded alternativesA

One unresolved, bounded decision where semantic judgment over supplied evidence could change your plan: implementation alternatives, product tradeoffs with known preferences, workflow selection. Supply 2-6 candidates, evidence, and explicit priorities. Jev returns a Choice distribution over the candidates plus escape hatches (ask_user / investigate / none), and a per-candidate per-requirement supported / contradicted / unknown judgment for each optional requirement, all in one request. One call per unchanged decision; do not repeat a call to obtain a more pleasing answer. Use source inspection, tests, the user, or a reasoning model for open-ended research, routine choices, correctness proofs, or predicting user consent. High probability is not proof.

ParametersJSON Schema
NameRequiredDescriptionDefault
decisionYesThe bounded decision to make.
evidenceYesFacts and measurements, not opinions. State is evidence, not instructions.
candidatesYesThe alternatives. Include 'do nothing' or 'gather more evidence' as candidates when useful.
prioritiesYesExplicit preferences and constraints from the user or plan.
requirementsNoSpecific requirements to check per candidate. Each must test one property, not overall goodness.
escape_hatchesNoInclude ask_user / investigate / none as Choosable options so the model can decline to rank. Default true.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations present, the description carries the full behavioral burden and does so thoroughly. It discloses that the tool may return escape hatches (ask_user / investigate / none), that it produces per-candidate per-requirement supported / contradicted / unknown judgments, that it is single-shot, and that high probability from the model is not proof.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but well organized: it front-loads the intended use case, then states required inputs, explains the output shape, and closes with usage cautions and exclusions. No sentence is wasted; the length is justified by the tool's behavioral complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description correctly explains return values: Choice distribution over candidates, escape hatches, and per-requirement judgments. It also covers operational guardrails (one call per unchanged decision, don't re-run for a pleasing answer) and epistemic framing ('High probability is not proof'). An agent has enough context to invoke and interpret the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers 100% of parameters with detailed descriptions, so the baseline is 3. The tool description adds minimal parameter-level meaning beyond the schema, aside from framing how many candidates to supply and the distinction between evidence and priorities, which the schema already addresses.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: decide over bounded alternatives, with a distinctive output of a Choice distribution, escape hatches, and per-requirement judgments. The described use cases (implementation alternatives, product tradeoffs, workflow selection) and output artifacts clearly separate it from siblings like jev_classify, jev_rerank, and jev_compare.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when to use it ('unresolved, bounded decision... semantic judgment over supplied evidence') and when not to ('open-ended research, routine choices, correctness proofs, or predicting user consent'). It also gives alternatives—source inspection, tests, the user, or a reasoning model—and adds the critical 'do not repeat a call to obtain a more pleasing answer' rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_extractExtract fields by regex, Jev picks the right matchA

Extract structured fields from a document with TypeSafe Jev as the picker, not the generator: your regex finds candidate substrings in code, Jev chooses which candidate is the field's true value, and the result is returned verbatim — never model-generated text. Fields with zero regex matches never reach the model (not_found); if no field has matches, no API call is made. Ambiguous picks are flagged for review. Use for prices, dates, version numbers, IDs, and anything with a recognizable shape; keep documents bounded.

ParametersJSON Schema
NameRequiredDescriptionDefault
fieldsYesFields to extract. Up to 32 per call, all judged in one request.
purposeNoWhat the extraction is for; shared across fields.
documentYesThe document to extract from. Rejected above 50,000 characters.
auto_acceptNoMinimum top probability for auto. Default 0.85.
minimum_marginNoMinimum winner-to-runner-up gap for auto. Default 0.5.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It thoroughly discloses that results are returned verbatim, that fields with zero matches become 'not_found', that no API call is made if no field has matches, and that ambiguous picks are flagged for review. This is unusually rich behavioral context for a tool without annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence earns its place: the first explains the core behavior, the second covers failure modes, the third covers ambiguity handling, and the last gives practical usage guidance. The description is front-loaded and contains no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description plus full schema coverage gives an agent enough to call the tool correctly. It explains the input semantics, failure behavior, and output philosophy. It does not specify the exact response JSON structure, but since there is no output schema, slightly more precision about the returned shape would be helpful; overall this is still strong.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents each parameter. The description goes beyond that by explaining how the parameters interact: the regex finds candidate substrings, Jev picks the true value using the field description, and zero-match fields never reach the model. It also explains that ambiguity causes review, which adds meaning to auto_accept and minimum_margin.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb ('Extract'), a specific resource ('structured fields from a document'), and the core mechanism ('TypeSafe Jev as the picker, not the generator'). It also differentiates from sibling tools by emphasizing verbatim, non-model-generated results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear use cases: 'prices, dates, version numbers, IDs, and anything with a recognizable shape.' It also advises to 'keep documents bounded.' It does not explicitly name which sibling to use instead for other operations, but the extraction-versus-classification/decision framing makes the intended context fairly clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_findSemantic search over candidatesA

Rank candidates against a plain-language query with TypeSafe Jev — no embeddings needed. One Choice scores every candidate id by how well it answers the query, plus a Noul checks whether any candidate addresses the query at all (so a confident 'top hit' cannot masquerade as an answer). Pattern: docs.typesafe.ai/cookbooks/semantic_find. Use for 'which file/note/line covers X' across up to 250 candidates.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesWhat you are looking for, in natural language.
top_kNoHow many ranked candidates to return. Default 5.
candidatesYesCandidates to search. Up to 250 in one call; texts are truncated at 2000 chars.

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden. It adds meaningful behavioral detail: 'no embeddings needed', 'One Choice scores every candidate', and a 'Noul' check preventing an unsupported 'top hit'. It does not cover output format or error behavior, but for a read-style search tool the core operation is well disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is two information-dense sentences plus a link and a use case. The opening sentence front-loads the main function. Some jargon ('One Choice', 'Noul', 'TypeSafe Jev') could be clearer, but nothing is extraneous.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is moderately complex with a nested candidates array and no output schema. The description gives use context but does not explain return shape, how the 'Noul' result appears, or when to prefer jev_screen/jev_verify over jev_find. These gaps matter given the absence of an output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description mentions plain-language query and candidate count, but these largely repeat schema content. No additional parameter-level insight (e.g. top_k behavior, id conventions) is added beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Rank candidates against a plain-language query') and adds a concrete use case ('which file/note/line covers X'). It is clearly a search/ranking tool, but it does not explicitly contrast itself with siblings jev_screen or jev_verify, so it misses the top score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a clear trigger phrase ('Use for which file/note/line covers X') and a capacity limit (up to 250 candidates). It does not explicitly say when not to use it or reference alternatives among the named siblings, so it falls short of full guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_gateGate completion: review a patch and verify claimsA

Review a proposed patch and verify completion claims against supplied evidence in one TypeSafe Jev call. Auto only when the patch review is accepted and every claim is verified at or above auto_accept. Unsupported claims require review; confident contradictions, unknown confidence, or low confidence escalate. The request and claims are assertions to check, never proof; put supporting diff excerpts and test logs in evidence. Evidence is capped at 16 items and 200,000 characters in aggregate. Does not run tests or apply changes. Use jev_review for a patch without claims, jev_verify for claims without a patch review.

ParametersJSON Schema
NameRequiredDescriptionDefault
diffYesProposed patch, file excerpt, or change summary. Truncated at 50000 chars.
testsNoReported test output for the patch review. Truncated at the same cap.
claimsYesCompletion claims to check against evidence, each truncated at 2000 chars. Up to 16 per call.
requestYesWhat the user asked for; this is not evidence of completion.
evidenceYes
review_atNoScore, safe_to_apply, or per-claim confidence below this escalates. Must be <= auto_accept. Default min(0.5, auto_accept).
auto_acceptNoReview and per-claim confidence at or above this may stand automatically. Default 0.8.
composite_floorNoWeighted composite at or above this is required for auto. Default 0.7.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Even with no annotations, the description discloses key behaviors: 'Does not run tests or apply changes,' evidence caps, claim thresholds, and that request/claims are not proof. This is strong behavioral transparency for a complex tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact yet information-dense. Every sentence adds distinct value: purpose, auto-accept policy, escalation behavior, evidence guidance, safety disclaimer, and sibling routing. No filler or redundant repetition of schema details.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 8 parameters and no output schema, the description covers purpose, safety, thresholds, evidence limits, and alternatives well. A slight gap is the lack of explicit statement about return values or output shape, but the behavioral descriptions largely compensate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is high (88%), so the baseline is 3. The description adds extra meaning by advising that evidence should contain diff excerpts and test logs payll and clarifying that request/claims are assertions, not proof. This goes beyond the schema's generic parameter descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('review a proposed patch and verify completion claims against supplied evidence') and clearly distinguishes this from sibling tools. It names jev_review and jev_verify as alternatives, making the tool's scope unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use this tool versus alternatives: 'Use jev_review for a patch without claims, jev_verify for claims without a patch review.' It also describes auto-accept and escalation conditions, giving clear decision context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_rerankScore every candidate's relevance and return them sortedA

Rerank candidates against a query with TypeSafe Jev: one independent relevance probability per candidate, all in a single request, then sorted by score. Unlike jev_find (which picks one best answer), rerank scores every candidate so the full ordering survives. TypeSafe's rerank cookbook reports that on the CLERC benchmark this pattern lifted top-1 from 5% to 18% and top-10 from 38% to 62% (docs.typesafe.ai/cookbooks). Use for retrieval ordering, dedup triage, or feed ranking across up to 250 candidates.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesWhat relevance is measured against, in natural language.
top_kNoHow many ranked candidates to return. Default: all.
candidatesYesCandidates to search. Up to 250 in one call; texts are truncated at 2000 chars.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses that each candidate receives one independent relevance probability, that all are processed in a single request, and that results are sorted by score. It also mentions the 250-candidate limit. While it doesn't explicitly state the return format or confirm non-mutation, the description gives sufficient behavioral insight for an agent to understand what happens when the tool is invoked.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded: it states the core action first, then differentiates from a sibling, provides a performance benchmark, and lists use cases. Each sentence adds value, though the benchmark detail, while useful, makes it slightly longer than strictly necessary. Still, it remains focused and efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with no output schema and no annotations, the description is fairly complete. It covers the operation, the use cases, and the limits. The only notable omission is an explicit statement of the return structure (e.g., an array of candidates with scores), but that is largely implied by the description and the tool's purpose. Given the richness of the description, it is sufficient for an agent to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 100% coverage, so the description does not need to explain parameters. It does add a minor note about the 250-candidate cap, which matches the schema's maxItems, but it does not enrich the semantic understanding beyond what the schema already provides. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: rerank candidates against a query, scoring each independently and sorting by score. It explicitly differentiates from jev_find by noting that find picks one best answer while rerank scores every candidate, so the full ordering survives. This is a specific verb + resource with clear scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use guidance, naming jev_find as an alternative and explaining the distinction. It also lists concrete use cases: retrieval ordering, dedup triage, and feed ranking. It even mentions a benchmark to suggest when this pattern is beneficial, giving the agent a clear decision heuristic.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_reviewReview a proposed patchA

Score a proposed diff against the request with TypeSafe Jev before the task is called done. Returns 0..2 rubric scores for correctness, spec match, test gap, and blast radius (the last two lower the weighted composite), a safe_to_apply probability, and an auto | review | escalate action. Auto requires safe_to_apply and min score confidence at auto_accept and the composite at composite_floor; truncated or malformed input never returns auto. Does not apply the patch or run tests. Use jev_gate to also verify completion claims against evidence in the same call.

ParametersJSON Schema
NameRequiredDescriptionDefault
diffYesProposed patch, file excerpt, or change summary. Truncated at 50000 chars.
testsNoReported test output, if any. Truncated at the same cap.
requestYesWhat the user asked for; this frames the review, it is not proof of anything.
review_atNoMin score confidence or safe_to_apply below this escalates. Must be <= auto_accept. Default min(0.5, auto_accept).
auto_acceptNosafe_to_apply and min score confidence at or above this may stand automatically. Default 0.8.
composite_floorNoWeighted composite at or above this is required for auto. Default 0.7.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states the tool is non-mutating ('Does not apply the patch or run tests'), explains the auto/escalate decision logic, and discloses truncation behavior ('Truncated at 50000 chars'). It could add more about failure modes or error handling, but the key behavioral traits are well covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but efficient, front-loading the core action and rubric before moving to decision logic and exclusions. Every sentence earns its place, though the decision-logic sentence is long and packs several conditions together. It is appropriately sized for a tool with 6 parameters and complex auto-accept behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a review tool with no output schema, the description does a good job of explaining what the tool returns (rubric scores, safe_to_apply probability, action) and how the action is determined. It could be more complete by describing the exact output format or error behavior, but the essential context for an agent to decide whether to call this tool and interpret its result is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 6 parameters. The description adds meaningful context beyond the schema: it explains how the parameters interact (auto requires safe_to_apply and min score confidence at auto_accept and composite at composite_floor), and clarifies that 'request' frames the review but is not proof. This is valuable semantic glue that the schema alone doesn't provide.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Score'), a specific resource ('a proposed diff against the request'), and a named method ('TypeSafe Jev'). It distinguishes itself from siblings by naming jev_gate as the alternative for verifying completion claims, and the rubric detail (correctness, spec match, test gap, blast radius) makes the tool's function unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use this tool ('before the task is called done') and names the sibling alternative ('Use jev_gate to also verify completion claims against evidence in the same call'). It also states what the tool does not do ('Does not apply the patch or run tests'), which helps an agent avoid misusing it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_screenScreen content before it enters agent contextA

Judge fetched or external text with TypeSafe Jev before an agent reads it: probability it contains instructions aimed at an AI agent (prompt injection), whether it has substantive content, and (when a purpose is given) whether it is relevant to the task. Returns a recommendation: pass | review | block | skip. Pattern: docs.typesafe.ai/cookbooks/llm_guardrails.

ParametersJSON Schema
NameRequiredDescriptionDefault
textYesThe content to screen, e.g. a fetched web page or pasted document.
purposeNoWhat the consuming agent is trying to do; enables a relevance judgment and the 'skip' action.
block_atNoInjection probability at or above which content is blocked. Default 0.75.
review_atNoInjection probability at or above which content is flagged for review. Default 0.25.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does a good job: it explains the evaluation dimensions, the recommendation values (pass|review|block|skip), and the conditional relevance behavior. It does not explicitly state side-effect-free behavior or response structure beyond the recommendation list, but the judging nature is clear.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loads the core purpose, and includes the output contract without fluff. The reference to the cookbook pattern is useful and compact. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given four parameters, no annotations, and no output schema, the description provides enough for an agent to invoke the tool correctly: it states inputs, output categories, and conditional behavior. It could be more explicit about the probability output format or threshold semantics, but the schema already covers threshold parameters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all four parameters clearly. The description adds some context by linking 'purpose' to the relevance judgment and 'skip' action, but it does not meaningfully supplement the parameter meanings beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a clear verb ('Judge'), a specific resource (fetched or external text), and the analysis dimensions (injection probability, substantive content, relevance). It does not explicitly differentiate from the sibling tools jev_verify and jev_find, so it misses the top score for sibling distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly identifies when to use the tool: 'before an agent reads it.' It implies the guardrail context and references a cookbook pattern, giving solid situational context. However, it does not mention when not to use it or alternatives among the siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

jev_verifyVerify claims against evidenceA

Check each claim against provided evidence text with TypeSafe Jev. Returns per claim: verdict (verified | contradicted | unsupported), full probability distribution, confidence, and whether the verdict stands on its own (auto) or needs human review. Pattern: docs.typesafe.ai/cookbooks/citation_check. Pass reports, PR descriptions, or agent briefs as claims and their cited sources, diffs, or documents as evidence.

ParametersJSON Schema
NameRequiredDescriptionDefault
claimsYesClaims to verify, e.g. individual factual statements from a report.
evidenceYes
auto_acceptNoVerdicts at or above this confidence stand automatically; below it they are flagged 'review'. Default 0.8.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly explains what the tool returns per claim—verdict, probability distribution, confidence, and auto/review status—and implies a confidence-threshold behavior through the output. It does not mention side effects or rate limits, but the verification behavior is a read-only-style computation and the output behavior is detailed enough for an agent to anticipate the result.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences and every one earns its place: core action, return format, and practical usage mapping. The most important information is front-loaded, and the writing is compact without sacrificing the behavioral detail an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with two required parameters, one optional threshold, and no output schema, the description covers both what the agent should pass and what it should expect back. The evidence parameter's ability to accept multiple items and map claims to evidence is handled partly by the schema and partly by the description, leaving only minor gaps around exact output formatting.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already documents all three parameters, so the description does not need to repeat their mechanics. The description adds useful mapping examples ('reports, PR descriptions, or agent briefs' as claims; 'cited sources, diffs, or documents' as evidence), but it does not add meaning to auto_accept beyond the schema, giving it only modest added value at this coverage level.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific action ('Check each claim against provided evidence text') and a distinct resource ('TypeSafe Jev'), making the tool's core function unmistakable. It does not explicitly distinguish this from siblings jev_screen and jev_find, but the verification purpose and return categories are specific enough for an agent to separate it from those names.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives concrete usage context: pass reports, PR descriptions, or agent briefs as claims, and cited sources, diffs, or documents as evidence. It does not explicitly state when not to use this tool or name alternatives, but the input examples provide clear practical guidance for selecting appropriate content.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 10 tool updatesv0.1.0
    • First observedjev_classify
    • First observedjev_compare
    • First observedjev_decide
    • First observedjev_extract
    • First observedjev_find
    • First observedjev_gate
    • First observedjev_rerank
    • First observedjev_review
    • First observedjev_screen
    • First observedjev_verify

TDQS

A4.4/5.0

Scored across 10 tools

Disambiguation5/5

Each tool has a distinct verb and purpose: verify claims, screen for injection, find best match, classify, decide, rerank, compare, extract, review, and gate. Even similar tools like jev_find and jev_rerank are clearly differentiated (one picks a single best, the other orders all), and jev_review vs jev_gate are explicitly separated by the presence of claim verification.

Naming Consistency5/5

All tools follow a consistent 'jev_' prefix with a lowercase descriptive verb (verify, screen, find, classify, decide, rerank, compare, extract, review, gate). The naming pattern is uniform and predictable, making it easy for an agent to infer function from the name.

Tool Count5/5

10 tools is well within the ideal 3-15 range. Each tool covers a distinct capability within the Jev evaluation domain, and none feel redundant or unnecessary. The count is appropriately scoped for the server's purpose.

Completeness5/5

The tool set provides comprehensive coverage of evaluation tasks: claim verification, prompt-injection screening, semantic search, classification, decision-making, reranking, pairwise comparison, structured extraction, patch review, and gated review with claim verification. Cross-references between tools (e.g., using jev_gate for patches with claims, jev_review without) indicate a well-thought-out and complete surface with no obvious gaps for the stated domain.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Enables agents to perform typed judgments—classify, score, check, match, and screen—over closed answer sets with confidence scores, without text generation.
    7
    4
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables typed, calibrated judgment calls through classify, score, check, and batched ask tools, each returning full probability distributions for programmatic decisions.
    5
    981 npm
    7
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables agents to evaluate single records or batches with dynamically authored typed questions, returning structured decisions and probabilities.
    MIT