This server provides a read-only, bounded semantic-judgment layer (powered by TypeSafe Jev) that coding agents can use to make routing, ranking, extraction, verification, and escalation decisions without executing actions or changing state.
typesafe_route: Choose one route from a closed set, with probabilities, confidence, and a proceed/review recommendation.
typesafe_rank: Score and order up to 50 retrieved candidates against a concrete rubric.
typesafe_extract: Select supported values from candidates already found in a source, returning no_match instead of inventing values.
typesafe_verify: Judge whether supplied evidence supports a claim, returning a support probability and recommendation.
typesafe_judge: Run up to 25 independent narrow Choice, Score, or Noul questions over shared state.
typesafe_usage_summary: Report current-process Jev call counts, tokens, elapsed time, per-tool breakdown, and declared substitute-model intent.
typesafe_session_mode: Enable, disable, or inspect automatic Jev usage for the current session.
typesafe_escalation_gate: Deterministically combine confidence/risk signals into a proceed/review decision without making model calls.
The tools are idempotent, read-only, and enforce limits on candidates, fields, questions, and signals; they never edit files, approve actions, invoke other models, or contact third parties.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@typesafe-as-a-judge rank these 3 test suites by relevance to the failing test"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
TypeSafe-as-a-Judge is a dual Codex and Claude Code plugin that gives coding agents a bounded semantic-judgment layer powered by TypeSafe Jev.
The agent remains responsible for orchestration, policy, permissions, code changes, and side effects. Jev supplies small typed signals—choices, scores, yes/no probabilities, and confidence—that the agent can use to route work, rank candidates, verify evidence, and decide when to ask for review.
It is deliberately not an autonomous authority layer.
Unofficial community integration: TypeSafe-as-a-Judge is not affiliated with, sponsored by, or endorsed by TypeSafe AI, OpenAI, Anthropic, Codex, or Claude. TypeSafe, Jev, OpenAI, Codex, Claude, and related names are the property of their respective owners.
Arik Aizikovich is currently the sole maintainer and contributor. Contributions are welcome: bug reports, documentation improvements, new examples, test cases, security feedback, and pull requests are all useful.
Please read CONTRIBUTING.md before opening a pull request. For security concerns, follow SECURITY.md instead of opening a public issue. The project uses Apache-2.0; contributions are accepted under the same license unless we explicitly agree otherwise.
Related MCP server: jev-domain-mcp
Tool | Use it when | What it returns |
| One item must be sent to one known route | A route, probability distribution, confidence, and |
| Code has already retrieved a bounded candidate list | Ordered candidates with scores, probabilities, confidence, and a review recommendation |
| Code has already found possible values in a source | A selected candidate or |
| One claim needs to be checked against evidence | Support probability and a |
| Several independent narrow questions should run over shared state | Raw Choice, Score, and Noul signals for the caller's policy |
| A user asks how Jev was used or what it theoretically substituted | Current-process calls, tokens, elapsed time, per-tool breakdown, and declared substitute-model intent |
| A user wants Jev enabled, disabled, or inspected for the current session |
|
| Existing confidence or risk signals need one deterministic gate |
|
The tools are read-only. They do not edit files, approve actions, change records, invoke another model, or contact a third party.
The bundled skill tells Codex and Claude to consider this MCP when a task contains a bounded semantic decision. They do not call it for every request.
The normal pattern is:
Observe or retrieve the relevant state.
Reduce it to the minimum safe context.
Choose the narrowest TypeSafe tool.
Preserve the raw answer, probabilities, confidence, and threshold.
Proceed only when the explicit policy allows it.
Ask for review when confidence is low, no candidate fits, or evidence conflicts.
Coding-agent situation | Tool combination | Autonomous behavior |
An issue could be a bug, feature, documentation task, or investigation |
| Route the issue to one known workflow, or choose |
Several files may implement the requested behavior |
| Rank a retrieved shortlist by relevance before opening or editing candidates |
Several test suites could cover a change |
| Rank candidate suites using changed files, test names, and the requirement |
A failing test could be code, configuration, environment, dependency, or test failure |
| Classify the failure and choose the next diagnostic path |
Several implementation approaches are already known |
| Compare them against a concrete rubric such as compatibility, scope, and testability |
A natural-language request contains a known command, framework, or option |
| Select one value from candidates already found by code; return |
A generated summary makes claims about code or test results |
| Check each important claim against source evidence before presenting it as confirmed |
A citation or documentation link may not support a statement |
| Flag unsupported or partial evidence for review |
An extracted record contains multiple fields |
| Run independent field-level checks and keep each signal separately inspectable |
A plan has conflicting evidence or low-confidence choices |
| Escalate to the user or a stronger reasoning path instead of silently guessing |
A pull request has several possible review areas |
| Rank likely risk areas and verify claims about tests, behavior, and regressions |
A set of available skills could apply |
| Choose among a closed set of skills, with an explicit no-match path |
A task queue contains mixed work |
| Classify each item, then rank candidates within a defined category |
These are recommendations for the agent's next step. A recommendation is not permission to perform a consequential action.
The examples below show the shape of the MCP calls. Codex and Claude normally construct these arguments themselves when the task matches the skill.
Use typesafe_route when exactly one route should receive an item.
{
"state": {
"request": "I was charged twice and need a refund.",
"channel": "support"
},
"instructions": "Which team should handle this request?",
"routes": {
"billing": "Payments, invoices, refunds, and duplicate charges.",
"technical": "Bugs, outages, integrations, and broken product behavior.",
"account": "Login, identity, profile, and account-access issues."
},
"confidence_threshold": 0.8
}Possible result:
{
"route": "billing",
"confidence": 0.93,
"probabilities": {
"billing": 0.96,
"technical": 0.02,
"account": 0.01,
"needs_review": 0.01
},
"recommended_action": "proceed"
}The tool adds needs_review automatically. If the selected route is needs_review or confidence is below the threshold, the result is review.
Use typesafe_rank only after code has retrieved a bounded candidate list.
{
"instructions": "Which implementation is the best fit for adding tenant-scoped audit logging without widening authority?",
"context": {
"constraints": [
"Preserve existing RLS boundaries",
"Keep audit records append-only",
"Do not change authentication semantics"
]
},
"candidates": [
{ "id": "middleware", "value": "Add an audit middleware around protected mutations." },
{ "id": "orm-hook", "value": "Add a global ORM hook for all writes." },
{ "id": "db-trigger", "value": "Add database triggers to protected tables." }
],
"criteria": [
"Poor fit: violates a constraint or requires broad unsafe changes.",
"Good fit: satisfies the constraints with manageable integration work.",
"Excellent fit: satisfies the constraints, is auditable, and has a narrow blast radius."
],
"confidence_threshold": 0.7
}The agent receives an ordered list with a score and confidence for every candidate. It can inspect the top candidate, but should ask for review if the top result is not sufficiently distinct or confident.
Use typesafe_extract when deterministic code has already found possible values. This is selection, not free-form generation.
{
"source": "The deployment target is the EU production cluster in Frankfurt.",
"fields": [
{
"id": "deployment_target",
"instructions": "Which known deployment target is explicitly named by the source?",
"candidates": [
{ "id": "us_staging", "value": "us-staging", "description": "United States staging environment." },
{ "id": "eu_production", "value": "eu-production", "description": "European production environment." },
{ "id": "eu_staging", "value": "eu-staging", "description": "European staging environment." }
]
}
],
"confidence_threshold": 0.8
}If no supplied value is supported, the tool returns no_match and recommends review. It never creates a new environment name from the prose.
Use typesafe_verify for one specific claim and its evidence.
{
"claim": "The migration was replayed successfully on the evaluation database.",
"evidence": {
"command": "pnpm eval:db:fresh",
"exit_code": 0,
"output": "Applied 42 migrations; seed verification passed."
},
"support_threshold": 0.85
}Possible result:
{
"support_probability": 0.91,
"support_threshold": 0.85,
"recommended_action": "proceed"
}The result means the evidence supports the claim according to the question. It is not a deployment attestation, security approval, or substitute for the actual command output.
Use typesafe_judge when several narrow questions share the same state.
{
"state": {
"claim": "The release is ready for production.",
"evidence": {
"tests": "All focused tests passed.",
"migration": "Greenfield replay passed; production migration replay was not run.",
"approval": "No production approval is attached."
}
},
"questions": {
"tests_complete": {
"type": "noul",
"instructions": "Is the available test evidence sufficient for the stated release claim?",
"criteria": {
"true": "The relevant tests and checks are complete for the claim.",
"false": "Important tests or checks are missing."
}
},
"migration_evidence": {
"type": "choice",
"instructions": "What is the strongest migration evidence available?",
"criteria": {
"production_replay": "The intended production migration path was replayed.",
"greenfield_only": "Only a greenfield replay was performed.",
"none": "No meaningful migration evidence is present."
}
},
"approval_present": {
"type": "noul",
"instructions": "Is an explicit production approval present in the evidence?",
"criteria": {
"true": "The evidence contains an explicit approval.",
"false": "The evidence does not contain an explicit approval."
}
}
}
}The agent keeps each answer separate instead of collapsing the evidence into one vague overall score.
Use typesafe_escalation_gate after collecting model signals and deterministic checks.
{
"signals": [
{
"id": "route_confidence",
"value": 0.91,
"threshold": 0.8,
"comparator": "<=",
"reason": "Route confidence is too low."
},
{
"id": "citation_gap",
"value": 0.88,
"threshold": 0.8,
"comparator": ">=",
"reason": "A citation may not support the claim."
}
]
}Result:
{
"recommended_action": "review",
"fired_signals": [
{
"id": "citation_gap",
"value": 0.88,
"threshold": 0.8,
"comparator": ">=",
"reason": "A citation may not support the claim."
}
],
"gate": "max"
}The gate is deterministic. It does not decide what the reviewer should do and cannot execute the next action.
Use typesafe_usage_summary when a user asks how much Jev was used, how long it took, or what it theoretically saved. It records only metadata from Jev calls made by the current MCP server process: call count, input/output tokens, elapsed time, tool name, and an optional declared substitute model.
When a Jev call intentionally replaces a named model, include comparison_model in the call:
{
"comparison_model": "gpt-5.6-luna"
}That records substitution intent; it does not run Luna or assert a measured saving. A summary can then report that Jev handled a number of calls that were declared as alternatives to Luna, alongside the observed Jev tokens and elapsed time. It must say that token, time, and cost savings are theoretical unless a real baseline run exists.
The telemetry is process-local and resets when the MCP server restarts. It stores no request bodies, source text, credentials, or customer data.
Use typesafe_session_mode when a user explicitly controls the feature:
{
"mode": "enabled",
"billing_context": "plan",
"reason": "Prefer bounded Jev judgments for this session to conserve the main agent's plan context."
}Use mode: "disabled" to block Jev tool calls for the current MCP process, or mode: "auto" to restore the automatic policy. Add "use_default": "jev-preview" (or jev-1.13.0, jev-latest, any Jev model id) to choose which model this session's calls use; see Models and long context. The tool does not edit Codex or Claude configuration and does not change the user's subscription or API billing.
In automatic mode, the skill favors Jev for repeated or bounded semantic decisions—routing, ranking, candidate selection, verification, and confidence gates—especially when plan/quota conservation or review reduction matters. It does not favor Jev for exact deterministic work, open-ended implementation, permissions, tenant isolation, security authority, financial actions, destructive operations, or unsafe sensitive data.
An LLM alternative will often consume a different token shape than Jev. Jev receives one narrow structured state and question set; an agentic LLM may also receive system instructions, repository rules, skill guidance, tool schemas, retrieved context, and reasoning tokens. Do not assume that Jev input and output tokens equal the alternative model's tokens.
The measured development-planning example above showed a harness-specific input factor of 11.3x: the end-to-end Luna run reported 13,236 input tokens while Jev received 1,174 input tokens for the same internal planning question. That is 13,236 / 1,174 = 11.27. It is not a universal Jev-versus-LLM multiplier:
the Luna number includes Codex's agent and repository context;
only 4,276 of those Luna input tokens were uncached, while 8,960 were cached;
the output relationship did not match the input relationship (52 Luna output tokens versus 71 Jev output tokens);
another task, client, reasoning setting, cache state, or retrieval strategy can produce a very different factor.
When estimating an unrun alternative, report the method explicitly. A useful format is:
estimated baseline input = observed Jev input x chosen input factor
estimated baseline output = observed Jev output x chosen output factor
estimated baseline cost = estimated tokens x documented API ratesUse an observed historical median for the same task class when available. If none exists, a same-token estimate is a simple lower-confidence assumption; the 11.3x input factor above is only an illustrative context-heavy Codex harness observation, not a default to apply automatically.
Jev API usage and a coding-agent subscription are different cost planes:
A Jev call creates external TypeSafe API usage and therefore can add direct cash spend.
A Codex or Claude subscription may not create an incremental per-call API charge, but it still consumes plan quota, rate-limit capacity, context budget, and elapsed time.
An API-equivalent price for a subscription-backed run is an analytical estimate, not necessarily what the user was billed.
A theoretical alternative-model estimate can help compare likely token and API-price exposure, but it cannot prove time saved, quality improved, or money saved without a baseline run.
The intended decision is therefore not just “which call is cheaper?” Consider direct cash spend, subscription/quota consumption, latency, consistency, the probability of review, and the cost of an incorrect decision. typesafe_usage_summary must label any unrun baseline as theoretical and must never present it as measured savings.
Choice and Score answers include a confidence value derived from their probability distribution. Noul answers provide a probability that the statement is true; Noul does not provide a separate confidence value.
The agent should:
preserve the raw probability distribution and threshold in its work record;
treat needs_review and no_match as review outcomes;
escalate low-confidence results instead of silently selecting a winner;
use a max-style gate when one serious independent signal should be enough to trigger review;
calibrate thresholds against representative examples and the cost of a wrong decision;
keep useful uncertainty visible in the final explanation.
Confidence is not correctness. A high-confidence answer can still be wrong, and a low-confidence answer can reflect missing evidence rather than a false conclusion.
This example uses a realistic internal planning question that arises while extending this MCP. It is included to illustrate the decision shape, latency, and token accounting—not as a general performance benchmark.
User request: Add an MCP tool named typesafe_compare that compares two proposed implementation plans, returns an ordered score with a review recommendation, and adds tests and documentation.
Internal question: Which existing file should the coding agent inspect first?
Candidates:
server/judge.mjs - TypeSafe API client and existing tool handlers.
server/index.mjs - MCP tool schemas and JSON-RPC dispatch.
tests/judge.test.mjs - unit tests for tool behavior.
README.md - tool contracts and use cases.
Observation: The agent should locate the primary MCP integration surface before planning the implementation, tests, and documentation.
Measure | Ordinary reasoning | TypeSafe MCP judgment |
Model |
|
|
Result |
|
|
Elapsed time | 6.40 s | 0.861 s |
Agent usage | 13,236 input, 8,960 cached input, 52 output tokens | 1,174 input, 71 output tokens |
Price estimate | ~$0.00110 API-equivalent | At least ~$0.0000493 input-only |
Action |
|
|
Reason | It defines MCP schemas and JSON-RPC dispatch, so it is the first integration surface to inspect. | It ranked first with score 1.98 and confidence 0.98. |
Both paths selected the same first file and the same action. The TypeSafe path adds a reusable score, probability distribution, and confidence signal that a deterministic gate can consume.
The Luna estimate uses the current API rates of $0.20 per million uncached input tokens, $0.02 per million cached input tokens, and $1.20 per million output tokens. It is an API-equivalent calculation; Codex subscription billing can differ. See the GPT-5.6 Luna model page.
The TypeSafe estimate uses the published $42 per billion input tokens: 1,174 x $42 / 1,000,000,000 = $0.000049308. TypeSafe's public figure did not show output-token pricing, so this is a lower bound. See TypeSafe AI.
The two runs answer the same internal question, but they are not a controlled benchmark. The left side is an end-to-end Codex run; the right side is a direct MCP ranking call. Their token accounting measures different layers.
Do not use this MCP as the authority for:
authorization, authentication, RLS, tenant isolation, or permission grants;
financial calculations, payments, billing, or account changes;
security approval or vulnerability disposition;
production deployment, migration completion, or release approval;
destructive actions or irreversible external operations;
exact arithmetic, dates, identifiers, or deterministic lookups;
open-ended writing or general reasoning that does not need a typed judgment.
Those decisions belong to deterministic code, existing governance, explicit evidence, and human approval gates.
Send only the minimum state needed for one judgment. Do not send:
API keys, passwords, access tokens, or private keys;
payment data or secrets from configuration files;
unrelated tenant or company data;
unredacted sensitive personal information;
large repositories when a small excerpt or candidate list is sufficient.
The plugin returns TypeSafe usage metadata, but it does not persist a semantic decision history. Callers should decide what to retain in their own auditable work records.
codex plugin marketplace add E-FL/typesafe-as-a-judge
codex plugin add typesafe-as-a-judge@typesafe-as-a-judgeThen configure the TypeSafe token from a checkout of this repository:
git clone https://github.com/E-FL/typesafe-as-a-judge.git
cd typesafe-as-a-judge
powershell -ExecutionPolicy Bypass -File .\scripts\setup.ps1The Windows setup prompts without echoing the token, validates it with a harmless Jev request, and stores it in the Windows user environment. Restart Codex and start a new task after setup.
claude plugin marketplace add E-FL/typesafe-as-a-judge
claude plugin install typesafe-as-a-judge@typesafe-as-a-judgeRun the same guided setup from the repository checkout. Restart Claude Code and approve the project/plugin MCP server when prompted.
Claude Code reads the root .mcp.json (it locates the launcher with ${CLAUDE_PLUGIN_ROOT}); Codex reads .codex-mcp.json. Keep the two in sync when you change the server command.
git clone https://github.com/E-FL/typesafe-as-a-judge.git
cd typesafe-as-a-judge
chmod +x ./scripts/setup.sh
./scripts/setup.shThe Unix setup stores the token at ~/.config/typesafe-as-a-judge/token with owner-only permissions. The MCP launcher reads that file when TYPESAFE_API_KEY is not already present.
Never paste a TypeSafe token into Codex or Claude chat, commit it to a file, or place it in MCP command-line arguments.
This plugin is a local STDIO MCP server. Codex and Claude launch it as a local process and pass TYPESAFE_API_KEY through the session environment. It does not use an OAuth login button or a hosted credential broker.
The TypeSafe request is:
POST https://api.typesafe.ai/v1/systemone
Authorization: Bearer <TYPESAFE_API_KEY>
model: jev-latestThe token is never included in the plugin package or repository.
The built-in default is jev-latest, not the preview. TypeSafe points jev-latest at the latest stable release (jev-1.13.0 today). Jev 1.14 (jev-1.14.0, available from 14 October 2026) is reachable as jev-preview or by its versioned id. Nothing changes for you until you opt in.
Set TYPESAFE_USE_DEFAULT once in the environment that launches Codex or Claude, then restart them. Every session uses that model from then on, and both plugin manifests already pass the variable through.
[Environment]::SetEnvironmentVariable("TYPESAFE_USE_DEFAULT", "jev-preview", "User")To go back to the built-in default, remove the variable (or set it to jev-latest).
Any Jev model id is accepted: a version such as jev-1.13.0 or jev-1.14.0, or an alias such as jev-latest or jev-preview. Choose it at the level you need:
Per call (explicit): pass model to typesafe_route, typesafe_rank, typesafe_extract, typesafe_verify, or typesafe_judge, for example "model": "jev-1.13.0" or "model": "jev-latest". This wins over everything else, so it also overrides a jev-preview environment default for that one call.
Per session: call typesafe_session_mode with use_default, for example "use_default": "jev-1.14.0". It applies to every Jev call in the running MCP process until it restarts. Pass an empty string or "default" to clear it.
Per environment: TYPESAFE_USE_DEFAULT, as above.
Precedence, highest first: the per-call model, the session use_default, TYPESAFE_USE_DEFAULT, then jev-latest.
Pin a versioned id when you have tuned confidence thresholds to a specific Jev version. Aliases can move to a newer release. The response model field always shows the versioned id that answered.
Pass long_context: true to the same tools to send TypeSafe-Danger-Zone-Long-Context: true. This raises TypeSafe's per-request and per-question limits from 64k to 262k tokens, and the plugin's local request limit from 256 KiB to 1 MiB. TypeSafe documents it as slower and less accurate, so use it only when the state does not fit.
TypeSafe's experimental image input and box question type (header-gated, aimed at jev-preview) need file and image handling that this read-only, text-only plugin does not have.
Node.js 20 or later is required.
Requests are limited to 256 KiB before being sent (1 MiB with long_context).
Transient TypeSafe 429 and 529 responses receive bounded exponential backoff.
The server returns TypeSafe model and usage metadata with judgments.
typesafe_rank accepts at most 50 candidates.
typesafe_extract accepts at most 20 fields and 100 candidates per field.
typesafe_judge accepts at most 25 independent questions.
typesafe_escalation_gate accepts at most 100 signals.
The MCP server writes protocol messages to stdout; diagnostics must not be written to stdout.
Install no runtime dependency: the MCP server uses Node.js built-ins so a fresh marketplace install works without node_modules.
Run the local tests:
npm testThe test suite covers tool behavior, missing-token failure, the launcher, and the JSON-RPC MCP handshake without making a live TypeSafe request.
Validate the Codex and Claude manifests:
py -3 "$env:USERPROFILE\.codex\skills\.system\plugin-creator\scripts\validate_plugin.py" (Get-Location)
claude plugin validate .For direct MCP debugging:
npm startThe process communicates over standard input/output. Do not write normal logs to stdout.
Apache-2.0. See LICENSE.
Apply a deterministic max-style review gate to explicit probabilities or confidences. No model call, writes, or external action occurs.
Inputs: signals (required).
Select supported values from candidates code already found in a source. It cannot generate values. Returns selected values, confidence, and review fields.
Inputs: fields (required), source (required), comparison_model, confidence_threshold.
Run up to 25 independent, narrow TypeSafe Choice, Score, or Noul questions against shared state. Returns raw typed signals for the caller's explicit policy.
Inputs: state (required), questions (required), comparison_model.
Score and rank up to 50 retrieved candidates against a concrete rubric. Returns scores, confidence, and a review recommendation; it never selects or changes a record.
Inputs: criteria (required), candidates (required), instructions (required), context, +2 more.
Choose one route from a closed set using TypeSafe Jev. Returns a route plus a confidence-aware proceed/review recommendation; it never performs the route.
Inputs: state (required), routes (required), instructions (required), review_route, +2 more.
Summarize Jev calls made by the current MCP server process: tool count, token usage, elapsed time, and declared substitute-model intent. It never runs a baseline and labels savings as theoretical.
Inputs: comparison_model.
Judge whether supplied evidence directly supports a claim. Returns a support probability and an explicit proceed/review recommendation; it never treats a claim as proven on its own.
Inputs: claim (required), evidence (required), context, comparison_model, +1 more.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 7 tools
Most tools are individually clear, but typesafe_judge can execute Choice, Score, and Noul questions, which conceptually overlaps with route, rank, extract, and verify. The descriptions provide enough context to help avoid misselection, but the boundaries are not always crisp.
All tools share a uniform typesafe_ prefix and snake_case style, making the family recognizable. The majority use verb-style names like route, rank, extract, verify, and judge, though usage_summary and escalation_gate are noun-style deviations.
Seven tools is well-scoped for a specialized judging and evaluation server. Each tool addresses a distinct phase or concern without feeling padded or redundant.
The surface covers the core judge lifecycle: route, rank, extract, verify, batch judging, usage tracking, and escalation gating. Minor gaps such as a dedicated audit or explanation tool are possible, but they are not obvious blockers for the server's stated purpose.
Pre-execution governance for AI agents. Deterministic PASS/FAIL/REVIEW verdicts, replayable proof.
Typed AI decisions for agents: classify, route, screen evidence, score and batch-label text.
Human judgment for AI agents: discover capabilities, get quotes, and track paid human tasks.
Trust signals for AI agents: an open agent-readiness standard and developer tool guide. Read-only.
