Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@JevLab MCP Server benchmark my recorded judge against golden labels and show the calibration report"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Evaluate the judge. Measure bias. Calibrate confidence.
JevLab is an offline, standard-library Python toolkit for LLM-as-judge evaluation, bias diagnostics and calibration. Compare recorded judges with golden labels, fit probability calibrators, inspect an HTML reliability report, and expose the same tools to an MCP client.
Apache-2.0 · Python 3.10+ · Repository
Install from this checkout (a package-registry release is not assumed):
python3 -m pip install .
jevlab bench
jevlab bench --fixture fixtures/synthetic_sample.json --json
jevlab calibrate --output calibration_report.html
jevlab calibrate --fixture fixtures/synthetic_sample.json --pairs fixtures/bias_pairs.json --json > calibration_report.json
jevlab bias-audit --pairs fixtures/bias_pairs.json --json
jevlab rubrics list
jevlab rubrics show code_review_correctnessOpen calibration_report.html in a browser. It is a self-contained dark report with inline SVG reliability curves, before/after ECE, MCE and Brier, agreement heatmaps, bias tables and fit-selected thresholds. It requires no server, JavaScript, remote fonts or network access. Expand each method to compare Platt, temperature and histogram calibration. The React studio consumes the JSON version.
Without installation: PYTHONPATH=. python3 -m jevlab_oss.cli bench. python3 -m jevlab, jevlab, and jevlab-oss use the same CLI. Both jevlab.* and historical jevlab_oss.* imports work. The original metrics.py, fixtures.py, judges.py, and bench.py are preserved; the enhanced API lives in calibration.py.
Related MCP server: agent-eval-gate
For binary judgments, supply p(yes) and a boolean golden label, not confidence in whichever class won. Humans supply the labels; bundled labels and judge outputs are simulated. Fit calibrators and select decision thresholds on split: "fit". Only then evaluate on split: "holdout". Keep cases, related variants and near duplicates in the same split to prevent leakage.
For occupied equal-width bins Bₘ, with mean predicted probability p̄ₘ and observed positive frequency ȳₘ:
ECE = Σₘ (|Bₘ| / N) · |p̄ₘ − ȳₘ|.
MCE = maxₘ |p̄ₘ − ȳₘ|; empty bins are omitted.
Brier = (1/N) Σᵢ (pᵢ − yᵢ)².
Platt scaling: p′ = sigmoid(a · logit(p) + b).
Temperature scaling: p′ = sigmoid(logit(p) / T), T > 0; the intercept is fixed at zero.
Histogram calibration: p′ = (positive count + α) / (bin count + 2α), α = 1 by default. Empty fit bins retain p.
Platt and temperature scaling use pure-Python gradient descent with backtracking on binary cross-entropy and a tiny ridge penalty. Logits clip probabilities to [10⁻⁶, 1−10⁻⁶]. Temperature scaling preserves ranking; unconstrained Platt can invert a systematically reversed judge. Histogram calibration need not preserve ranking. The legacy calibrator in metrics.py retains its original damped Newton implementation.
from jevlab.calibration import PlattCalibrator, HistogramCalibrator, TemperatureCalibrator
fit_scores = [.99, .99, .99, .01, .01, .01]
fit_labels = [True, True, False, False, False, True]
# Use independently labeled cases for evaluation, even when scores happen to match.
holdout_scores = [.98, .97, .96, .02, .03, .04]
holdout_labels = [True, False, True, False, True, False]
model = PlattCalibrator().fit(fit_scores, fit_labels)
comparison = model.evaluate(holdout_scores, holdout_labels)
print(comparison["improvement"]) # before minus after: negative means worseA positive improvement means lower error. ECE depends on binning and sample size; a better ECE does not guarantee a better Brier score. The report displays regressions without clipping them. Do not repeatedly pick models against the same holdout and call that an unbiased evaluation. Use a separate validation split for model selection and a final untouched test set for a release decision. See Guo et al., 2017 for background on temperature scaling.
The threshold recommendation minimizes empirical equal-cost errors on fit data, with deterministic tie breaking toward 0.5. It includes all-positive and all-negative rules (a threshold just above 1 means always negative). The report separately shows holdout accuracy at that threshold. optimal_threshold(..., false_positive_cost=..., false_negative_cost=...) supports explicit positive error costs. It does not infer your business cost model.
A pair fixture has synthetic: true, license: "Apache-2.0", and a pairs array:
{
"synthetic": true,
"license": "Apache-2.0",
"pairs": [{
"judge": "judge-a", "a": "Four.", "b": "## Answer\n- Four.",
"forward": 0.8, "swapped": 0.2,
"judge_model": "model-a", "a_model": "model-a", "b_model": "model-b"
}]
}forward is p(first wins) for A/B; swapped is p(first wins) for B/A. The same candidate winning both orders is consistent. If both probabilities exceed 0.5, the judge picked the first position twice: the candidate winner flipped. Exact 0.5 is a tie, excluded from decisive flip rates and reported separately.
BiasAuditor().audit(pairs) returns per-judge:
Position swap flip rate; first/second win rates; signed first-minus-second bias; mean probability inconsistency.
Spearman rank correlation (average ranks for ties) between character-length delta and order-averaged A preference (forward + 1 − swapped) / 2.
Correlation with markdown/list-line density delta. Density is the fraction of nonblank lines containing recognizable markdown markers.
Self-preference using judge_model, a_model, and b_model: only pairs with exactly one own-model response are eligible. Reports mean own-response preference and correlation with own-model position.
Constant vectors, insufficient samples and absent provenance produce null, not a fabricated zero. Style correlation alone cannot establish self-preference. Match quality and content across controlled pairs; observational correlations are not proof of causal bias. The default bias demo is separate from the calibration fixture and is explicitly synthetic; custom calibration fixtures get no demo bias rows unless supplied via --pairs.
from jevlab import AgreementMatrix
matrix = AgreementMatrix({"small": [True, False, True], "large": [True, False, False]})
print(matrix.pairwise()) # Cohen's kappa, including undefined (None) cells
print(matrix.fleiss_kappa()) # binary multi-rater kappa
print(matrix.pareto_frontier({"small": 0.01, "large": 0.05}, [True, False, False]))All raters must label the same aligned items; missing ratings are rejected. Cohen’s κ = (observed agreement − chance agreement) / (1 − chance agreement). Fleiss’ κ uses each item's pairwise agreement and pooled category proportions; it is undefined with fewer than two raters or perfect chance agreement. Agreement is not accuracy.
For cost comparisons, supply judge_costs in your fixture as a map of every judge name to a finite, non-negative cost in one common unit on the same workload. A judge dominates another only if it is no more expensive and no less accurate, with at least one strict improvement. Rank 1 is the Pareto frontier; subsequent ranks remove preceding frontiers. Duplicate points share a rank. Missing costs produce no ranking; nothing is assumed to be free.
{
"schema_version": 1, "synthetic": true, "license": "Apache-2.0",
"rubric": {"id": "example", "questions": [{"id": "q1", "text": "Does the answer cite evidence?"}]},
"judge_costs": {"judge-a": 0.01},
"items": [
{"id": "f1", "split": "fit", "question_id": "q1", "text": "Example evidence.", "truth": true, "judgments": {"judge-a": 0.9}},
{"id": "h1", "split": "holdout", "question_id": "q1", "text": "Another example.", "truth": false, "judgments": {"judge-a": 0.4}}
]
}The two-row snippet illustrates structure, not an adequate calibration dataset. Every item must record the same judges and both splits must exist. Existing fixture scrub rules are preserved: the release harness requires synthetic fixtures under Apache-2.0 and rejects non-example email domains. A declaration alone is not a privacy scrub; inspect all data before sharing. The Python calibrator API can process separately governed local arrays without publishing them as fixtures.
Legacy commands remain available:
jevlab validate fixtures/synthetic_sample.json
jevlab template
jevlab run fixtures/synthetic_sample.json --out benchmark.json
jevlab run fixtures/synthetic_sample.json --judge-cmd 'custom=python3 adapter.py'A CommandJudge receives {"question": ..., "text": ...} on stdin and must return {"p_yes": 0.8}. The executable inherits the launching environment; use only trusted adapters and environment variable names for provider secrets. The core harness has no provider SDK or networking dependency. MCP deliberately does not expose command judges.
rubrics/*.yaml are JSON-form YAML 1.2 for dependency-free, safe loading. Each has a purpose, scoring instructions and three evidence-based yes/no questions. Adapt them to your task and validate with human labels:
ID | Measures |
| Grounded findings, behavior and actionable fixes |
| Source support, citation entailment and uncertainty |
| Scope, necessary changes and compatibility |
| Trust boundaries, plausible impact and mitigations |
| Tool fit, arguments and result handling |
| Row multisets, NULL semantics and ordering |
| Source support, coverage and qualifiers |
| Schemas, error semantics and authorization contracts |
| Observable behavior, regression detection and boundaries |
| Progress, evidence and justified tool use |
These are generic, newly authored templates, not production compliance rubrics.
After pip install ., configure a client supporting local stdio servers:
{
"mcpServers": {
"jevlab": { "command": "jevlab-mcp", "args": [] }
}
}Use the installed executable's absolute path if the client has a different PATH. Alternatively use python3 -m jevlab_oss.mcp_server from an environment where JevLab is installed. There is no HTTP endpoint or API key to configure.
Tool | Optional arguments | Result |
|
| Legacy benchmark JSON |
|
| Calibration JSON, including agreement and bias |
|
| Paired bias JSON |
| none | All ten rubric objects |
Omitted data uses bundled synthetic demos. Tools accept inline data only, with a 1 MiB character message limit, 1,000 items/pairs and 16 recorded judges. Larger runs use the Python API. MCP tools are read-only and perform no command execution, path reads, report writes or network calls. Use the CLI to create HTML.
The server implements the MCP 2025-06-18 stdio transport: one JSON-RPC message per line, protocol-only stdout, initialization followed by notifications/initialized, ping, tools/list, and tools/call. Results include text plus structured JSON. Tool validation failures return isError; protocol failures use JSON-RPC error codes. Notifications produce no replies and do not execute tools.
Capability | Open-source JevLab (Apache-2.0) | OmniSI Cloud JEV Oracle |
Offline evaluation and synthetic fixtures | Included, pure Python | Managed integration |
Calibration, bias diagnostics and agreement | Included | Managed production workflows |
HTML reliability reports and React studio | Included | Hosted application integration |
Public rubric starter templates | Ten included | Separate proprietary production compliance rubrics |
Local stdio MCP tools | Included | Separate authenticated cloud interfaces |
Multi-tenant Rust oracle and ledger | Excluded | Proprietary |
Live consensus router | Excluded | Proprietary |
Customer corpora, provider credentials, internal prompts | Excluded | Private, outside this release |
No Rust source, crates, Cargo manifests, proprietary router logic or production compliance rubrics are distributed. There is no Rust dependency or build step. JevLab can evaluate outputs from any judge; it does not require the cloud service.
python3 scripts/audit_release.py
python3 -m unittest discover -s tests
python3 -m pip install build
python3 -m build
cd ui
npm install --ignore-scripts --package-lock=false
npm run typecheck
npx playwright install chromium
npx playwright testCI runs on GitHub Actions with Python 3.10/3.12/3.13, tests the wheel outside the source tree, checks the standalone component, and exercises report imports, tabs and desktop/mobile rendering. CI builds artifacts only; it never deploys or publishes a package. .github/workflows/ci.yml works at this repository's root without any parent repository files.
The source fixtures/ and rubrics/ assets are mirrored into jevlab_oss/data/ for wheel installation. Update both copies together; the release audit rejects drift. See CONTRIBUTING.md, SECURITY.md, NOTICE, and the explicit export boundary in audit-manifest.json.
Benchmark recorded judges on synthetic gold labels; default is an offline demo.
| Name | Required | Description | Default |
|---|---|---|---|
| n_bins | No | ||
| fixture | No | Synthetic v1 fixture; omit for bundled demo. At most 1000 items and 16 judges. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive, closed-world behavior, so the safety profile is covered. The description adds a real behavioral caveat not present in structured data — that scoring runs against synthetic gold labels and defaults to an offline demo — but says nothing about runtime, cost, or what a benchmark run actually produces.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with a semicolon separating the core action from the default-behavior note. No filler, no repetition of the title or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only tool with no output schema the bar is lower, but with one of two parameters undocumented, a nested fixture object, and no explanation of what a benchmark returns or how it relates to the other jevlab_* tools, the description is thinner than the tool's surface area warrants.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: the fixture parameter is documented in the schema, but n_bins has no description anywhere and its meaning (histogram binning? something else?) is left entirely to inference. The description text itself contributes no parameter information at all.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource ('benchmark recorded judges') plus the data source ('synthetic gold labels'), so an agent knows this evaluates judges rather than auditing or calibrating them. It does not explicitly differentiate itself from jevlab_calibrate or jevlab_bias_audit, and 'recorded judges' is jargon that isn't defined.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The clause 'default is an offline demo' hints at a zero-argument invocation mode but gives no when-to-use guidance, no conditions for supplying a fixture, and no routing against the sibling tools. The agent must infer when to prefer this over calibrate or bias_audit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Measure position, length, format and provenance-based self-preference; default is synthetic.
| Name | Required | Description | Default |
|---|---|---|---|
| pairs | No | Synthetic matched response pairs with a, b, forward, swapped and optional judge/model IDs. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the full safety profile (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description usefully adds that four distinct bias classes are measured, but says nothing about how the audit runs, what it returns, or what 'synthetic' default implies operationally.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One front-loaded sentence with no filler. It is efficient, though the trailing 'default is synthetic' clause is cryptic and slightly under-explained for the space it consumes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries the burden of explaining what the audit reports, and it does not. For a tool whose result is the whole point (bias scores across four dimensions), the absence of any return-value or result-format context leaves a real gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents the single 'pairs' parameter and its expected a/b/forward/swapped structure. The description's 'default is synthetic' only loosely connects to that and adds no syntax or format detail, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Measure') and enumerates the bias dimensions examined (position, length, format, provenance-based self-preference), which is far more than a restatement of the tool name. It is clearly distinguishable from siblings like jevlab_bench, jevlab_calibrate and jevlab_list_rubrics, though it never explicitly references them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The only guidance is 'default is synthetic,' which hints at a default mode but does not say when to use this audit versus bench or calibrate. There are no prerequisites, exclusions, or named alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fit Platt, temperature and histogram scaling; return holdout reliability and thresholds as JSON.
| Name | Required | Description | Default |
|---|---|---|---|
| pairs | No | Synthetic matched response pairs with a, b, forward, swapped and optional judge/model IDs. | |
| n_bins | No | ||
| fixture | No | Synthetic v1 fixture; omit for bundled demo. At most 1000 items and 16 judges. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so safety is covered structurally. The description adds useful behavioral context by disclosing the return payload (holdout reliability and thresholds as JSON) and that it is a fitting/read-style computation, but says nothing about input constraints (1000-pair cap, 16-judge limit) or failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single tightly packed sentence: methods first, output second, zero filler. Every clause earns its place and the most important information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description correctly summarizes the return payload, which is its main virtue. However, it omits the mutual exclusivity/priority of pairs vs fixture, the size limits, and any routing guidance against sibling tools, leaving real gaps for a 3-parameter nested-input tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 67%, so the schema already documents pairs and fixture reasonably well; n_bins has no description in either place, and 'histogram scaling' only obliquely hints at it. The description adds no syntax, format, or interaction details (e.g., fixture vs pairs precedence) beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action (fit calibration), the concrete methods (Platt, temperature, histogram scaling), and the deliverable (holdout reliability and thresholds as JSON). It is distinguishable from siblings like jevlab_bench or jevlab_bias_audit, though it never explicitly contrasts itself with them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no when-to-use guidance, no statement of prerequisites, and no mention of alternatives among jevlab_bias_audit, jevlab_bench, or jevlab_list_rubrics. The agent must infer from the name and parameter names alone that this is the calibration step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the ten public Apache-2.0 rubric templates.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and closed-world, so the safety/idempotency profile is covered. The description adds genuinely useful behavioral facts beyond that: exactly ten results and an Apache-2.0 public licensing scope. It does not discuss caching, versioning, or the shape of the returned templates.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with no filler; every word (ten, public, Apache-2.0) carries information. Nothing could be trimmed without losing detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, so the description carries some burden of describing what comes back. It discloses count and license but not the structure or fields of a rubric template, which an agent would need to act on the results. Adequate but leaves a clear gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to clarify and the baseline of 4 applies. The description does not need to explain argument semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Return') and resource ('rubric templates') with useful scope qualifiers: 'ten', 'public', 'Apache-2.0'. This distinguishes it from sibling tools like jevlab_bench or jevlab_calibrate, though it never explicitly names a sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: an agent can infer you call this to enumerate the built-in rubric templates. There is no explicit statement of when to call it versus alternatives, nor any prerequisite or caveat (e.g., whether these are the only rubrics available).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0jevlab_benchjevlab_bias_auditjevlab_calibratejevlab_list_rubricsScored across 4 tools
Each tool targets a distinct evaluation activity: bias auditing, rubric listing, benchmarking, and calibration. The bias audit and benchmarking tools could be loosely confused as both assess judges, but the descriptions clearly distinguish bias measurement from gold-label benchmarking.
All tools share the jevlab_ prefix and snake_case, but the naming pattern is mixed: jevlab_list_rubrics is verb_noun, jevlab_calibrate is a bare verb, jevlab_bench is a short form, and jevlab_bias_audit is a noun phrase. This is readable but not predictably consistent.
Four tools is a well-scoped set for a focused judge-evaluation toolkit. Each tool covers a distinct core capability without excess or apparent redundancy.
The surface covers the key lifecycle steps for judge auditing: listing rubrics, auditing bias, benchmarking, and calibrating. Minor gaps exist around custom rubric management and fetching individual rubrics, but agents can work around them.
Independent statistical feature evaluation for machine learning and forecasting. ML Referee tests up to 1,000 candidate features in one run against a declared baseline, helping researchers and AI agents decide which inputs to pursue, which bounded feature repairs merit further testing, and where more evidence is needed—without a separate ablation run for every feature. Use the MCP tools to discover input schemas, check access and credits, preflight a dataset, obtain a quote, submit an authorized evaluation, and retrieve structured results and reports. Results are scoped to the submitted evidence and evaluation design; they do not guarantee future model performance. Invite-only beta with manual approval. Public tool discovery requires no authentication; account and evaluation tools require OAuth through approved client connections and available evaluation credits. Connect directly to https://api.mlreferee.com/mcp. Website: https://mlreferee.com Request beta access: https://mlreferee.com/beta Agent setup and documentation: https://mlreferee.com/agents
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
