JevLab

Evaluate the judge. Measure bias. Calibrate confidence.

JevLab is an offline, standard-library Python toolkit for LLM-as-judge evaluation, bias diagnostics and calibration. Compare recorded judges with golden labels, fit probability calibrators, inspect an HTML reliability report, and expose the same tools to an MCP client.

Apache-2.0 · Python 3.10+ · Repository

Quickstart

Install from this checkout (a package-registry release is not assumed):

python3 -m pip install .
jevlab bench
jevlab bench --fixture fixtures/synthetic_sample.json --json
jevlab calibrate --output calibration_report.html
jevlab calibrate --fixture fixtures/synthetic_sample.json --pairs fixtures/bias_pairs.json --json > calibration_report.json
jevlab bias-audit --pairs fixtures/bias_pairs.json --json
jevlab rubrics list
jevlab rubrics show code_review_correctness

Open calibration_report.html in a browser. It is a self-contained dark report with inline SVG reliability curves, before/after ECE, MCE and Brier, agreement heatmaps, bias tables and fit-selected thresholds. It requires no server, JavaScript, remote fonts or network access. Expand each method to compare Platt, temperature and histogram calibration. The React studio consumes the JSON version.

Without installation: PYTHONPATH=. python3 -m jevlab_oss.cli bench. python3 -m jevlab, jevlab, and jevlab-oss use the same CLI. Both jevlab.* and historical jevlab_oss.* imports work. The original metrics.py, fixtures.py, judges.py, and bench.py are preserved; the enhanced API lives in calibration.py.

Related MCP server: judgment-pack-runtime

Calibration, with an honest holdout

For binary judgments, supply p(yes) and a boolean golden label, not confidence in whichever class won. Humans supply the labels; bundled labels and judge outputs are simulated. Fit calibrators and select decision thresholds on split: "fit". Only then evaluate on split: "holdout". Keep cases, related variants and near duplicates in the same split to prevent leakage.

For occupied equal-width bins Bₘ, with mean predicted probability p̄ₘ and observed positive frequency ȳₘ:

• ECE = Σₘ (|Bₘ| / N) · |p̄ₘ − ȳₘ|.

• MCE = maxₘ |p̄ₘ − ȳₘ|; empty bins are omitted.

• Brier = (1/N) Σᵢ (pᵢ − yᵢ)².

• Platt scaling: p′ = sigmoid(a · logit(p) + b).

• Temperature scaling: p′ = sigmoid(logit(p) / T), T > 0; the intercept is fixed at zero.

• Histogram calibration: p′ = (positive count + α) / (bin count + 2α), α = 1 by default. Empty fit bins retain p.

Platt and temperature scaling use pure-Python gradient descent with backtracking on binary cross-entropy and a tiny ridge penalty. Logits clip probabilities to [10⁻⁶, 1−10⁻⁶]. Temperature scaling preserves ranking; unconstrained Platt can invert a systematically reversed judge. Histogram calibration need not preserve ranking. The legacy calibrator in metrics.py retains its original damped Newton implementation.

from jevlab.calibration import PlattCalibrator, HistogramCalibrator, TemperatureCalibrator

fit_scores = [.99, .99, .99, .01, .01, .01]
fit_labels = [True, True, False, False, False, True]
# Use independently labeled cases for evaluation, even when scores happen to match.
holdout_scores = [.98, .97, .96, .02, .03, .04]
holdout_labels = [True, False, True, False, True, False]
model = PlattCalibrator().fit(fit_scores, fit_labels)
comparison = model.evaluate(holdout_scores, holdout_labels)
print(comparison["improvement"])  # before minus after: negative means worse

A positive improvement means lower error. ECE depends on binning and sample size; a better ECE does not guarantee a better Brier score. The report displays regressions without clipping them. Do not repeatedly pick models against the same holdout and call that an unbiased evaluation. Use a separate validation split for model selection and a final untouched test set for a release decision. See Guo et al., 2017 for background on temperature scaling.

The threshold recommendation minimizes empirical equal-cost errors on fit data, with deterministic tie breaking toward 0.5. It includes all-positive and all-negative rules (a threshold just above 1 means always negative). The report separately shows holdout accuracy at that threshold. optimal_threshold(..., false_positive_cost=..., false_negative_cost=...) supports explicit positive error costs. It does not infer your business cost model.

Bias diagnostics

A pair fixture has synthetic: true, license: "Apache-2.0", and a pairs array:

{
  "synthetic": true,
  "license": "Apache-2.0",
  "pairs": [{
    "judge": "judge-a", "a": "Four.", "b": "## Answer\n- Four.",
    "forward": 0.8, "swapped": 0.2,
    "judge_model": "model-a", "a_model": "model-a", "b_model": "model-b"
  }]
}

forward is p(first wins) for A/B; swapped is p(first wins) for B/A. The same candidate winning both orders is consistent. If both probabilities exceed 0.5, the judge picked the first position twice: the candidate winner flipped. Exact 0.5 is a tie, excluded from decisive flip rates and reported separately.

BiasAuditor().audit(pairs) returns per-judge:

• Position swap flip rate; first/second win rates; signed first-minus-second bias; mean probability inconsistency.

• Spearman rank correlation (average ranks for ties) between character-length delta and order-averaged A preference (forward + 1 − swapped) / 2.

• Correlation with markdown/list-line density delta. Density is the fraction of nonblank lines containing recognizable markdown markers.

• Self-preference using judge_model, a_model, and b_model: only pairs with exactly one own-model response are eligible. Reports mean own-response preference and correlation with own-model position.

Constant vectors, insufficient samples and absent provenance produce null, not a fabricated zero. Style correlation alone cannot establish self-preference. Match quality and content across controlled pairs; observational correlations are not proof of causal bias. The default bias demo is separate from the calibration fixture and is explicitly synthetic; custom calibration fixtures get no demo bias rows unless supplied via --pairs.

Agreement and cost frontier

from jevlab import AgreementMatrix
matrix = AgreementMatrix({"small": [True, False, True], "large": [True, False, False]})
print(matrix.pairwise())       # Cohen's kappa, including undefined (None) cells
print(matrix.fleiss_kappa())   # binary multi-rater kappa
print(matrix.pareto_frontier({"small": 0.01, "large": 0.05}, [True, False, False]))

All raters must label the same aligned items; missing ratings are rejected. Cohen’s κ = (observed agreement − chance agreement) / (1 − chance agreement). Fleiss’ κ uses each item's pairwise agreement and pooled category proportions; it is undefined with fewer than two raters or perfect chance agreement. Agreement is not accuracy.

For cost comparisons, supply judge_costs in your fixture as a map of every judge name to a finite, non-negative cost in one common unit on the same workload. A judge dominates another only if it is no more expensive and no less accurate, with at least one strict improvement. Rank 1 is the Pareto frontier; subsequent ranks remove preceding frontiers. Duplicate points share a rank. Missing costs produce no ranking; nothing is assumed to be free.

Fixture schema and existing integrations

{
  "schema_version": 1, "synthetic": true, "license": "Apache-2.0",
  "rubric": {"id": "example", "questions": [{"id": "q1", "text": "Does the answer cite evidence?"}]},
  "judge_costs": {"judge-a": 0.01},
  "items": [
    {"id": "f1", "split": "fit", "question_id": "q1", "text": "Example evidence.", "truth": true, "judgments": {"judge-a": 0.9}},
    {"id": "h1", "split": "holdout", "question_id": "q1", "text": "Another example.", "truth": false, "judgments": {"judge-a": 0.4}}
  ]
}

The two-row snippet illustrates structure, not an adequate calibration dataset. Every item must record the same judges and both splits must exist. Existing fixture scrub rules are preserved: the release harness requires synthetic fixtures under Apache-2.0 and rejects non-example email domains. A declaration alone is not a privacy scrub; inspect all data before sharing. The Python calibrator API can process separately governed local arrays without publishing them as fixtures.

Legacy commands remain available:

jevlab validate fixtures/synthetic_sample.json
jevlab template
jevlab run fixtures/synthetic_sample.json --out benchmark.json
jevlab run fixtures/synthetic_sample.json --judge-cmd 'custom=python3 adapter.py'

A CommandJudge receives {"question": ..., "text": ...} on stdin and must return {"p_yes": 0.8}. The executable inherits the launching environment; use only trusted adapters and environment variable names for provider secrets. The core harness has no provider SDK or networking dependency. MCP deliberately does not expose command judges.

Ten public rubric templates

rubrics/*.yaml are JSON-form YAML 1.2 for dependency-free, safe loading. Each has a purpose, scoring instructions and three evidence-based yes/no questions. Adapt them to your task and validate with human labels:

These are generic, newly authored templates, not production compliance rubrics.

MCP setup

After pip install ., configure a client supporting local stdio servers:

{
  "mcpServers": {
    "jevlab": { "command": "jevlab-mcp", "args": [] }
  }
}

Use the installed executable's absolute path if the client has a different PATH. Alternatively use python3 -m jevlab_oss.mcp_server from an environment where JevLab is installed. There is no HTTP endpoint or API key to configure.

Omitted data uses bundled synthetic demos. Tools accept inline data only, with a 1 MiB character message limit, 1,000 items/pairs and 16 recorded judges. Larger runs use the Python API. MCP tools are read-only and perform no command execution, path reads, report writes or network calls. Use the CLI to create HTML.

The server implements the MCP 2025-06-18 stdio transport: one JSON-RPC message per line, protocol-only stdout, initialization followed by notifications/initialized, ping, tools/list, and tools/call. Results include text plus structured JSON. Tool validation failures return isError; protocol failures use JSON-RPC error codes. Notifications produce no replies and do not execute tools.

Open-Source JevLab vs. OmniSI Cloud JEV Oracle (Proprietary Rust Engine)

No Rust source, crates, Cargo manifests, proprietary router logic or production compliance rubrics are distributed. There is no Rust dependency or build step. JevLab can evaluate outputs from any judge; it does not require the cloud service.

Development and verification

python3 scripts/audit_release.py
python3 -m unittest discover -s tests
python3 -m pip install build
python3 -m build
cd ui
npm install --ignore-scripts --package-lock=false
npm run typecheck
npx playwright install chromium
npx playwright test

CI runs on GitHub Actions with Python 3.10/3.12/3.13, tests the wheel outside the source tree, checks the standalone component, and exercises report imports, tabs and desktop/mobile rendering. CI builds artifacts only; it never deploys or publishes a package. .github/workflows/ci.yml works at this repository's root without any parent repository files.

The source fixtures/ and rubrics/ assets are mirrored into jevlab_oss/data/ for wheel installation. Update both copies together; the release audit rejects drift. See CONTRIBUTING.md, SECURITY.md, NOTICE, and the explicit export boundary in audit-manifest.json.