English · 简体中文

Multi-Judge Consensus (MJC)

A cross-vendor model committee that reviews agent-generated content before it ships.

MJC assembles multiple LLMs from independent vendors (DeepSeek, Zhipu GLM, Alibaba Qwen, and others) into a review committee. Each judge produces a structured opinion; a purely rule-based arbiter votes, and divergent opinions trigger a bounded cross-debate. The result is an architecture-level defense against hallucination: one model's blind spot rarely overlaps with another's.

Zero third-party dependencies (Python stdlib only). Bring your own API keys. Runs fully locally.

What it does

• Structured multi-judge review — verdicts (pass / revise / reject / need_human), confidence, and itemized issues (factual_error, logical_error, hallucination, style) in machine-readable JSON.

• Rule-based arbitration — no LLM decides the final verdict. Majority vote; ties or high-confidence minority objections trigger cross-debate (≤ 2 rounds, stops on consensus).

• Deterministic verifier — date-span, percentage-base, and explicit-sum errors are checked by code, not by an LLM: zero cost, zero latency, no false positives by design.

• Cost-tiered routing — screen (1 call) → cheap pair → flagship committee. Escalation is monotonic; identical content re-reviews are free (cache).

• Administration console — a local Web UI for API keys, model catalog, tier presets, trust scores, live review streams, and per-issue disposition records.

• Adapter surface — MCP server, subprocess CLI with gate exit codes, HTTP API, and optional OpenClaw integration.

Related MCP server: design-review-mcp

How it works

content
  ├─ ⓪ verifier      deterministic checks (dates / percentages / sums) — 0 LLM cost
  ├─ ① cache         identical content → previous verdict, 0 calls
  ├─ ② screen        cheap judge (glm-4-flash); pass with conf ≥ threshold → done
  ├─ ③ committee     3 models review independently and in parallel
  ├─ ④ arbiter       pure rule vote: ≥2/3 pass → pass; disagreement → debate
  ├─ ⑤ debate ≤2 rds each judge sees the others' opinions, may change verdict
  ├─ ⑥ factcheck     non-pass only: factual issues re-checked by independent, non-complainant judges
  └─ ⑦ verdict       result + token usage + cost estimate, fully logged

Zero-hallucination design (v0.12.1)

Detect → arbitrate → repair → re-check — engineered so the system never "fixes" a fact into a different error:

• Fact arbitration (mjc/factcheck.py) — before any factual fix is attempted, the claim is independently re-checked by non-complainant, cross-vendor models (confirmed / refuted / unknown). Only confirmed replacements may touch specific facts; refuted means keep the original; unknown allows softening only.

• Falsifier pass (mjc/falsifier.py) — an adversarial red-team reviewer (cross-vendor by default) hunts for the weakest claims (premises, facts, absolutes); its challenges go through the same arbitration, and only confirmed challenges can escalate a committee pass to revise — no false-positive escalations by design.

• External knowledge (opt-in) (mjc/knowledge.py) — arbitration can pull web evidence snippets (Bing/Sogou/Baidu HTML backends, or a custom command backend; off by default, cached & rate-limited) so refutation/confirmation is grounded in retrievable sources rather than model memory alone. v0.11.0 reality check (measured 2026-09-13): the scraped backends are effectively dead — Sogou returns HTTP 403, Baidu returns its anti-bot page, Bing works but has poor recall for specific entities, and overseas endpoints are unreachable from this host. Each backend now records a health status (ok / blocked / unparsed / error) that is surfaced by evidence_check, so "we got blocked" is no longer reported as "no evidence exists"; probe with python3 -m mjc.cli knowledge-probe. For dependable evidence, wire a real search API through the cmd backend (MJC_KNOWLEDGE_CMD).

• Dual-producer repair (mjc/repair.py) — two vendors revise independently under the same rules. v0.7.2 value-level consensus: cross-model wording differs naturally, so "near-identical text" was too strict (blocked every real fix in A/B pilot C5). Agreements are now judged on numeric value sets — equal sets (both dropped the old value) or a true-subset revision (fewer claims) may be applied; text-only rewrites and gutted "no-info" revisions are never auto-applied. Disagreement keeps the original (no single-model error injection).

• Value-operation safety gate (v0.8.0) — after pilot C6 showed a both-producers-agree block can still gut a correct value when the committee false-rejects, only net value→value replacements are allowed: pure drops (losing values) and pure additions are blocked outright; replacements can additionally require blind resample support (settings repair.resample_gate, off by default — see mjc/resample.py, self-consistency line) before adoption.

• Knowledge evidence gate (v0.9.0) — when blind resampling is inconclusive (or the resample gate is off), a value replacement can still be cleared by external retrieval evidence: an opt-in gate (settings repair.evidence_gate, off by default) searches the question only (≤80 chars) and requires ≥ min_snippets snippets that contain the new value and not the old one. A resample conflict can never be overridden by evidence; retrieval errors block conservatively. Both gates off keeps the previous default behavior exactly.

• Backend-merge retrieval (v0.9.1) — evidence search now accumulates and dedupes snippets across the configured backends (merge_backends in repair.evidence_gate, on by default): stops at ≥3 unique snippets or 3 backends tried, retries a backend once on an empty result, and continues past failures — removing the single-backend flakiness reproduced in the csqa-07 demo (merge_backends: false = old path).

• Evidence independence + query hygiene (v0.11.0) — two validity bugs found by auditing the csqa-07 evidence demo, both fixed: (1) the query used to be task + proposed value, so searching "2009" trivially returned pages containing "2009" — the gate was self-certifying (support measured keyword echo, not independent corroboration). The query now contains the question only, and any old/new value literal is stripped defensively. (2) the query used to include the prompt's instruction wrapper ("请用一句话以内回答下面的问题："), which scraped back dictionary pages for the character "请" (6/6 snippets) — task_question() now strips the wrapper first. Also: producer failures are surfaced (producer_errors + note) instead of degrading to a bare "fewer than 2 revisions" message.

• Revision rules (mjc/revision.py) — the repairer must never introduce new specific facts; when in doubt, hedge or soften instead of substituting a guess (reviewer suggestions are leads, not truth).

• Gate default = full committee — deliverable gates skip the cheap screen by default (--screen to opt back in).

Lightweighting (production-oriented; experiment defaults unchanged)

Measured cost per question (2026-09-13, experiment arm C9f, api_calls): clean content 4 calls, ordinary non-pass 6-18, full chain (committee + arbitration + falsifier + two repair rounds + re-review) 32. The heavy tail sits on the non-pass path:

Every default stays as-is (batch=false, experiment arms pass no_screen=True, evidence gates are opt-in) so A/B comparisons remain valid; the table above is the recommended production recipe.

Tier presets (one-click in the admin console):

Measured results

Adversarial benchmark v1 — 21 samples (cross-document contradictions, temporal hallucinations, numerical traps, instruction deviation, plus clean controls), run with real API calls, 2026-09-06:

Full-set recall 1.0 · precision 1.0 · F1 1.0. Per-category: 5/5, 5/5, 5/5, 3/3. Layering: 1 deterministic catch by the verifier (0 LLM cost), 17 by committee + debate.

Reproduce: python3 -m mjc.cli bench --set v1-full (≈ ¥0.6 in API spend). History is appended to logs/bench-history.jsonl for regression tracking.

Code-review case study (2026-09-13)

A 518-line / 14,044-char Python module (an order-settlement engine, compiles clean) was written with 12 defects deliberately injected inside MJC's declared scope — docstring-vs-implementation mismatches (4), numeric/date errors (4), logical contradictions such as dead or unreachable branches (4) — plus 4 correct-but-suspicious decoys (an inclusive >= boundary, a documented discount-stacking order, ROUND_HALF_UP quantization, a negative-value clamp) to measure false positives. Compile/runtime errors were excluded on purpose: those belong to the test suite, not to MJC.

• Zero false positives. All 6 issues MJC raised were verified against the sealed ground truth, and none of the 4 decoys was reported.

• The single-model arm did not produce usable output. Its response hit the output-token ceiling, leaving the JSON array unterminated and unparseable; from item 11 onward it also fell into a repetition loop — roughly only 22 of the 114 recoverable objects are distinct — the remaining 92 restate an existing complaint (the most frequent one appears 13 times, so the 114 objects are 5.2x the distinct content).

• Recall by category (MJC / single model): doc-vs-impl 2/4 vs 1/4, numeric 2/4 vs 2/4, logic 2/4 vs 1/4.

• Cost per defect found: 0.049 for MJC vs ~0.053 for the single-model arm (which was unusable). 59,678 tokens in total (51,545 prompt / 8,133 completion).

• Where MJC is weak (stated plainly): the 6 misses all lack a direct textual contradiction — they require business-semantics reasoning (tax computed on the pre-discount subtotal, a refund path that can exceed the amount paid, an inconsistent coupon threshold, an inverted top-N sort, a missing "unopened" check, a delivery estimate whose comment and constant disagree). MJC is a pre-delivery proofreader, not a replacement for unit tests. The 12 defects are constructed, not a production distribution, and the sample is one module.

Getting started

Step-by-step walkthrough for first-time users (key setup, first review, web console, agent integrations, cost table, troubleshooting): QUICKSTART.md

git clone https://github.com/ElonAug7/multi-judge-consensus.git
cd multi-judge-consensus

python3 -m mjc.cli setup      # ① enter API keys interactively (skippable; stored locally, 0600)
python3 -m mjc.cli webui      # ② admin console at http://127.0.0.1:8123

# ③ review a piece of content
python3 -m mjc.cli judge-only \
  --task "Summarize tomorrow's weather in Beijing" \
  --output "Beijing will have a heavy storm tomorrow"

Requirements: Python ≥ 3.9. No pip install needed. Without any keys you can still run the verifier, the UI, and the offline test suites; with one vendor key the committee shrinks automatically (recommended: two or more vendors).

Keys are read from environment variables (MJC_DEEPSEEK_KEY, MJC_GLM_KEY, MJC_<PROVIDER>_KEY) or keys.local.json (gitignored, chmod 600). See settings.example.json for the configuration template.

CLI

Integrating with other agents

MCP (Claude Desktop/Code, Cursor, Windsurf, Cline, …):

python3 -m mjc.mcp
# Claude Code:  claude mcp add mjc -- python3 -m mjc.mcp

The review(task, output) tool returns {verdict, votes, issues, api_calls, tokens, cost_yuan}.

Subprocess CLI — one-line JSON plus exit-code gate semantics; usable from any language or CI.

HTTP — POST /api/review with optional token auth (Dify/Coze/n8n custom tools, cross-machine).

OpenClaw — optional native integration: transcript scanning, coding stage gates, live review stream.

Architecture

mjc/
├── providers.py     vendor registry (deepseek/glm/qwen/dashscope/doubao/kimi; add keys to enable)
├── judge.py         single reviewer: structured JSON, memory injection, same-vendor fallback
├── arbiter.py       vote/debate arbitration; per-opinion live events
├── pipeline.py      verifier → screen → committee; fault-tolerant degradation
├── verifier.py      deterministic checks (dates/percentages/sums), zero false positives by design
├── bench.py         adversarial benchmark runner (recall/precision/F1 + history)
├── webui.py         admin console (review / live tasks / settings)
├── mcp_server.py    MCP stdio server
└── settings.py      runtime config (provider registry, model catalog, tier presets)
tests/               18 offline suites (0 API calls; key-dependent cases skip gracefully)
samples/bench-v1.json   adversarial benchmark corpus

Development

python3 tests/test_phase3.py     # pipeline/cache/degradation
python3 tests/test_verifier.py   # deterministic verifier
python3 tests/test_mcp.py        # MCP protocol
# …18 suites total, all offline. GitHub Actions runs them on Python 3.9/3.11/3.12.

Security & notes

• Keys live only in environment variables or a local keys.local.json (0600, gitignored).

• Runtime config, logs, and review records stay local and are never committed.

• Cost figures are estimates from per-vendor price tables (¥/1K tokens), marked as approximations; token counts come from API usage fields.

• The benchmark measures worst-case interception on a constructed corpus, not a natural production distribution.

License

GPL-3.0. Local invocation only; keys are the user's own.