<p align="center">
  <img src="assets/logo-wide.svg" alt="Multi-Judge Consensus" width="560"/>
</p>

[English](README.md) · [简体中文](README.zh.md)

# Multi-Judge Consensus (MJC)

**A cross-vendor model committee that reviews agent-generated content before it ships.**

MJC assembles multiple LLMs from independent vendors (DeepSeek, Zhipu GLM, Alibaba Qwen, and others) into a
review committee. Each judge produces a structured opinion; a purely rule-based arbiter votes, and divergent
opinions trigger a bounded cross-debate. The result is an architecture-level defense against hallucination:
one model's blind spot rarely overlaps with another's.

Zero third-party dependencies (Python stdlib only). Bring your own API keys. Runs fully locally.

---

## What it does

- **Structured multi-judge review** — verdicts (`pass` / `revise` / `reject` / `need_human`), confidence, and
  itemized issues (`factual_error`, `logical_error`, `hallucination`, `style`) in machine-readable JSON.
- **Rule-based arbitration** — no LLM decides the final verdict. Majority vote; ties or high-confidence
  minority objections trigger cross-debate (≤ 2 rounds, stops on consensus).
- **Deterministic verifier** — date-span, percentage-base, and explicit-sum errors are checked by code,
  not by an LLM: zero cost, zero latency, no false positives by design.
- **Cost-tiered routing** — screen (1 call) → cheap pair → flagship committee. Escalation is monotonic;
  identical content re-reviews are free (cache).
- **Administration console** — a local Web UI for API keys, model catalog, tier presets, trust scores,
  live review streams, and per-issue disposition records.
- **Adapter surface** — MCP server, subprocess CLI with gate exit codes, HTTP API, and optional OpenClaw
  integration.

## How it works

```
content
  ├─ ⓪ verifier      deterministic checks (dates / percentages / sums) — 0 LLM cost
  ├─ ① cache         identical content → previous verdict, 0 calls
  ├─ ② screen        cheap judge (glm-4-flash); pass with conf ≥ threshold → done
  ├─ ③ committee     3 models review independently and in parallel
  ├─ ④ arbiter       pure rule vote: ≥2/3 pass → pass; disagreement → debate
  ├─ ⑤ debate ≤2 rds each judge sees the others' opinions, may change verdict
  ├─ ⑥ factcheck     non-pass only: factual issues re-checked by independent, non-complainant judges
  └─ ⑦ verdict       result + token usage + cost estimate, fully logged
```

### Zero-hallucination design (v0.12.1)

Detect → arbitrate → repair → re-check — engineered so the system never "fixes" a fact into a different error:

- **Fact arbitration** (`mjc/factcheck.py`) — before any factual fix is attempted, the claim is independently
  re-checked by non-complainant, cross-vendor models (`confirmed` / `refuted` / `unknown`). Only `confirmed`
  replacements may touch specific facts; `refuted` means keep the original; `unknown` allows softening only.
- **Falsifier pass** (`mjc/falsifier.py`) — an adversarial red-team reviewer (cross-vendor by default) hunts for
  the weakest claims (premises, facts, absolutes); its challenges go through the same arbitration, and only
  confirmed challenges can escalate a committee `pass` to `revise` — no false-positive escalations by design.
- **External knowledge (opt-in)** (`mjc/knowledge.py`) — arbitration can pull web evidence snippets
  (Bing/Sogou/Baidu HTML backends, or a custom command backend; off by default, cached & rate-limited) so
  refutation/confirmation is grounded in retrievable sources rather than model memory alone.
  **v0.11.0 reality check (measured 2026-09-13):** the scraped backends are effectively dead — Sogou
  returns HTTP 403, Baidu returns its anti-bot page, Bing works but has poor recall for specific entities,
  and overseas endpoints are unreachable from this host. Each backend now records a health status
  (`ok` / `blocked` / `unparsed` / `error`) that is surfaced by `evidence_check`, so "we got blocked" is no
  longer reported as "no evidence exists"; probe with `python3 -m mjc.cli knowledge-probe`. For dependable
  evidence, wire a real search API through the `cmd` backend (`MJC_KNOWLEDGE_CMD`).
- **Dual-producer repair** (`mjc/repair.py`) — two vendors revise independently under the same rules.
  v0.7.2 value-level consensus: cross-model wording differs naturally, so "near-identical text" was too strict
  (blocked every real fix in A/B pilot C5). Agreements are now judged on **numeric value sets** — equal sets
  (both dropped the old value) or a true-subset revision (fewer claims) may be applied; text-only rewrites and
  gutted "no-info" revisions are never auto-applied. Disagreement keeps the original (no single-model error injection).
- **Value-operation safety gate (v0.8.0)** — after pilot C6 showed a both-producers-agree block
  can still gut a *correct* value when the committee false-rejects, only net **value→value replacements**
  are allowed: pure drops (losing values) and pure additions are blocked outright; replacements can
  additionally require **blind resample support** (settings `repair.resample_gate`, off by default —
  see `mjc/resample.py`, self-consistency line) before adoption.
- **Knowledge evidence gate (v0.9.0)** — when blind resampling is inconclusive (or the resample gate is
  off), a value replacement can still be cleared by **external retrieval evidence**: an opt-in gate
  (settings `repair.evidence_gate`, off by default) searches **the question only** (≤80 chars) and
  requires ≥ `min_snippets` snippets that contain the new value and **not** the old one. A resample
  `conflict` can never be overridden by evidence; retrieval errors block conservatively. Both gates
  off keeps the previous default behavior exactly.
- **Backend-merge retrieval (v0.9.1)** — evidence search now accumulates and dedupes snippets across the
  configured backends (`merge_backends` in `repair.evidence_gate`, **on by default**): stops at ≥3 unique
  snippets or 3 backends tried, retries a backend once on an empty result, and continues past failures —
  removing the single-backend flakiness reproduced in the csqa-07 demo (`merge_backends: false` = old path).
- **Evidence independence + query hygiene (v0.11.0)** — two validity bugs found by auditing the csqa-07
  evidence demo, both fixed:
  (1) the query used to be `task + proposed value`, so searching "2009" trivially returned pages containing
  "2009" — the gate was **self-certifying** (`support` measured keyword echo, not independent corroboration).
  The query now contains the question only, and any old/new value literal is stripped defensively.
  (2) the query used to include the prompt's instruction wrapper ("请用一句话以内回答下面的问题："), which
  scraped back dictionary pages for the character "请" (6/6 snippets) — `task_question()` now strips the
  wrapper first. Also: producer failures are surfaced (`producer_errors` + note) instead of degrading to a
  bare "fewer than 2 revisions" message.
- **Revision rules** (`mjc/revision.py`) — the repairer must never introduce new specific facts; when in doubt,
  hedge or soften instead of substituting a guess (reviewer suggestions are leads, not truth).
- **Gate default = full committee** — deliverable gates skip the cheap screen by default (`--screen` to opt back in).

### Lightweighting (production-oriented; experiment defaults unchanged)

Measured cost per question (2026-09-13, experiment arm C9f, `api_calls`): clean content **4 calls**,
ordinary non-pass **6-18**, full chain (committee + arbitration + falsifier + two repair rounds +
re-review) **32**. The heavy tail sits on the non-pass path:

| Lever | Setting | Measured / expected | Trade-off |
|---|---|---|---|
| **Browser retrieval backend** | `knowledge.backends=["browser"]` | evidence wait **300s -> 2.5s** (no anti-bot, no cooldown) | needs Chrome on the host |
| **Batch arbitration** | `factcheck.batch=true` | arbitration block **issues x arbiters -> 2** calls (8 -> 2 at 4 issues) | one call must judge several issues; slightly coarser |
| **Screening** | `screen_enabled=true` + `screen_model` | clean content passes in **1 call** (skips the 3-seat committee) | miss risk bounded by `screen_conf` |
| **Economy tier** | `current.tier="eco"` | 3 models -> 2 models | detection rate needs re-testing |
| **Single repair round** | caller passes `--max-rev 1` | saves a whole round (committee + falsifier + arbitration) | one less chance to converge |

Every default stays as-is (`batch=false`, experiment arms pass `no_screen=True`, evidence gates are
opt-in) so A/B comparisons remain valid; the table above is the recommended **production** recipe.

**Tier presets** (one-click in the admin console):

| Tier | Committee | Strategy |
|---|---|---|
| Economy | 2x budget models | cheapest, screen-first |
| Standard | 2 budget + 1 flagship | default (benchmark-verified) |
| Strict | flagship only, no screen | maximum rigor |

## Measured results

Adversarial benchmark v1 — 21 samples (cross-document contradictions, temporal hallucinations, numerical
traps, instruction deviation, plus clean controls), run with real API calls, 2026-09-06:

| Pipeline | Defect pass-through | False kills on clean content |
|---|---|---|
| No review (shipped as-is) | 100% (18/18) | — |
| Single-model self-check (glm-4-plus) | 11.1% (2/18 missed) | — |
| **MJC full pipeline** | **0% (18/18 caught)** | **0/3** |

Full-set recall 1.0 · precision 1.0 · F1 1.0. Per-category: 5/5, 5/5, 5/5, 3/3.
Layering: 1 deterministic catch by the verifier (0 LLM cost), 17 by committee + debate.

Reproduce: `python3 -m mjc.cli bench --set v1-full` (≈ ¥0.6 in API spend).
History is appended to `logs/bench-history.jsonl` for regression tracking.

### Code-review case study (2026-09-13)

A **518-line / 14,044-char** Python module (an order-settlement engine, compiles clean) was written with
**12 defects deliberately injected inside MJC's declared scope** — docstring-vs-implementation mismatches (4),
numeric/date errors (4), logical contradictions such as dead or unreachable branches (4) — plus **4
correct-but-suspicious decoys** (an inclusive `>=` boundary, a documented discount-stacking order,
`ROUND_HALF_UP` quantization, a negative-value clamp) to measure false positives. Compile/runtime errors were
excluded on purpose: those belong to the test suite, not to MJC.

| Arm | API calls | Latency | Cost | Defects found | False positives | Output usable |
|---|---|---|---|---|---|---|
| A Deterministic verifier (no API) | 0 | 0.0s | 0 | 0/12 | 0 | yes |
| B Single-model self-check (glm-4-plus) | 1 | 88s | ~0.21 | 4/12 | n/a | **no** |
| **C MJC full pipeline** | **7** | **346s** | **0.29192** | **6/12** | **0** | **yes** |

- **Zero false positives.** All 6 issues MJC raised were verified against the sealed ground truth, and none of
  the 4 decoys was reported.
- **The single-model arm did not produce usable output.** Its response hit the output-token ceiling, leaving the
  JSON array unterminated and unparseable; from item 11 onward it also fell into a repetition loop — roughly
  only 22 of the 114 recoverable objects are distinct — the remaining 92 restate an existing
  complaint (the most frequent one appears 13 times, so the 114 objects are 5.2x the distinct content).
- **Recall by category** (MJC / single model): doc-vs-impl 2/4 vs 1/4, numeric 2/4 vs 2/4, logic 2/4 vs 1/4.
- **Cost per defect found**: 0.049 for MJC vs ~0.053 for the single-model arm (which was unusable).
  59,678 tokens in total (51,545 prompt / 8,133 completion).
- **Where MJC is weak (stated plainly):** the 6 misses all lack a direct textual contradiction — they require
  business-semantics reasoning (tax computed on the pre-discount subtotal, a refund path that can exceed the
  amount paid, an inconsistent coupon threshold, an inverted top-N sort, a missing "unopened" check, a delivery
  estimate whose comment and constant disagree). MJC is a **pre-delivery proofreader**, not a replacement for
  unit tests. The 12 defects are constructed, not a production distribution, and the sample is one module.

## Getting started

> Step-by-step walkthrough for first-time users (key setup, first review, web console,
> agent integrations, cost table, troubleshooting): **[QUICKSTART.md](QUICKSTART.md)**

```bash
git clone https://github.com/ElonAug7/multi-judge-consensus.git
cd multi-judge-consensus

python3 -m mjc.cli setup      # ① enter API keys interactively (skippable; stored locally, 0600)
python3 -m mjc.cli webui      # ② admin console at http://127.0.0.1:8123

# ③ review a piece of content
python3 -m mjc.cli judge-only \
  --task "Summarize tomorrow's weather in Beijing" \
  --output "Beijing will have a heavy storm tomorrow"
```

Requirements: Python ≥ 3.9. No `pip install` needed. Without any keys you can still run the verifier,
the UI, and the offline test suites; with one vendor key the committee shrinks automatically
(recommended: two or more vendors).

Keys are read from environment variables (`MJC_DEEPSEEK_KEY`, `MJC_GLM_KEY`, `MJC_<PROVIDER>_KEY`) or
`keys.local.json` (gitignored, chmod 600). See `settings.example.json` for the configuration template.

## CLI

| Command | Purpose |
|---|---|
| `setup` | interactive first-run key configuration + connectivity probes |
| `doctor` | health check: keys, tier, trust scores, cache, cumulative usage |
| `judge-only --task --output` | review given text with the committee |
| `review --task` | generate → review → rewrite loop (≤ 3 rejections) |
| `auto --task --content [--kind code]` | single-shot review with memory context, JSON output |
| `gate --stage design\|code\|deliver --task --content` | stage gate; exit codes pass=0 / revise=2 / reject=3 |
| `bench [--set quick\|v1-full] [--ablation 0,1,2]` | red-team benchmark with history |
| `dispose --in <json>` | record per-issue dispositions (adopted / rejected with reason) |
| `savings [--days N] [--json]` | savings ledger: tokens / cost / wall-clock avoided by each mechanism |
| `knowledge-probe` | probe every retrieval backend and report health (ok / blocked / unparsed / error) |
| `switch on\|off\|status` | pause/resume the auto-review hook (0 cost, fully silent; manual commands unaffected) |
| `webui` | local admin console (127.0.0.1:8123) |
| `mcp` | MCP stdio server |

## Integrating with other agents

**MCP** (Claude Desktop/Code, Cursor, Windsurf, Cline, …):

```bash
python3 -m mjc.mcp
# Claude Code:  claude mcp add mjc -- python3 -m mjc.mcp
```

The `review(task, output)` tool returns `{verdict, votes, issues, api_calls, tokens, cost_yuan}`.

**Subprocess CLI** — one-line JSON plus exit-code gate semantics; usable from any language or CI.

**HTTP** — `POST /api/review` with optional token auth (Dify/Coze/n8n custom tools, cross-machine).

**OpenClaw** — optional native integration: transcript scanning, coding stage gates, live review stream.

## Architecture

```
mjc/
├── providers.py     vendor registry (deepseek/glm/qwen/dashscope/doubao/kimi; add keys to enable)
├── judge.py         single reviewer: structured JSON, memory injection, same-vendor fallback
├── arbiter.py       vote/debate arbitration; per-opinion live events
├── pipeline.py      verifier → screen → committee; fault-tolerant degradation
├── verifier.py      deterministic checks (dates/percentages/sums), zero false positives by design
├── bench.py         adversarial benchmark runner (recall/precision/F1 + history)
├── webui.py         admin console (review / live tasks / settings)
├── mcp_server.py    MCP stdio server
└── settings.py      runtime config (provider registry, model catalog, tier presets)
tests/               18 offline suites (0 API calls; key-dependent cases skip gracefully)
samples/bench-v1.json   adversarial benchmark corpus
```

## Development

```bash
python3 tests/test_phase3.py     # pipeline/cache/degradation
python3 tests/test_verifier.py   # deterministic verifier
python3 tests/test_mcp.py        # MCP protocol
# …18 suites total, all offline. GitHub Actions runs them on Python 3.9/3.11/3.12.
```

## Security & notes

- Keys live only in environment variables or a local `keys.local.json` (0600, gitignored).
- Runtime config, logs, and review records stay local and are never committed.
- Cost figures are estimates from per-vendor price tables (¥/1K tokens), marked as approximations;
  token counts come from API usage fields.
- The benchmark measures worst-case interception on a constructed corpus, not a natural production
  distribution.

## License

GPL-3.0. Local invocation only; keys are the user's own.