answer-check-mcp

An MCP server that checks an AI agent's draft answer against its sources before the agent replies. By default no model is called, no network is used, and the same input always gives the same result. An optional judge mode also sends each sentence to an LLM endpoint you choose.

An agent calls check_answer with its draft and the sources it was given. Each sentence comes back supported, unsupported or unchecked. An unsupported sentence names the claims no source backs and, where the source holds the right value, offers it as the fix. A sentence with nothing the check can test comes back unchecked, so prose is never reported as verified.

source:  The Pro plan costs $20 a month and includes 5 seats.
         Customers must request a refund within 30 days of purchase.

draft:   The Pro plan costs $25 a month.                  unsupported, the source says $20
         It includes 5 seats.                             supported
         Support is friendly.                             unchecked
         Customers may request a refund within 30 days.   unsupported, the source says must

Results

The check is the deterministic layer of legal-rag-verifier. On that package's held-out set of 346 sentences on UK law, sealed before it ran once, it caught 83 of 155 planted errors and rejected 10 of 191 correct sentences. When it rejected a sentence, it was right 89.3% of the time. The method and raw results are in the held-out section of the measurements.

What it does not show

The held-out set is UK law only. The check reads numbers, money, percentages, dates, durations, legal citations, the names of Acts and other instruments, qualifiers such as "the lower of", and obligation wording such as "must" against "may". A wrong statement made only in prose gets past it and comes back unchecked, so the agent or a judge model still has to read those sentences. This server's judge mode does that; see Judge mode.

Related MCP server: footnote

Use it

With Claude Code:

claude mcp add answer-check -- uvx answer-check-mcp

Any other MCP host takes the same command in its server configuration:

{
  "mcpServers": {
    "answer-check": {
      "command": "uvx",
      "args": ["answer-check-mcp"]
    }
  }
}

The server tells the agent to check any draft that states figures, dates, citations or obligations from its sources, to rewrite or remove each unsupported sentence, and never to call an unchecked sentence verified.

Tools

Each source can carry a title (the Act it comes from) and a citation (the provision), so the answer may name them. By default both tools are read-only and make no network calls. In judge mode check_answer calls the configured judge endpoint; resolve_uk_citation never does. Input is bounded: an answer of up to 20,000 characters, and up to 50 sources with 200,000 characters in total.

To turn on resolve_uk_citation, download the index as the router's README describes, then:

uvx --with legal-rag-router answer-check-mcp --uk-index data/index

Judge mode

Judge mode is off unless --judge-url is set. With it, every sentence is also judged by an LLM over an OpenAI-compatible API such as SGLang, vLLM or a hosted API, using the verifier's measured STRICT_FIGURES configuration. It makes one call per clause, one at a time, until the cap is reached. Each sentence gains checked_by, judge_label and judge_passage. The report gains a judge object with model, preset, measured, prompt_sha256, calls and failed_calls.

--judge-model and --judge-preset are mutually exclusive, and either needs --judge-url. Past the cap, the claim check still runs and each sentence says judge cap of N calls reached; not judged.

The two test sets below have 354 and 151 sentences that the judges never saw.

gemma4-e4b and qwen27-nothink are defined by this server in src/answer_check_mcp/presets.py, not by legal-rag-verifier. Their recorded labels were replayed through the library's judge and reproduced the published counts.

Any other model is unmeasured. --judge-model NAME runs a custom model with the short reply and the server's default sampling. The report says measured: false and the checked text says unmeasured: nobody knows how accurate it is.

There is a limit on answers written with the sources in hand. In the verifier's second fresh test, every judge scored below no judge: 88.9 % against 94.4 %, on 72 sentences. That was measured on qwen27-ar4; stop 56 found a similar gap on the FP8 model with another prompt; it was not measured for Gemma. The judge mostly helps sentences the model wrote from its own knowledge. In a live loop (stop 70) it still helped a 7B model with the source in its prompt: answers with a wrong sentence fell from 25 to 13 of 240. Judge mode is most useful when the agent mixes its own knowledge with the sources, and a judge can add false rejections on answers that stay close to their sources.

Cost is one call per clause. The cap bounds a check, and the p50 times are in the table above.

In judge mode the answer and the sources are sent to the judge endpoint, and the tool's open_world_hint is true. A sentence the judge marked supported is only as good as the judge's measured accuracy.

At start the server checks GET /v1/models and refuses to start if the endpoint is unreachable or does not list the model. If every judge call in a check fails, check_answer returns judge unavailable.

Example with SGLang 0.5.21 for the default preset on one L40S:

python -m sglang.launch_server \
  --model-path Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ \
  --revision 1b9d56b428e9f841d7ad566f87660b5f47553c97 \
  --reasoning-parser qwen3 \
  --context-length 65536 \
  --mem-fraction-static 0.85 \
  --port 30000

claude mcp add answer-check -- uvx answer-check-mcp --judge-url http://localhost:30000

Hosted API variant:

claude mcp add answer-check -- uvx answer-check-mcp --judge-url https://api.example.com --judge-model model-name --judge-api-key-env ANSWER_CHECK_JUDGE_API_KEY

The variable named by --judge-api-key-env must be set in the server's environment (for Claude Code, claude mcp add -e ANSWER_CHECK_JUDGE_API_KEY=...); the key never appears in output or errors.

Tests

uv sync --extra router
uv run pytest                                   # 42 pass, 4 citation tests skip
ANSWER_CHECK_UK_INDEX=data/index uv run pytest  # all 46, with the router index

The tests call the server through an MCP client, as an agent does, and one of them launches the installed command over stdio. The judge tests use a local stub server, with no model and no network beyond localhost.

Licence

AGPL-3.0-only, like the two packages it builds on. Built by Abdullah Memon at Memon Systems Ltd.