validation-mcp

An MCP server that lets Claude act as an independent credit-model validator, and will not save a report that contains a number the code did not compute.

Give Claude a file of model output (each borrower's predicted probability of default and whether they defaulted) and it runs the tests a bank's second line would: discrimination, calibration, population stability and variable stability. Python computes every figure. Claude does what a validator adds on top: deciding which results matter, how severe they are and what the model owner should do. Then its report is checked against the numbers before it can be written to disk.

Why this is interesting beyond credit risk

An LLM writing a validation report is useful right up to the moment it misquotes a statistic. A prompt that says "only use numbers from the tool output" asks for care; this server enforces it.

The same idea, applied to words instead of numbers, runs advice-guardrails: there an LLM judge's verdict counts only if the sentence it quotes is really in the reply.

What the check caught, for real

The example report was written by Claude (Opus 5.5, in Claude Code) from the tools' output. Its first draft said the model's PDs were "an over-prediction of 2.1 times" the observed default rate. The save was refused:

{"saved": false, "reason": "unsupported numbers",
 "unsupported": [{"number": "2.1",
   "context": "rved default rate of 20.0%, an over-prediction of 2.1 times. The overall binomial t",
   "matched_ledger_keys": []}]}

The arithmetic was right (41.2% / 20.0% = 2.07). That is the point: the figure came from the model, not the code, and a reviewer cannot tell correct model arithmetic from a plausible invention. The fix was to make the calibration tool report the ratio itself (2.07), after which the same draft passed with all 25 numbers matched.

Related MCP server: dq-mcp

The tests

Stability selection: my 2020 thesis, as a tool

My BSc thesis at TU Dublin ("Stability Selection in Logistic Regression", 2020) studied a question validators ask of every scorecard: would the same variables have been chosen from slightly different data?

The method (Meinshausen and Bühlmann, 2009): draw 100 random half-samples. On each, walk an L1-penalised logistic regression from a strong penalty to a weak one and keep the first q columns to enter. A variable's selection probability is the share of half-samples that kept it; at 0.7 or above (the threshold my thesis used) it is stable. Capping each half-sample at q is what gives an error bound: at most q² / ((2 × 0.7 − 1) × p) noise columns are expected among the stable ones.

When you name the variables the model under review uses, q defaults to their number, so the test asks: if the data picked this many variables on its own, would it reliably pick these? On the example model:

Run it

git clone https://github.com/Jayaramgokul/validation-mcp.git && cd validation-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m pytest tests/ -q              # 34 offline tests
python examples/make_example.py         # writes examples/hmeq_scored.csv

Connect it to Claude

Claude Code:

claude mcp add validation -- /full/path/to/.venv/bin/python -m valmcp.server

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "validation": {
      "command": "/full/path/to/.venv/bin/python",
      "args": ["-m", "valmcp.server"],
      "env": {"VALMCP_OUT": "/full/path/to/reports"}
    }
  }
}

Then ask, for example: "Load examples/hmeq_scored.csv (outcome BAD, score pd, split column sample with baseline dev, all twelve HMEQ variables as features). The model uses DELINQ, DEROG, CLAGE, NINQ, DEBTINC, YOJ and LOAN. Validate it and save a findings report."

Your data stays local: the server reads files from your disk over stdio and nothing is uploaded except what the tools return to Claude.

Layout

valmcp/
  metrics.py    Gini, KS, bootstrap, binomial, Hosmer-Lemeshow, Brier, PSI
  selection.py  stability selection, design matrix, error bound
  ledger.py     the ledger and the report check
  server.py     MCP server exposing the 8 tools
tests/          34 known-answer and end-to-end tests, offline
examples/       a deliberately flawed HMEQ model, and the report Claude wrote on it
data/           HMEQ (public teaching dataset, see data/README.md)

Limitations, stated plainly

• The check proves a number exists, not that it is attached to the right claim. "Gini 0.20" would pass if 0.20 is some other ledger value. Each match names its ledger key so a reviewer can see the mapping; low-precision numbers ("4", "0.1") are only weakly checked.

• Words are not checked. A report can still call a High finding Low. Severity is judgement, and stays the reviewer's job.

• The example split is random. HMEQ has no dates, so stability passes by construction; the tool says so. Use an out-of-time sample in practice.

• The error bound assumes exchangeable noise variables, which real credit data rarely satisfies. It is reported as a guide, and the tool warns when it is too loose to mean anything.

• PD models only. The tests assume a binary default outcome and a probability score.

Licence

MIT. Data: Baesens, Roesch & Scheule, Credit Risk Analytics (Wiley, 2016).