validation-mcp
by Jayaramgokul
README.md
# validation-mcp
**An MCP server that lets Claude act as an independent credit-model validator,
and will not save a report that contains a number the code did not compute.**
Give Claude a file of model output (each borrower's predicted probability of
default and whether they defaulted) and it runs the tests a bank's second
line would: discrimination, calibration, population stability and variable
stability. Python computes every figure. Claude does what a validator adds on
top: deciding which results matter, how severe they are and what the model
owner should do. Then its report is checked against the numbers before it
can be written to disk.
---
## Why this is interesting beyond credit risk
An LLM writing a validation report is useful right up to the moment it
misquotes a statistic. A prompt that says "only use numbers from the tool
output" asks for care; this server enforces it.
| Control | How it is implemented |
|---|---|
| Model never computes | Every statistic comes from a tool; tools write each value into a ledger |
| Ground every number | `save_report` finds every number in the draft and matches it to a ledger entry at the precision written ("0.78" matches 0.7781; "20.3%" matches 0.2031) |
| No override | An unmatched number blocks the save. There is no flag to force it |
| Traceable | Each match names the ledger key it matched, and the saved report carries the ledger as an appendix |
| Honest uncertainty | Tools report what their results can't show: chance failures in grade tests, a weak error bound, a random split that makes stability look fine |
The same idea, applied to words instead of numbers, runs
[advice-guardrails](https://github.com/Jayaramgokul/advice-guardrails): there an
LLM judge's verdict counts only if the sentence it quotes is really in the reply.
### What the check caught, for real
The [example report](examples/example_report.md) was written by Claude
(Opus 5.5, in Claude Code) from the tools' output. Its first draft said the
model's PDs were "an over-prediction of 2.1 times" the observed default rate.
The save was refused:
```json
{"saved": false, "reason": "unsupported numbers",
"unsupported": [{"number": "2.1",
"context": "rved default rate of 20.0%, an over-prediction of 2.1 times. The overall binomial t",
"matched_ledger_keys": []}]}
```
The arithmetic was right (41.2% / 20.0% = 2.07). That is the point: the
figure came from the model, not the code, and a reviewer cannot tell correct
model arithmetic from a plausible invention. The fix was to make the
calibration tool report the ratio itself (2.07), after which the same draft
passed with all 25 numbers matched.
## The tests
| Tool | Question | Method |
|---|---|---|
| `load_dataset` | Is the data fit to test? | Outcome is 0/1, score is a probability, default rate when each variable is missing vs present |
| `test_discrimination` | Does the score rank defaulters first? | Gini with bootstrap 95% interval, KS, development vs hold-out gap |
| `test_calibration` | Are the PDs right? | Mean PD vs observed (exact binomial), Hosmer-Lemeshow, Brier, per-grade binomial test marked *conservative* or *not conservative*, failures expected by chance |
| `test_stability` | Has the population moved? | PSI on the score and every variable, with missing values as their own bin |
| `test_variable_stability` | Would the same variables be chosen again? | Stability selection, below |
| `get_ledger` / `check_report` / `save_report` | Is every number in the report real? | The grounding check above |
### Stability selection: my 2020 thesis, as a tool
My BSc thesis at TU Dublin ("Stability Selection in Logistic Regression",
2020) studied a question validators ask of every scorecard: would the same
variables have been chosen from slightly different data?
The method (Meinshausen and Bühlmann, 2009): draw 100 random half-samples. On
each, walk an L1-penalised logistic regression from a strong penalty to a
weak one and keep the first *q* columns to enter. A variable's selection
probability is the share of half-samples that kept it; at 0.7 or above (the
threshold my thesis used) it is stable. Capping each half-sample at *q* is what
gives an error bound: at most q² / ((2 × 0.7 − 1) × p) noise columns are
expected among the stable ones.
When you name the variables the model under review uses, *q* defaults to
their number, so the test asks: **if the data picked this many variables on
its own, would it reliably pick these?** On the example model:
| Variable | Selected in | In the model? |
|---|---:|---|
| DEBTINC, DELINQ, DEROG | 100 of 100 | yes |
| VALUE | 100 of 100 | **no** |
| CLAGE | 99 of 100 | yes |
| NINQ | 41 of 100 | yes, **unstable** |
| YOJ, LOAN | 0 of 100 | yes, **unstable** |
## Run it
```bash
git clone https://github.com/Jayaramgokul/validation-mcp.git && cd validation-mcp
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python -m pytest tests/ -q # 34 offline tests
python examples/make_example.py # writes examples/hmeq_scored.csv
```
### Connect it to Claude
Claude Code:
```bash
claude mcp add validation -- /full/path/to/.venv/bin/python -m valmcp.server
```
Claude Desktop (`claude_desktop_config.json`):
```json
{
"mcpServers": {
"validation": {
"command": "/full/path/to/.venv/bin/python",
"args": ["-m", "valmcp.server"],
"env": {"VALMCP_OUT": "/full/path/to/reports"}
}
}
}
```
Then ask, for example: *"Load examples/hmeq_scored.csv (outcome BAD, score pd,
split column sample with baseline dev, all twelve HMEQ variables as features).
The model uses DELINQ, DEROG, CLAGE, NINQ, DEBTINC, YOJ and LOAN. Validate it
and save a findings report."*
Your data stays local: the server reads files from your disk over stdio and
nothing is uploaded except what the tools return to Claude.
## Layout
```
valmcp/
metrics.py Gini, KS, bootstrap, binomial, Hosmer-Lemeshow, Brier, PSI
selection.py stability selection, design matrix, error bound
ledger.py the ledger and the report check
server.py MCP server exposing the 8 tools
tests/ 34 known-answer and end-to-end tests, offline
examples/ a deliberately flawed HMEQ model, and the report Claude wrote on it
data/ HMEQ (public teaching dataset, see data/README.md)
```
## Limitations, stated plainly
- **The check proves a number exists, not that it is attached to the right
claim.** "Gini 0.20" would pass if 0.20 is some other ledger value. Each
match names its ledger key so a reviewer can see the mapping; low-precision
numbers ("4", "0.1") are only weakly checked.
- **Words are not checked.** A report can still call a High finding Low.
Severity is judgement, and stays the reviewer's job.
- **The example split is random.** HMEQ has no dates, so stability passes by
construction; the tool says so. Use an out-of-time sample in practice.
- **The error bound assumes exchangeable noise variables**, which real
credit data rarely satisfies. It is reported as a guide, and the tool
warns when it is too loose to mean anything.
- **PD models only.** The tests assume a binary default outcome and a
probability score.
## Licence
MIT. Data: Baesens, Roesch & Scheule, *Credit Risk Analytics* (Wiley, 2016).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues