# psychometrics-mcp

Classical test theory, differential item functioning (DIF), local item
dependence and standard setting, exposed both as a Python library and as an MCP
server. Built to replace the ad hoc scripts and spreadsheets used in
certification exam review, where the caller is as often an agent as a person.

The library is the product; the MCP server is one interface onto it.

## Install

```sh
uv sync --frozen                   # the environment the tests were verified in
pip install -e '.[server,test]'    # or resolve fresh: server + tests
pip install -e .                   # library only: numpy and scipy
```

Python 3.10+. The core depends on numpy and scipy only, so it installs in a
locked-down analysis environment; the `mcp` dependency is an extra. `uv.lock`
is committed and pins all 46 packages with hashes; the version specifiers in
`pyproject.toml` stay floors, so a consumer installing from an index gets a
normal resolution rather than this machine's.

## Use as a library

Every example below runs against a file in `fixtures/`, which is committed, and
the values shown are the ones actually returned -- only the very small p-value
exponents are abbreviated here for width.

```python
from psychometrics import load_response_matrix, compute_classical_stats

matrix, diag = load_response_matrix("fixtures/flawed_form.csv")
stats = compute_classical_stats(matrix, diagnostics=diag)
```

`load_response_matrix` returns a `(ResponseMatrix, Diagnostics)` pair, and
every analysis function takes and returns that `Diagnostics` so caveats
travel with the numbers instead of being reconstructed later.

`matrix.summary()` describes a dataset without echoing a cell of it:

```python
{'n_examinees': 400, 'n_items': 12,
 'missing_policy': 'omit_incorrect_notreached_missing',
 'n_omitted_responses': 0, 'n_not_reached_responses': 0,
 'pct_cells_blank': 0.0, 'mean_total_score': 6.1375,
 'max_not_reached_rate_any_item': 0.0}
```

`flawed_form.csv` has a mis-keyed first item. Its p-value is unremarkable, so
only the corrected point-biserial exposes it -- which is why that is the value
flags are computed from:

```python
>>> stats["items"][0]
{'item_id': 'item_01', 'n_valid': 400, 'n_missing': 0,
 'p_value': 0.4825, 'p_value_ci95': [0.433931, 0.531402],
 'point_biserial': -0.438899,              # item-rest, corrected
 'point_biserial_uncorrected': -0.229897,  # item-total, inflated by construction
 'biserial': -0.550271, 'discrimination_index_d': -0.546296,
 'flags': [{'code': 'negative_discrimination',
            'severity': 'investigate',
            'reason': 'Higher-scoring examinees did WORSE on this item than '
                      'lower-scoring ones. At ordinary difficulty this is the '
                      'canonical signature of a mis-keyed item or of two '
                      'defensible answers, not of a merely weak item.',
            'observed': -0.438899, 'threshold': 0.0,
            'next_step': 'Verify the answer key before interpreting any other '
                         'statistic for this item. Do not recommend deletion '
                         'until the key has been checked -- a mis-key is a '
                         'repair, not a retirement.'}]}
```

A flag is an object, not a string: it carries the threshold it crossed and a
recommended next step, so a caller reporting it does not have to invent one.

### DIF

```python
from psychometrics import load_response_matrix
from psychometrics.dif import flag_dif

matrix, diag = load_response_matrix("fixtures/dif_form.csv", group_column="group")
result = flag_dif(matrix, diagnostics=diag)
comparison = result["comparisons"][0]      # focal vs reference, 600 each
```

`item_02` carries planted uniform DIF of +0.8 on the *b* scale:

```python
>>> comparison["items"][1]["mantel_haenszel"]
{'n_strata_total': 19, 'n_strata_informative': 18,
 'n_reference': 600.0, 'n_focal': 600.0,
 'odds_ratio': 2.085446, 'mh_d_dif': -1.72721, 'se_mh_d_dif': 0.328571,
 'chi_square': 27.424746, 'p_value': 1.633e-07, 'estimable': True,
 'ets': {'category': 'C',
         'interpretation': 'large DIF; content review required before operational use',
         'direction': 'favours reference group (harder for focal)',
         'significant': True, 'significantly_beyond_1.0': True},
 'p_value_bh_adjusted': 3.267e-06}
```

The Benjamini-Hochberg adjusted p-value is reported beside the raw one without
being asked for, because screening a form is one test per item. The summary
block says what the counts do and do not mean:

```python
>>> comparison["summary"]["ets_category_counts"]
{'A': 18, 'B': 1, 'C': 1, 'not_estimable': 0}
>>> comparison["summary"]["items_flagged_c"]
['item_02']
```

The focal group in this file also has a lower mean proficiency, so raw p-values
differ on *every* item. That is impact, not DIF, and no clean item is flagged
for it.

### Local item dependence

Classical statistics look at one item at a time, so a pair of items measuring the
same thing twice is invisible to them. `dependent_pairs_form.csv` is the case:
every item passes every per-item check, alpha reads a healthy 0.837, and six of
its 190 pairs are entangled.

```python
from psychometrics import load_response_matrix, compute_local_dependence

matrix, diag = load_response_matrix("fixtures/dependent_pairs_form.csv")
result = compute_local_dependence(matrix, diagnostics=diag)
```

```python
>>> result["flagged_pairs"][0]
{'item_a': 'item_02', 'item_b': 'item_03',
 'q3': 0.500011,          # Yen's Q3: correlation of the two items' residuals
 'q3_star': 0.552522,     # Q3 minus this form's mean Q3 -- what is compared
 'sd_above_form_mean': 6.431, 'n_examinees_both_items': 600,
 'flags': [{'code': 'positive_local_dependence',
            'severity': 'investigate',
            'reason': 'These two items share variance that the rest of the '
                      'form does not explain: ...',
            'observed': 0.552522, 'threshold': 0.2,
            'next_step': 'Read the two items side by side and decide whether '
                         'one gives away the other, or whether they are the '
                         'same question twice. ...'}]}
```

The whole Q3 distribution comes back beside the flags, because a flagged pair
cannot be read without it -- the same Q3 of 0.25 is unremarkable on a form
averaging 0.10 and glaring on one averaging −0.07:

```python
>>> result["distribution"]
{'n_pairs': 190, 'n_estimable': 190, 'n_not_estimable': 0,
 'q3_mean': -0.052511, 'q3_sd': 0.085916,
 'q3_min': -0.411887, 'q3_max': 0.500011,
 'q3_mean_expected_bias': -0.052632}
```

Q3 is biased downward by about −1/(*k*−1) purely by construction, and that
observed mean sits on the theoretical value to four decimals. Comparing against
the form's own mean rather than a fixed 0.20 is what makes one threshold
transport across form lengths; `threshold_reference="absolute"` is available for
reconciling with other software, and on a 12-item form it invents five pairs on a
file with no dependence planted in it at all.

```python
>>> result["summary"]["items_in_flagged_pairs_count"]
{'item_02': 1, 'item_03': 1, 'item_05': 1, 'item_06': 1, 'item_09': 1,
 'item_10': 1, 'item_16': 3, 'item_17': 1, 'item_18': 1, 'item_19': 1}
>>> result["summary"]["hub_items"]
['item_16']
```

`item_16` cues three other items, and it is reported separately from the pair
list because a hub is one item to fix rather than three pairs to triage.

There are deliberately no p-values here. Residualising on a criterion estimated
from the same items means Q3's null is not a raw correlation's, and formal
inference needs a parametric bootstrap over an IRT calibration this package does
not perform. This is an effect-size screen that routes a pair to human content
review; it is not an enemy-item check, and which of the two items is the
redundant one is not in the number.

### Standard setting

Panel ratings are aggregate expert judgements rather than candidate records, so
these may be passed inline:

```python
import numpy as np
from psychometrics import run_standard_setting

ratings = np.loadtxt("fixtures/panel_ratings.csv", delimiter=",")  # 8 x 15
result = run_standard_setting(ratings.tolist(), method="angoff", sem=1.4)
```

```python
>>> result["cut_score"]
{'raw': 9.506025, 'percent': 63.3735, 'median_based': 9.36905,
 'sd_across_panelists': 0.297662, 'standard_error': 0.105239,
 'ci95': [9.299759, 9.712291]}
>>> result["agreement"]["icc_single_rater"], result["agreement"]["icc_panel_mean"]
(0.808053, 0.971163)
>>> sorted(result["adjustments"])
['minus_1_sem', 'plus_1_sem', 'rounded_down', 'rounded_nearest',
 'rounded_up', 'unrounded']
```

Every rounding and SEM option is returned together, each with its implied pass
rate when `score_distribution` is supplied, because picking whichever one
produces the pass rate you wanted is the failure this is meant to make visible.

The diagnostics also name the optional arguments that were *not* supplied and
what each omission cost -- no reality check, no chance-level check, no implied
pass rate -- so a payload never reads the same whether a check passed or never
ran.

## Use as an MCP server

```sh
psychometrics-mcp        # stdio, local only
```

Register it with any MCP client, e.g.:

```json
{
  "mcpServers": {
    "psychometrics": { "command": "psychometrics-mcp" }
  }
}
```

Five tools:

| Tool | Purpose |
| --- | --- |
| `describe_dataset` | Shape, blank breakdown and subgroup sizes. Call first on an unfamiliar file. |
| `compute_classical_stats` | Per-item difficulty and discrimination, per-form reliability and SEM. |
| `compute_local_dependence` | Yen's Q3 across every item pair, flagging pairs that share variance the form does not explain. |
| `flag_dif` | Mantel-Haenszel, standardised p-difference and logistic DIF across every item. |
| `run_standard_setting` | Angoff, yes/no Angoff and Bookmark aggregation with the cut's standard error. |

## What is deliberately unusual

**Refusal is a result.** Below a hard sample-size floor these functions raise
`InsufficientData` rather than return an imprecise number; below an advisory
floor they return the number together with a warning. An agent will format
whatever number it is given into a report, and a person can fail a candidate
on that report, so a number too weak to support the decision is worse than no
number at all. Both `InsufficientData` and `Diagnostics` carry `to_dict()`,
and the server turns a refusal into a structured payload rather than an
exception string.

**Every threshold is labelled.** Each constant in `guardrails.py` is marked
either `FIELD CONVENTION` (a published rule of thumb, cited) or `MODELLING
ASSUMPTION` (ours, and overridable through the `overrides` dict). A threshold
with no provenance comment is a defect.

**Corrected discrimination drives decisions.** Both item-rest (corrected) and
item-total (uncorrected) point-biserials are reported; flags use the corrected
one, because the uncorrected value is spuriously inflated on short forms.

**Multiple comparisons are corrected by default.** Screening a 65-item form
for DIF is 65 simultaneous tests, so Benjamini-Hochberg adjusted p-values are
reported alongside the raw ones and the diagnostics say which to use.

**Missing data is three things.** Omitted (blank mid-form, scored incorrect)
is distinguished from not-reached (a trailing run of blanks, excluded, because
scoring it wrong confounds difficulty with speededness), and raw-versus-scored
input is never guessed at silently.

**Variance is `ddof=1` everywhere.** That is what makes Cronbach's alpha and
KR-20 algebraically identical for dichotomous items, an identity the test
suite asserts.

**A DIF flag is not a finding of bias.** It means an item behaves differently
for two groups of equal overall proficiency, which routes that item to human
content review. Flag counts are not counts of biased items.

**A dependent pair is not an enemy item.** Q3 supplies the statistical half of
that question only; whether two items overlap in content, and which of the two is
the redundant one, is not in the number. Pairs are also flagged for reasons that
have nothing to do with content, such as a shared stimulus or adjacent positions
on a speeded form.

**Three refusal channels, not one, because the remedies differ.**
`insufficient_data` means collect more data; `invalid_input` means the same data,
corrected, would work, and the message says what to correct; `no_estimable_pairs`
means neither would help and the form itself is the problem. A bad value refused
through the wrong channel sends an agent off to recruit more panelists over what
is a units error.

## Privacy posture

No network calls, no telemetry. Tools that touch candidate data take file
paths, never inline matrices, so a multi-thousand-examinee response file never
enters a conversation transcript; `ResponseMatrix.summary()` exists so a
dataset can be described without echoing examinee-level data. Panel ratings
are the exception and may be passed inline, being aggregate expert judgements
rather than candidate records.

## Logging

Logging goes to stderr, and that is a transport requirement rather than a
preference: the server speaks JSON-RPC over stdio, so stdout *is* the protocol
stream and a log line written there is a parse error at the client, not noise.
The only handler here that emits anything names `sys.stderr` explicitly.

Imported as a library the package attaches nothing that emits -- only a
`NullHandler` -- so no output appears unless the host application configures
logging itself. Run as a server, `main()` adds that one stderr handler, at
WARNING by default, which emits refusals and diagnostic warnings:

```sh
PSYCHOMETRICS_LOG_LEVEL=INFO psychometrics-mcp     # adds the notes
PSYCHOMETRICS_LOG_LEVEL=CRITICAL psychometrics-mcp # effectively silent
```

An unrecognised value falls back to WARNING and says so rather than refusing to
start. Log records carry paths, shapes, counts, thresholds and diagnostic
prose -- the same short list that is safe to put in a transcript, and for the
same reason. A log file outlives the process and may sit under weaker controls
than the response file did, so no cell of examinee data is ever written to one.

## Tests

```sh
pytest        # 236 tests
```

`fixtures/` holds the synthetic corpus they and the examples above run against,
regenerated by `fixtures/generate.py` from a fixed seed. `fixtures/README.md`
records what is planted in each file and what it should produce, so a change in
those numbers is a change in behaviour that needs explaining.

## Not in this version

No IRT calibration, so `bookmark` requires `ordered_item_thetas` from an
external calibration and will not substitute classical difficulty for them.

## License

MIT