# healthsec-mcp

MCP connector for clinical-AI security evaluation: adversarial robustness,
privacy leakage, and standards-compliance tools composed into a Security
Posture Score, callable directly by AI agents.

Full paper structure, research contribution, and milestones:
[`../STRUCTURE.md`](../STRUCTURE.md). Practical tool-by-tool usage
reference: [`docs/TOOLS.md`](docs/TOOLS.md). Illustrative worked example:
[`examples/end_to_end_security_evaluation.md`](examples/end_to_end_security_evaluation.md).
Two ways to run this: locally via Claude Desktop (below), or via the
agent-usefulness study harness (`reproduce/agent_usefulness_study/`).

**Status:** M1-M5 implemented, with two exceptions noted below. All 10
tools (the 9 planned + `get_audit_log`, added to close a gap found during
M4) are registered on the server, lint-clean (`ruff`), type-clean
(`mypy`), and covered by CI (`.github/workflows/ci.yml`). One documented
open limitation remains: boundary attack's `flip_rate`/`auroc_drop` don't
reproduce the reference numbers even though `auroc_clean` matches exactly
(see TECHNICAL_DESIGN.md section 12; golden test marked `xfail(strict=True)`).
Everything else that has a reference target is golden-verified -- exactly
for the deterministic tools (M3 standards, M4 `compute_sps`), within
tolerance for the ML-based ones (M1 `run_fgsm`). `run_membership_inference`
(M2) has no published golden target (see TECHNICAL_DESIGN.md section 4.3)
so it's covered by property tests only. The **agent-usefulness study has
been run once (6 tasks, real transcripts), but its rating data does not
match the protocol's design** — see
`reproduce/agent_usefulness_study/PROTOCOL.md`'s "Actual execution status"
section for the full, honest account: the "2 raters" were, in turn, one
person filling both columns, then one person plus ChatGPT as an
LLM-judge cross-check. Neither is two independent human raters. A real
second human rater still needs to score `transcripts/blinded/` from
scratch before this data can support the paper's agent-usefulness claim.
Everything else in M5 (`docs/TOOLS.md`, CI, MCP contract tests,
`examples/`) is done. See Milestones in STRUCTURE.md.

## Layout

```
healthsec-mcp/
├── src/healthsec_mcp/
│   ├── adversarial/       # fgsm.py, boundary.py, plausibility.py -- implemented (M1)
│   ├── privacy/           # membership_inference.py -- implemented (M2)
│   ├── standards/         # attack_coverage.py, rbac.py, audit.py, compliance.py -- implemented (M3)
│   ├── tools/             # adversarial_tools.py, privacy_tools.py, standards_tools.py, sps_tools.py
│   ├── io/                # schemas.py -- FeatureBatch (n<=100), FeaturePool (n<=5000)
│   ├── registry.py        # in-session model-handle registry: handle -> model object map only
│   ├── authz.py           # authorization gate (M4) -- wraps registry.resolve(), records every
│   │                      # attempt (success or denial) to audit.py, regardless of outcome
│   ├── audit.py           # append-only log of this connector's own tool calls (M4)
│   ├── sps.py             # Security Posture Score composer (M4) -- weights are a parameter,
│   │                      # not hardcoded, per the open design question this resolved
│   ├── report.py          # generate_security_report's logic (M5) -- never states a deployment
│   │                      # recommendation unless compute_sps's output was actually supplied
│   ├── server.py          # FastMCP server, registers all 10 implemented tools
│   └── local_datasets.py  # shared loader/registrar for the 4 local models -- the one place
│                          # CONNECTOR_DATA_ROOT is resolved, used by study_server.py and
│                          # scripts/run_local_server.py so path logic isn't duplicated per script
├── tests/
│   ├── unit/              # per-module unit tests (synthetic models + pure-function cases, fast) --
│   │                      # includes test_sps.py (exact SPS=78.9 golden match), test_authz.py/
│   │                      # test_audit.py (gate + audit-trail behavior), test_report.py
│   ├── golden/            # regression tests against reference/ (validated result tables) --
│   │                      # standards tools + compute_sps match exactly; ML-attack tools match
│   │                      # within tolerance (boundary attack currently xfails, see Status)
│   ├── contract/          # MCP tool-schema contract tests (M5) -- every tool has a real
│   │                      # description + valid JSON Schema; spot-checks required params
│   └── fault_injection/   # bad models/inputs, cap enforcement (FGSM n≤100, MI pool≤5k)
├── reproduce/
│   ├── diagnose_boundary_discrepancy.py     # fast standalone RNG/AUROC diagnostic
│   ├── diagnose_nearest_vs_first.py         # confirms nearest vs. first-found pick different points
│   ├── diagnose_nearest_full_compare.py     # confirms they still give identical attack outcomes
│   ├── run_attacks.py     # run FGSM + boundary attack against any of the 4 local models
│   ├── results/           # tracked: JSON output of run_attacks.py (aggregate metrics, no PHI)
│   └── agent_usefulness_study/    # (M5) PROTOCOL.md + full execution harness (study_server.py,
│                                  # run_study.py, blind_transcripts.py, analyze_ratings.py) --
│                                  # has been run once, see Status and PROTOCOL.md
├── scripts/
│   └── run_local_server.py  # stdio entry point for Claude Desktop -- pre-registers one of the
│                             # 4 local models under a fixed handle, see "Running with Claude Desktop"
├── reference/              # tracked: validated result tables (aggregate metrics, no PHI)
├── examples/               # (M5) end_to_end_security_evaluation.md -- illustrative, hand-authored
│                           # transcript with real reference numbers, not a live-recorded session
├── docs/                   # (M5) TOOLS.md -- practical tool-by-tool usage reference
└── .github/workflows/      # (M5) ci.yml -- ruff + mypy + pytest, path-scoped to this connector

../data/                    # sibling to this package, gitignored (see ../.gitignore)
├── models/                 # icu_mortality_rf.pkl, ed_admission_rf.pkl, ckd_rf.pkl, wdbc_rf.pkl, *_meta.json
├── mimic/                  # icu_cohort/, ed_cohort/ train+test splits
├── ckd/processed/          # train+test splits
└── breast_cancer/processed/  # train+test splits
```

## Running the tests

This project's path (deeply nested under the thesis directory tree) exceeds
Windows' 260-character limit for scikit-learn's compiled binaries, so the
venv must live at a short path outside the project. Replace `<you>` below
with your actual Windows username (e.g. `C:\Users\wisdo\...`) -- it's a
placeholder, not literal text to paste:

```powershell
uv venv --python 3.11 "C:\Users\<you>\.venvs\healthsec-mcp"
uv pip install -e ".[dev]" --python "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe"

# fast suites (~25s) -- unit, fault-injection, and MCP contract tests
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" -m pytest tests/unit tests/fault_injection tests/contract -v

# golden regression against ../data/ (~7 min -- LIME explains each sample individually)
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" -m pytest tests/golden -v -s

# lint + type-check (instant, what CI runs)
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" -m ruff check src/ tests/ reproduce/ scripts/
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" -m mypy src/ scripts/run_local_server.py
```

`../data/models/` and `../data/mimic/` must be populated first (see Data below).
CI (`.github/workflows/ci.yml`) runs `pytest tests/` directly, letting the
MIMIC-IV-dependent golden tests skip automatically via their own
`skipif` — `../data/` is gitignored and never present in CI.

## Running attacks against a model

`reproduce/run_attacks.py` runs FGSM + boundary attack against any of the
four locally available models. `icu_mortality`/`ed_admission` are the
regression baseline (compared against `reference/` in the golden tests);
`ckd`/`wdbc` have no published ground truth -- this is new evaluation used
to demonstrate the tools generalize beyond the two validated cohorts.

```powershell
# one dataset at a time
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" reproduce\run_attacks.py --dataset ckd
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" reproduce\run_attacks.py --dataset wdbc
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" reproduce\run_attacks.py --dataset icu_mortality
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" reproduce\run_attacks.py --dataset ed_admission

# all four in one run (slowest -- icu_mortality/ed_admission each have up to
# 100 samples for boundary attack, similar runtime to the golden tests)
& "C:\Users\<you>\.venvs\healthsec-mcp\Scripts\python.exe" reproduce\run_attacks.py --dataset all
```

Each run writes its results to `reproduce/results/<dataset>_attack_results.json`
(tracked in git, overwritten on each run) so nothing is lost once the
terminal scrolls past it.

## Running with Claude Desktop

This connects `healthsec-mcp` to Claude Desktop as a local MCP server --
Claude Desktop spawns it as a subprocess and talks to it over stdio. This
is the intended way to actually use the connector day to day; the other
supported path is the agent-usefulness study harness
(`reproduce/agent_usefulness_study/`), which spawns the same server
programmatically to script Condition A/B comparisons instead.

### Two categories of tools, two setup paths

The 7 standards/SPS/report tools (`assess_attack_coverage`, `check_rbac`,
`score_audit_completeness`, `score_compliance`, `compute_sps`,
`generate_security_report`, `get_audit_log`) don't touch a registered
model at all -- they score or compose evidence you pass directly in the
tool call. **These work with zero setup** the moment Claude Desktop can
launch the server at all.

The 3 model-touching tools (`run_fgsm`, `run_boundary_attack`,
`run_membership_inference`) need a model registered under a `model_handle`
*before* Claude can call them. This is the part that trips people up:
Claude Desktop spawns `healthsec-mcp` as a brand-new subprocess with an
empty registry every time -- there's no interactive Python session inside
that subprocess for you to call `registry.register()` from after the fact.
`scripts/run_local_server.py` solves this: it's a small wrapper that
loads one of your 4 local models, registers it under a fixed, predictable
handle, and *then* starts the same server -- point Claude Desktop at this
script instead of the bare `healthsec-mcp` command if you need the
model-touching tools.

### Setup

**1. Make sure the venv is set up** (see "Running the tests" above if not).

**2. Decide which path you need:**

- Only need the standards/SPS/report tools? Skip to step 4 and point
  Claude Desktop at `healthsec-mcp` directly (or the equivalent
  `python -m healthsec_mcp.server`) -- no dataset flag needed.
- Need the model-touching tools too? Use `scripts/run_local_server.py`
  with one of `--dataset icu_mortality|ed_admission|ckd|wdbc`. This
  requires `../data/models/` and the matching processed dataset to
  already be populated (see "Data" below) -- `icu_mortality`/`ed_admission`
  need PhysioNet-credentialed MIMIC-IV data; `ckd`/`wdbc` are public and
  work out of the box if you've run the setup in "Running the tests."

**3. Find (or create) Claude Desktop's config file:**

```
%APPDATA%\Claude\claude_desktop_config.json
```

On Windows that's typically
`C:\Users\<you>\AppData\Roaming\Claude\claude_desktop_config.json`. If the
file doesn't exist yet, create it with just `{"mcpServers": {}}` and add
your entry inside.

**4. Add an entry under `mcpServers`.** Replace `<you>` with your actual
Windows username and adjust the repo path to match where you've cloned
this project. Model-touching setup (recommended default -- gives you all
10 tools):

```json
{
  "mcpServers": {
    "healthsec": {
      "command": "C:\\Users\\<you>\\.venvs\\healthsec-mcp\\Scripts\\python.exe",
      "args": [
        "C:\\Users\\<you>\\OneDrive\\Desktop\\University_of_the_Cumberlands\\Courses\\Thesis_2026_Proposing\\Papers_and_Code\\ai-agents-connectors\\01-healthcare-ai-security-connector\\healthsec-mcp\\scripts\\run_local_server.py",
        "--dataset",
        "icu_mortality"
      ]
    }
  }
}
```

Standards/report-tools-only setup (no model registration, works without
`../data/` at all):

```json
{
  "mcpServers": {
    "healthsec": {
      "command": "C:\\Users\\<you>\\.venvs\\healthsec-mcp\\Scripts\\healthsec-mcp.exe"
    }
  }
}
```

JSON requires double backslashes in Windows paths (`\\`, not `\`) --
copy the pattern above exactly, don't use single backslashes.

**5. Restart Claude Desktop completely** (quit from the system tray, not
just close the window) so it picks up the config change.

**6. Verify it connected.** In a new Claude Desktop chat, look for a
tools/connector icon indicating `healthsec` is available, or just ask
Claude something that requires a tool, e.g. *"What MCP tools do you have
available from healthsec?"* If nothing shows up, check Claude Desktop's
logs (Help menu, or `%APPDATA%\Claude\logs\`) for a subprocess spawn error
-- the most common cause is a typo'd path or JSON syntax error in the
config file.

### Using it

If you registered a model (step 2's model-touching path), reference the
handle directly in your prompt -- it equals the dataset name you chose,
e.g.:

> Using model_handle="icu_mortality", check whether this model is
> vulnerable to small adversarial perturbations.

For the standards/report tools, just supply the evidence directly (Claude
will ask for it, or you can paste it inline) -- see
[`docs/TOOLS.md`](docs/TOOLS.md) for a worked example of every tool,
including the exact input shapes each one expects.

### Troubleshooting

- **Claude says it has no tools from `healthsec`** -- almost always a
  config path typo, or Claude Desktop wasn't fully restarted. Check the
  logs mentioned in step 6.
- **Model-touching tools fail with "model_handle is not registered"** --
  you're pointed at the bare `healthsec-mcp` command instead of
  `scripts/run_local_server.py`, or the `--dataset` you chose doesn't
  match the handle you referenced in your prompt (they're the same
  string, e.g. `--dataset ckd` gives you `model_handle="ckd"`, not
  anything else).
- **`run_local_server.py` crashes on startup** -- almost always a missing
  file under `../data/`. `icu_mortality`/`ed_admission` specifically
  require the PhysioNet-credentialed MIMIC-IV data (see "Data" below);
  `ckd`/`wdbc` should work if you've completed the venv setup in "Running
  the tests," since those two are the ones with no such restriction.

## MCP tools

| Tool | Status | Authz-gated? | Input | Output |
|---|---|:---:|---|---|
| `run_fgsm` | implemented, golden-verified | yes | model, batch (n≤100), ε | flip rate, AUROC drop, plausibility rate |
| `run_boundary_attack` | implemented (known limitation, see Status) | yes | model, batch | flip rate, drop, mean steps |
| `run_membership_inference` | implemented, no published golden target | yes | model, member_pool + nonmember_pool (≤5k each) | MI accuracy/AUROC, privacy risk, patients-at-risk (direct count, not extrapolated) |
| `assess_attack_coverage` | implemented, golden-verified exactly | no | control set | PASS/PARTIAL/FAIL + mitigated/tested counts + coverage % |
| `check_rbac` | implemented, golden-verified exactly | no | already-executed probe results | pass count + enforcement rate |
| `score_audit_completeness` | implemented, golden-verified exactly | no | audit log entries | completeness rate |
| `score_compliance` | implemented, golden-verified exactly | no | HIPAA/FHIR checklist | per-standard % + overall % |
| `compute_sps` | implemented, golden-verified exactly (SPS=78.9 on reference inputs) | no | the 9 subscore inputs above | composite SPS 0–100, deployment tier, per-dimension breakdown |
| `generate_security_report` | implemented | no | any subset of the above tools' outputs | Markdown + structured report; only states a deployment recommendation if `sps` was supplied |
| `get_audit_log` | implemented | no | (none) | this session's full audit trail |

"Authz-gated" tools resolve `model_handle` through `authz.authorize()`, which
records every attempt to the audit trail whether it succeeds or is denied.
Standards, `compute_sps`, `generate_security_report`, and `get_audit_log`
don't touch a model at all -- they score/compose evidence the caller
already has -- so they aren't gated.

See [`docs/TOOLS.md`](docs/TOOLS.md) for a worked example of every tool
plus a full end-to-end workflow.

## Audit trail

Every authz-gated tool call is recorded to `audit.default_audit_log`
in-memory, for the life of the server process: `{timestamp, tool,
model_handle, authorized, input_hash, detail}`. This is the connector's own
non-repudiation record; retrieve it via the `get_audit_log` tool. You can
audit the auditor: `standards.audit.score_audit_completeness` will happily
score `default_audit_log.entries()` against the same completeness check it
runs on any other log, though the field names differ (this log's schema is
`authz.py`'s own, not the validated methodology's `REQUIRED_FIELDS`).

## Tech stack

Python 3.11 (not 3.13 — passlib/bcrypt incompatibility; use `import bcrypt`
directly), `mcp` (FastMCP), scikit-learn, numpy/pandas, FastAPI, pytest.
License: Apache 2.0.

## Data

MIMIC-IV ICU + ED cohorts and trained RF models. `reference/` (this
directory) holds the validated result tables — small, aggregate, no PHI,
safe to commit. `../data/` (one level up, sibling to this package) holds
the actual models and MIMIC-IV-derived data used to run the golden tests
locally — this requires PhysioNet credentialing and is gitignored at the
connector root (`../.gitignore`); it must never be committed or published.
Override its location with `CONNECTOR_DATA_ROOT` if needed.

## Safety

Adversarial and privacy-attack tools are scoped to a user's own models and
gated behind an authorization check (`authz.py`) — a model only becomes
usable by being registered directly through `registry.py`'s Python API in
the user's own script; there is no MCP tool that lets an agent register or
guess a handle. Every authorized *and* denied attempt is recorded to the
audit trail (`audit.py`) — see "Risks / limitations / ethics" in
`../STRUCTURE.md`.