# MedImageParse MCP — Healthcare-AI Segmentation as an Agent Tool

## What this repo tries to achieve

This repo answers one question: **"can a small, open healthcare-AI vision
model be wired up as a tool an AI agent can call, safely and cheaply?"**

It deploys Microsoft's **MedImageParse** (2D) and **MedImageParse3D**
foundation models — biomedical image segmentation models that take a
free-text entity prompt ("tumor", "white matter lesion", "liver", ...) and
return a segmentation mask — as **Azure ML managed online endpoints**, then
wraps them two ways so any agent framework can use them as a tool:

1. **A local MCP server** (`scripts/mcp_server.py`, stdio transport) for
   agent runtimes that spawn local MCP servers directly.
2. **An Azure API Management (APIM) MCP gateway** (`apim/`) that fronts the
   same endpoints with a hosted, network-reachable MCP server — the shape
   needed for **Microsoft Foundry agents**, which call tools over HTTP, not
   via a local process.

Both paths were validated against **real clinical images** (an MS-lesion
FLAIR scan and a brain-tumor T1Gad scan), not just synthetic data — see
[`reports/medimageparse-validation.md`](reports/medimageparse-validation.md)
for the actual detection numbers.

> ⚠️ **Research/demo use only.** MedImageParse/MedImageParse3D are released
> for research purposes; this project is a technical validation, not a
> clinical tool. Do not use outputs for diagnosis.

## Architecture

```
                         ┌─────────────────────────────┐
  Foundry / any MCP      │   APIM MCP gateway            │
  agent  ───HTTPS───────▶│   /medimageparse-mcp/mcp      │
                         │   (JSON-RPC 2.0 over HTTPS)   │
                         └──────────────┬────────────────┘
                                        │ backend + policy
                                        │ (injects endpoint key)
                         ┌──────────────▼────────────────┐
                         │  REST APIs (per-endpoint)      │
                         │  /mip2d/score   /mip3d/score   │
                         └──────────────┬────────────────┘
                                        │ HTTPS + key auth
                    ┌───────────────────┴────────────────────┐
                    ▼                                          ▼
     Azure ML managed online endpoint          Azure ML managed online endpoint
     medimageparse2d (MedImageParse:17)        medimageparse3d (MedImageParse3D:6)
     Standard_NC24ads_A100_v4                  Standard_NC24ads_A100_v4

  (local alternative, no APIM/network hop:)
  Agent ──stdio──▶ scripts/mcp_server.py ──HTTPS+key──▶ same two AML endpoints
```

## Prerequisites

- An Azure subscription with **AML/GPU quota** for `Standard_NC24ads_A100_v4`
  (24 cores each; MedImageParse/3D need an A100-class SKU).
- Azure CLI, logged in (`az login`), with access to create resource groups.
- Python 3.10+.
- No pre-existing resources are required or touched — this project creates
  its own resource group(s) end to end (see `scripts/config.py`).

## 1. Configure your target (no secrets committed)

Every script reads its Azure target from environment variables via
`scripts/config.py` — nothing subscription-specific is hardcoded in this repo.

```powershell
$env:AZURE_SUBSCRIPTION_ID = "<your-subscription-guid>"
$env:AML_RESOURCE_GROUP    = "medimageparse-mcp-rg"   # optional, this is the default
$env:AML_WORKSPACE         = "medimageparse-mcp-ws"   # optional, this is the default
$env:AML_LOCATION          = "eastus2"                # optional, this is the default — must have your GPU quota
```

## 2. Install dependencies

```powershell
python -m venv .venv
.\.venv\Scripts\pip.exe install -r requirements.txt
```

## 3. Deploy — step by step

```powershell
# 1. Confirm the model versions available in the public azureml registry
.\.venv\Scripts\python.exe scripts\check_models.py

# 2. Create the resource group, then the AML workspace
az group create -n $env:AML_RESOURCE_GROUP -l $env:AML_LOCATION
.\.venv\Scripts\python.exe scripts\create_ws.py

# 3. Enable Managed Virtual Network on the workspace.
#    Required if your subscription has an org policy that forces
#    publicNetworkAccess=Disabled on storage accounts (common in
#    enterprise subs) — the managed VNet auto-provisions private endpoints
#    from AML to its own storage/keyvault so deployments can still write
#    their model manifest. Safe to run even if you don't have this policy.
#    IMPORTANT: must run before any online endpoint exists in the workspace.
.\.venv\Scripts\python.exe scripts\enable_managed_vnet.py

# 4. Deploy both models as managed online endpoints (each takes ~10-20 min)
.\.venv\Scripts\python.exe scripts\deploy_3d.py
.\.venv\Scripts\python.exe scripts\deploy_2d.py

# 5. Smoke-test both endpoints directly (bypassing MCP/APIM)
.\.venv\Scripts\python.exe scripts\smoke_test.py
```

`deploy_2d.py` / `deploy_3d.py` write `endpoint_2d.env` / `endpoint_3d.env`
(scoring URI + key) — **these are git-ignored**, never committed. Run all
commands from the repo root (not from inside `scripts/`), so these `.env`
files land next to `apim/` where `generate_params.py` expects them.

### If step 4 fails with a storage permissions error

If your subscription force-disables public storage access, the very first
deployment attempt after workspace creation can fail with:
`BadArgument: Unable to upload to storage manifest due to permissions error`.
This is an **RBAC/private-endpoint propagation delay**, not a real
misconfiguration — grant `Storage Blob Data Contributor` on the workspace's
storage account to the workspace's managed identity (`scripts/get_ws_identity.py`
prints the principal ID), delete the stuck deployment, **wait 30-50
minutes**, and retry. Short retries (a few minutes apart) will keep failing
even though the fix is already correct — this is the one step in the whole
pipeline that needs patience, not more debugging.

### If traffic is 0% after deployment succeeds

`deploy_2d.py`/`deploy_3d.py` set `traffic={"blue": 100}` as their last
step. If that step doesn't run (e.g. your terminal/token session drops
mid-deployment), the endpoint exists but returns connection resets because
no deployment is receiving traffic. Fix (run from the repo root):

```powershell
.\.venv\Scripts\python.exe -c "
import sys; sys.path.insert(0, 'scripts')
from azure.ai.ml import MLClient
from azure.ai.ml.entities import ManagedOnlineEndpoint
from azure.identity import AzureCliCredential
from config import SUBSCRIPTION_ID, RESOURCE_GROUP, WORKSPACE
c = MLClient(AzureCliCredential(), SUBSCRIPTION_ID, RESOURCE_GROUP, workspace_name=WORKSPACE)
c.online_endpoints.begin_create_or_update(ManagedOnlineEndpoint(name='medimageparse2d', auth_mode='key', traffic={'blue': 100})).result()
c.online_endpoints.begin_create_or_update(ManagedOnlineEndpoint(name='medimageparse3d', auth_mode='key', traffic={'blue': 100})).result()
"
```

## 4. Run the local MCP server

```powershell
.\.venv\Scripts\python.exe scripts\mcp_server.py
```

Tools exposed:
- `list_supported_prompts()` — example free-text entity prompts.
- `segment_2d_image(image_path, prompt)` — pixel stats + bounding box +
  a saved mask PNG path. Never returns raw base64 to the caller (keeps
  agent context small).
- `segment_3d_volume(nifti_path, prompt)` — voxel stats + physical volume
  (mm³) + bounding box + a saved mask NIfTI path.

## 5. Deploy the APIM MCP gateway (for Foundry / remote agents)

```powershell
az group create -n medimageparse-mcp-apim-rg -l eastus2
.\.venv\Scripts\python.exe scripts\generate_params.py   # builds apim/main.parameters.json from the endpoint_*.env files
az deployment group create -g medimageparse-mcp-apim-rg `
  --template-file apim\main.bicep --parameters apim\main.parameters.json
```

Read the `gatewayUrl`/`mcpEndpoint` outputs from the deployment result, or:

```powershell
az deployment group show -g medimageparse-mcp-apim-rg -n main --query properties.outputs -o json
```

`subscriptionRequired: false` on both the product and the MCP API —
anonymous access, meant for this demo only. Add a subscription key or
switch to AAD auth (`apim/main.bicep` has a `ponytail:` comment with the
upgrade path) before exposing this beyond a private test.

## End-to-end testing

Three levels, cheapest/fastest first:

```powershell
# 1. Direct-to-endpoint smoke test (no MCP, no APIM) — confirms the models
#    themselves are healthy and detecting real pathology correctly.
.\.venv\Scripts\python.exe scripts\smoke_test.py

# 2. Local MCP tool test — calls the mcp_server.py tool functions directly
#    (bypassing the MCP stdio transport) to verify the tool-wrapper logic
#    (stats/bbox extraction, mask file saving) end to end.
.\.venv\Scripts\python.exe scripts\test_mcp_tools.py

# 3. Through the APIM gateway — proves the hosted MCP path an external
#    agent would actually use.
```

For (3), test the REST layer and the native MCP layer separately:

```powershell
# REST (same input_data body shape as the raw AML endpoint)
curl -X POST "https://<your-apim-name>.azure-api.net/mip2d/score" `
  -H "Content-Type: application/json" `
  -d '{"input_data":{"columns":["image","text"],"index":[0],"data":[["<base64-png>","tumor"]]}}'

# MCP tools/list
curl -X POST "https://<your-apim-name>.azure-api.net/medimageparse-mcp/mcp" `
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" `
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# MCP tools/call
curl -X POST "https://<your-apim-name>.azure-api.net/medimageparse-mcp/mcp" `
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" `
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"segment_2d_image","arguments":{"ScoreRequest":{"input_data":{"columns":["image","text"],"index":[0],"data":[["<base64-png>","tumor"]]}}}}}'
```

A passing test returns HTTP 200 with a JSON-RPC `result.content` payload
containing the segmentation mask (base64-encoded inside `image_features`/
`nifti_file`, matching the raw AML endpoint's own response shape).

## Request execution sequence

```
Agent                 APIM MCP gateway            APIM REST API          AML online endpoint
 │  tools/call            │                            │                       │
 │  segment_2d_image ────▶│                            │                       │
 │                        │  maps MCP tool → operation  │                       │
 │                        │  ─────────────────────────▶│                       │
 │                        │                             │  policy: inject      │
 │                        │                             │  Authorization:      │
 │                        │                             │  Bearer {{key}}      │
 │                        │                             │  ────────────────────▶│
 │                        │                             │                       │  score.py runs
 │                        │                             │                       │  model inference
 │                        │                             │  ◀────────────────────│  returns mask JSON
 │                        │  ◀───────────────────────────                       │
 │  ◀─────────────────────│  wraps REST response as     │                       │
 │  JSON-RPC result       │  MCP tool content            │                       │
```

Key point: the **agent never sees or handles the AML endpoint key** — APIM's
policy injects it server-side from a named value, so the gateway URL alone
is a safe thing to hand to an agent or tool registry.

## Using this as a tool for a Microsoft Foundry agent

Foundry agents consume tools as **remote MCP servers reachable over HTTP(S)**
— they cannot spawn a local stdio process the way `scripts/mcp_server.py`
does, so the **APIM gateway from step 5 is the piece Foundry actually needs**.

1. Deploy the APIM gateway (step 5 above) and note the `mcpEndpoint` output,
   e.g. `https://<apim-name>.azure-api.net/medimageparse-mcp/mcp`.
2. In your Foundry agent/project, add a **tool** of type **MCP server** (or
   equivalent "Custom/Remote tool" entry in the agent's tool configuration),
   pointing at that URL. Foundry auto-discovers the tools via the server's
   `tools/list` response (`segment_2d_image`, `segment_3d_volume`) — no
   manual per-tool schema entry needed, since MCP tools are self-describing.
3. If the gateway isn't anonymous (recommended for anything beyond a private
   demo — see the auth note in step 5), configure the agent's tool
   credential as an APIM subscription key or AAD token, matching whichever
   auth you added to `apim/main.bicep`.
4. The agent can now call `segment_2d_image(ScoreRequest={...})` /
   `segment_3d_volume(ScoreRequest={...})` like any other tool — the base64
   image/volume bytes go in `input_data.data[0][0]`, the free-text prompt in
   `input_data.data[0][1]`, matching the `openapi-2d.json`/`openapi-3d.json`
   schemas in `apim/`.
5. No separate registration step is needed beyond adding the MCP server URL
   — APIM's `type: 'mcp'` API IS the tool registry Foundry talks to.

## Test / mock data (all included in this repo)

- `test_data/2d/*.png` — individual panels cropped from real MS-lesion
  (FLAIR) and brain-tumor (T1Gad) clinical figures, via `scripts/crop_panels.py`.
  The two source montage images were user-supplied chat attachments (not a
  public URL), so they aren't re-includable as a download source — the
  already-cropped panels in `test_data/2d/` **are** the checked-in data;
  `scripts/crop_panels.py` documents exactly how they were produced.
- `test_data/synthetic_*.nii.gz` — synthetic ellipsoid "brain" volumes with
  an inserted lesion sphere, generated by `scripts/make_synthetic_volumes.py`
  (run it to regenerate, or to make your own variants). For **3D pipeline
  validation only** — see the validation report for why real BraTS/MSSEG
  volumes weren't used (multi-GB downloads, registration required).

## Teardown

```powershell
.\.venv\Scripts\python.exe scripts\teardown.py
az group delete -n medimageparse-mcp-apim-rg --yes --no-wait
```

Both use `--no-wait` (async, safe to run and walk away). A100 deployments
cost ~$3-4/hr **each** while they exist (APIM Basicv2 adds ~$0.10/hr) — tear
down promptly after testing. **Caution:** resource group deletion cannot be
cancelled once triggered in Azure — confirm you're done before running this.