MedImageParse MCP — Healthcare-AI Segmentation as an Agent Tool

What this repo tries to achieve

This repo answers one question: "can a small, open healthcare-AI vision model be wired up as a tool an AI agent can call, safely and cheaply?"

It deploys Microsoft's MedImageParse (2D) and MedImageParse3D foundation models — biomedical image segmentation models that take a free-text entity prompt ("tumor", "white matter lesion", "liver", ...) and return a segmentation mask — as Azure ML managed online endpoints, then wraps them two ways so any agent framework can use them as a tool:

1. A local MCP server (scripts/mcp_server.py, stdio transport) for agent runtimes that spawn local MCP servers directly.

2. An Azure API Management (APIM) MCP gateway (apim/) that fronts the same endpoints with a hosted, network-reachable MCP server — the shape needed for Microsoft Foundry agents, which call tools over HTTP, not via a local process.

Both paths were validated against real clinical images (an MS-lesion FLAIR scan and a brain-tumor T1Gad scan), not just synthetic data — see reports/medimageparse-validation.md for the actual detection numbers.

⚠️ Research/demo use only. MedImageParse/MedImageParse3D are released for research purposes; this project is a technical validation, not a clinical tool. Do not use outputs for diagnosis.

Related MCP server: simple-vision-mcp

Architecture

                         ┌─────────────────────────────┐
  Foundry / any MCP      │   APIM MCP gateway            │
  agent  ───HTTPS───────▶│   /medimageparse-mcp/mcp      │
                         │   (JSON-RPC 2.0 over HTTPS)   │
                         └──────────────┬────────────────┘
                                        │ backend + policy
                                        │ (injects endpoint key)
                         ┌──────────────▼────────────────┐
                         │  REST APIs (per-endpoint)      │
                         │  /mip2d/score   /mip3d/score   │
                         └──────────────┬────────────────┘
                                        │ HTTPS + key auth
                    ┌───────────────────┴────────────────────┐
                    ▼                                          ▼
     Azure ML managed online endpoint          Azure ML managed online endpoint
     medimageparse2d (MedImageParse:17)        medimageparse3d (MedImageParse3D:6)
     Standard_NC24ads_A100_v4                  Standard_NC24ads_A100_v4

  (local alternative, no APIM/network hop:)
  Agent ──stdio──▶ scripts/mcp_server.py ──HTTPS+key──▶ same two AML endpoints

Prerequisites

• An Azure subscription with AML/GPU quota for Standard_NC24ads_A100_v4 (24 cores each; MedImageParse/3D need an A100-class SKU).

• Azure CLI, logged in (az login), with access to create resource groups.

• Python 3.10+.

• No pre-existing resources are required or touched — this project creates its own resource group(s) end to end (see scripts/config.py).

1. Configure your target (no secrets committed)

Every script reads its Azure target from environment variables via scripts/config.py — nothing subscription-specific is hardcoded in this repo.

$env:AZURE_SUBSCRIPTION_ID = "<your-subscription-guid>"
$env:AML_RESOURCE_GROUP    = "medimageparse-mcp-rg"   # optional, this is the default
$env:AML_WORKSPACE         = "medimageparse-mcp-ws"   # optional, this is the default
$env:AML_LOCATION          = "eastus2"                # optional, this is the default — must have your GPU quota

2. Install dependencies

python -m venv .venv
.\.venv\Scripts\pip.exe install -r requirements.txt

3. Deploy — step by step

# 1. Confirm the model versions available in the public azureml registry
.\.venv\Scripts\python.exe scripts\check_models.py

# 2. Create the resource group, then the AML workspace
az group create -n $env:AML_RESOURCE_GROUP -l $env:AML_LOCATION
.\.venv\Scripts\python.exe scripts\create_ws.py

# 3. Enable Managed Virtual Network on the workspace.
#    Required if your subscription has an org policy that forces
#    publicNetworkAccess=Disabled on storage accounts (common in
#    enterprise subs) — the managed VNet auto-provisions private endpoints
#    from AML to its own storage/keyvault so deployments can still write
#    their model manifest. Safe to run even if you don't have this policy.
#    IMPORTANT: must run before any online endpoint exists in the workspace.
.\.venv\Scripts\python.exe scripts\enable_managed_vnet.py

# 4. Deploy both models as managed online endpoints (each takes ~10-20 min)
.\.venv\Scripts\python.exe scripts\deploy_3d.py
.\.venv\Scripts\python.exe scripts\deploy_2d.py

# 5. Smoke-test both endpoints directly (bypassing MCP/APIM)
.\.venv\Scripts\python.exe scripts\smoke_test.py

deploy_2d.py / deploy_3d.py write endpoint_2d.env / endpoint_3d.env (scoring URI + key) — these are git-ignored, never committed. Run all commands from the repo root (not from inside scripts/), so these .env files land next to apim/ where generate_params.py expects them.

If step 4 fails with a storage permissions error

If your subscription force-disables public storage access, the very first deployment attempt after workspace creation can fail with: BadArgument: Unable to upload to storage manifest due to permissions error. This is an RBAC/private-endpoint propagation delay, not a real misconfiguration — grant Storage Blob Data Contributor on the workspace's storage account to the workspace's managed identity (scripts/get_ws_identity.py prints the principal ID), delete the stuck deployment, wait 30-50 minutes, and retry. Short retries (a few minutes apart) will keep failing even though the fix is already correct — this is the one step in the whole pipeline that needs patience, not more debugging.

If traffic is 0% after deployment succeeds

deploy_2d.py/deploy_3d.py set traffic={"blue": 100} as their last step. If that step doesn't run (e.g. your terminal/token session drops mid-deployment), the endpoint exists but returns connection resets because no deployment is receiving traffic. Fix (run from the repo root):

.\.venv\Scripts\python.exe -c "
import sys; sys.path.insert(0, 'scripts')
from azure.ai.ml import MLClient
from azure.ai.ml.entities import ManagedOnlineEndpoint
from azure.identity import AzureCliCredential
from config import SUBSCRIPTION_ID, RESOURCE_GROUP, WORKSPACE
c = MLClient(AzureCliCredential(), SUBSCRIPTION_ID, RESOURCE_GROUP, workspace_name=WORKSPACE)
c.online_endpoints.begin_create_or_update(ManagedOnlineEndpoint(name='medimageparse2d', auth_mode='key', traffic={'blue': 100})).result()
c.online_endpoints.begin_create_or_update(ManagedOnlineEndpoint(name='medimageparse3d', auth_mode='key', traffic={'blue': 100})).result()
"

4. Run the local MCP server

.\.venv\Scripts\python.exe scripts\mcp_server.py

Tools exposed:

• list_supported_prompts() — example free-text entity prompts.

• segment_2d_image(image_path, prompt) — pixel stats + bounding box + a saved mask PNG path. Never returns raw base64 to the caller (keeps agent context small).

• segment_3d_volume(nifti_path, prompt) — voxel stats + physical volume (mm³) + bounding box + a saved mask NIfTI path.

5. Deploy the APIM MCP gateway (for Foundry / remote agents)

az group create -n medimageparse-mcp-apim-rg -l eastus2
.\.venv\Scripts\python.exe scripts\generate_params.py   # builds apim/main.parameters.json from the endpoint_*.env files
az deployment group create -g medimageparse-mcp-apim-rg `
  --template-file apim\main.bicep --parameters apim\main.parameters.json

Read the gatewayUrl/mcpEndpoint outputs from the deployment result, or:

az deployment group show -g medimageparse-mcp-apim-rg -n main --query properties.outputs -o json

subscriptionRequired: false on both the product and the MCP API — anonymous access, meant for this demo only. Add a subscription key or switch to AAD auth (apim/main.bicep has a ponytail: comment with the upgrade path) before exposing this beyond a private test.

End-to-end testing

Three levels, cheapest/fastest first:

# 1. Direct-to-endpoint smoke test (no MCP, no APIM) — confirms the models
#    themselves are healthy and detecting real pathology correctly.
.\.venv\Scripts\python.exe scripts\smoke_test.py

# 2. Local MCP tool test — calls the mcp_server.py tool functions directly
#    (bypassing the MCP stdio transport) to verify the tool-wrapper logic
#    (stats/bbox extraction, mask file saving) end to end.
.\.venv\Scripts\python.exe scripts\test_mcp_tools.py

# 3. Through the APIM gateway — proves the hosted MCP path an external
#    agent would actually use.

For (3), test the REST layer and the native MCP layer separately:

# REST (same input_data body shape as the raw AML endpoint)
curl -X POST "https://<your-apim-name>.azure-api.net/mip2d/score" `
  -H "Content-Type: application/json" `
  -d '{"input_data":{"columns":["image","text"],"index":[0],"data":[["<base64-png>","tumor"]]}}'

# MCP tools/list
curl -X POST "https://<your-apim-name>.azure-api.net/medimageparse-mcp/mcp" `
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" `
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# MCP tools/call
curl -X POST "https://<your-apim-name>.azure-api.net/medimageparse-mcp/mcp" `
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" `
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"segment_2d_image","arguments":{"ScoreRequest":{"input_data":{"columns":["image","text"],"index":[0],"data":[["<base64-png>","tumor"]]}}}}}'

A passing test returns HTTP 200 with a JSON-RPC result.content payload containing the segmentation mask (base64-encoded inside image_features/ nifti_file, matching the raw AML endpoint's own response shape).

Request execution sequence

Agent                 APIM MCP gateway            APIM REST API          AML online endpoint
 │  tools/call            │                            │                       │
 │  segment_2d_image ────▶│                            │                       │
 │                        │  maps MCP tool → operation  │                       │
 │                        │  ─────────────────────────▶│                       │
 │                        │                             │  policy: inject      │
 │                        │                             │  Authorization:      │
 │                        │                             │  Bearer {{key}}      │
 │                        │                             │  ────────────────────▶│
 │                        │                             │                       │  score.py runs
 │                        │                             │                       │  model inference
 │                        │                             │  ◀────────────────────│  returns mask JSON
 │                        │  ◀───────────────────────────                       │
 │  ◀─────────────────────│  wraps REST response as     │                       │
 │  JSON-RPC result       │  MCP tool content            │                       │

Key point: the agent never sees or handles the AML endpoint key — APIM's policy injects it server-side from a named value, so the gateway URL alone is a safe thing to hand to an agent or tool registry.

Using this as a tool for a Microsoft Foundry agent

Foundry agents consume tools as remote MCP servers reachable over HTTP(S) — they cannot spawn a local stdio process the way scripts/mcp_server.py does, so the APIM gateway from step 5 is the piece Foundry actually needs.

1. Deploy the APIM gateway (step 5 above) and note the mcpEndpoint output, e.g. https://<apim-name>.azure-api.net/medimageparse-mcp/mcp.

2. In your Foundry agent/project, add a tool of type MCP server (or equivalent "Custom/Remote tool" entry in the agent's tool configuration), pointing at that URL. Foundry auto-discovers the tools via the server's tools/list response (segment_2d_image, segment_3d_volume) — no manual per-tool schema entry needed, since MCP tools are self-describing.

3. If the gateway isn't anonymous (recommended for anything beyond a private demo — see the auth note in step 5), configure the agent's tool credential as an APIM subscription key or AAD token, matching whichever auth you added to apim/main.bicep.

4. The agent can now call segment_2d_image(ScoreRequest={...}) / segment_3d_volume(ScoreRequest={...}) like any other tool — the base64 image/volume bytes go in input_data.data[0][0], the free-text prompt in input_data.data[0][1], matching the openapi-2d.json/openapi-3d.json schemas in apim/.

5. No separate registration step is needed beyond adding the MCP server URL — APIM's type: 'mcp' API IS the tool registry Foundry talks to.

Test / mock data (all included in this repo)

• test_data/2d/*.png — individual panels cropped from real MS-lesion (FLAIR) and brain-tumor (T1Gad) clinical figures, via scripts/crop_panels.py. The two source montage images were user-supplied chat attachments (not a public URL), so they aren't re-includable as a download source — the already-cropped panels in test_data/2d/ are the checked-in data; scripts/crop_panels.py documents exactly how they were produced.

• test_data/synthetic_*.nii.gz — synthetic ellipsoid "brain" volumes with an inserted lesion sphere, generated by scripts/make_synthetic_volumes.py (run it to regenerate, or to make your own variants). For 3D pipeline validation only — see the validation report for why real BraTS/MSSEG volumes weren't used (multi-GB downloads, registration required).

Teardown

.\.venv\Scripts\python.exe scripts\teardown.py
az group delete -n medimageparse-mcp-apim-rg --yes --no-wait

Both use --no-wait (async, safe to run and walk away). A100 deployments cost ~$3-4/hr each while they exist (APIM Basicv2 adds ~$0.10/hr) — tear down promptly after testing. Caution: resource group deletion cannot be cancelled once triggered in Azure — confirm you're done before running this.