mcp-nv-segment-ctmr

A FastMCP server exposing NV-Segment-CTMR (NVIDIA's VISTA3D-based 3D medical image segmentation foundation model) as MCP tools, deployed via Prefect Horizon. GPU inference is dispatched to a Modal serverless L40S endpoint, so the container needs no GPU, no MONAI, and no model weights.

Research use only. NV-Segment-CTMR is released under the NVIDIA OneWay Non-Commercial License. It is not a cleared diagnostic device and its outputs must not drive patient care decisions.

Why this one is different

The other servers in this family (locate-anything, fundus-agent, MIRAGE) pass images as base64 in a tool argument. That does not work here:

So the contract inverts. Volumes are passed by URI, the mask is written server-side and returned as a download URL, and long jobs use submit/poll rather than a blocking request. What crosses the MCP boundary is structure names in and volumes in mL out — things an agent can actually reason over.

Related MCP server: media-mcp

Architecture

MCP client ──► Horizon (this container, CPU)
                 │  name→index resolution, URI preflight, job brokering
                 ▼
              Modal ASGI app (CPU)  ──►  SegmentCTMR class (L40S)
                 │  /jobs /labels /files      VISTA3D sliding-window inference
                 ▼
              Modal Volume  ──►  mask .nii.gz download URL

The Modal side lives in modal_nv_segment_ctmr.py; see download_nv_segment_ctmr.py for fetching the HF repo and the volume setup steps.

Tools

Typical agent flow

list_anatomical_structures(query="kidney")
  → {"kidney_right": 14, "kidney_left": 5, "kidney_cyst_left": 116, …}

check_image_uri("https://…/abdomen_ct.nii.gz")
  → {"success": true, "size_mb": 62.4, "filename_looks_like_nifti": true}

segment_structures(
    image_uri="https://…/abdomen_ct.nii.gz",
    study_id="case_0042",
    structures=["liver", "spleen", "kidney_left", "kidney_right"])
  → {"structures": {"liver": {"volume_ml": 1487.2, …}, …},
     "mask_download_url": "https://…/files/case_0042-17.../case_0042_seg.nii.gz"}

segment_everything returns a call_id immediately; feed it to get_segmentation_status (optionally with wait_s) until it reports done.

Design notes

Name resolution happens here, not on the GPU. The class vocabulary is fetched once from Modal's CPU tier and memoized for an hour. Matching is case-insensitive exact, then normalized, then unique-substring — so "left kidney", "kidney_left", and "Kidney Left" all resolve. Ambiguous or unknown names come back as a validation error with a pointer to list_anatomical_structures rather than a MONAI stack trace.

Prompt validation is local. Indices are range-checked and filtered against the checkpoint's unsupported set (16, 129–131, 133, 137–145, 162) before anything is dispatched, so a bad request never costs GPU time.

Results are trimmed. A 117-class result is a lot of tokens to spend on organs with 0.2 mL of predicted volume. Structures are ranked by volume and capped at max_structures (default 40); bounding boxes are opt-in.

MRI_BRAIN has a guard. The checkpoint only supports skull-stripped, intensity-normalized T1. Running it on a raw T1 produces confident nonsense, so brain_preprocessed=True is required as an explicit acknowledgment.

Configuration

Connecting

{
  "mcpServers": {
    "nv-segment-ctmr": {
      "url": "https://nv-segment-ctmr.fastmcp.app/mcp"
    }
  }
}

Running

# Docker
docker build -t nv-segment-ctmr-mcp .
docker run -p 8080:8080 \
  -e MODAL_API_URL=https://<your-modal-app>.modal.run \
  nv-segment-ctmr-mcp

# Local
pip install -r requirements.txt
export MODAL_API_URL=https://<your-modal-app>.modal.run
python server.py

Multi-stage python:3.11-slim build, runs as a non-root user, exposes port 8080. Serves over stateless HTTP with JSON responses.

License

Apache License 2.0 for this code — see LICENSE. The underlying model is NVIDIA OneWay Non-Commercial.