Connects to InterSystems IRIS vector database for medical data storage, enabling GraphRAG entity and relationship queries, FHIR document search, and hybrid vector search with COSINE similarity.
Integrates NVIDIA NIM for generating 1024-dimensional embeddings used in medical document vectorization and semantic search.
Supports comprehensive testing suite including unit, integration, and end-to-end tests for the medical GraphRAG platform.
Implements MCP server tools using Python for medical search operations, FHIR document retrieval, and GraphRAG queries.
Provides interactive chat UI for medical queries with conversation history, execution transparency, and visualization of medical data and relationships.
Medical GraphRAG Assistant
A production-ready medical AI assistant platform built on Model Context Protocol (MCP), featuring GraphRAG multi-modal search, FHIR integration, NVIDIA NIM embeddings, and AWS Bedrock Claude Sonnet 4.5.
Originally forked from: FHIR-AI-Hackathon-Kit
Current Version: v2.14.0 (Auto Memory Recall & Interactive Graphs)
What This Is
An agentic medical chat platform with advanced capabilities:
š¤ Model Context Protocol (MCP) - Claude autonomously calls medical search tools
š§ GraphRAG - Knowledge graph-based retrieval with entity and relationship extraction
š¼ļø Medical Image Search - Semantic search over chest X-rays using NV-CLIP embeddings
š¾ Agent Memory System - Persistent semantic memory with vector search
š„ FHIR Integration - Full-text search of clinical documents
āļø AWS Deployment - Production deployment on AWS EC2 with NVIDIA A10G GPU
š Interactive UI - Streamlit interface with execution transparency
šļø InterSystems IRIS - Vector database with native VECTOR(DOUBLE, 1024) support
Quick Start
1. Run the Streamlit Chat Interface
Visit http://localhost:8501 and start chatting!
2. Use as MCP Server (Claude Desktop, etc.)
Architecture
System Overview
GraphRAG Data Flow
Component Interaction
IRIS Vector Package Architecture
This project uses the InterSystems IRIS Vector ecosystem:
Package Links:
iris-vector-rag- Production RAG framework with multiple pipelines (basic, graphrag, crag, multi_query_rrf)iris-vector-graph- Graph-oriented vector toolkit for GraphRAG workloads
NVIDIA NIM Architecture
This project uses NVIDIA NIM (Inference Microservices) for GPU-accelerated AI inference, deployed on AWS EC2 with NVIDIA A10G GPUs.
NIM Services:
Service | Model | Port | Purpose | Dimension |
NIM LLM |
| 8001 | Text generation, entity extraction | N/A |
NV-CLIP |
| 8002 | Multimodal embeddings (text + images) | 1024-dim |
NV-EmbedQA |
| Cloud API | Text embeddings for RAG | 1024-dim |
Deployment Options:
Self-hosted NIM (Production): Docker containers on GPU instances with HIPAA compliance
NVIDIA Cloud API (Development):
https://integrate.api.nvidia.com/v1with API key
Knowledge Graph: Entity & Relationship Extraction
The knowledge graph is built from FHIR DocumentReference resources using regex-based entity extraction with confidence scoring. No external medical ontology is currently used - entities are extracted using curated regex patterns.
Entity Types Extracted:
Type | Examples | Confidence Range |
| chest pain, shortness of breath, fever | 0.80 - 0.95 |
| hypertension, diabetes, pneumonia | 0.75 - 0.95 |
| aspirin, metformin, insulin | 0.85 - 0.95 |
| CT scan, MRI, blood test | 0.85 - 0.92 |
| chest, abdomen, heart | 0.75 - 0.90 |
| 3 days ago, last week | 0.75 - 0.95 |
Relationship Types:
Relationship | Pattern | Example |
| MEDICATION ā CONDITION/SYMPTOM | aspirin ā chest pain |
| CONDITION ā SYMPTOM | hypertension ā headache |
| SYMPTOM ā BODY_PART | pain ā chest |
| SYMPTOM ā SYMPTOM (within window) | fever ā cough |
Current Limitations & Future Work:
No medical ontology (SNOMED-CT, ICD-10, RxNorm) - extraction is pattern-based
No FHIR native queries - clinical notes are hex-decoded from DocumentReference.content
Future: LLM-based entity extraction for improved coverage and ontology mapping
MCP Tools Architecture
The MCP server exposes 14+ tools that Claude (or other LLMs) can autonomously call to search medical data.
Tool Categories:
Category | Tools | Data Source | Query Type |
FHIR Search |
| ClinicalNoteVectors | Full-text SQL LIKE |
GraphRAG |
| Entities, EntityRelationships | Entity + Vector + RRF |
Medical Images |
| MIMICCXRImages | VECTOR_COSINE (NV-CLIP) |
Agent Memory |
| AgentMemoryVectors | VECTOR_COSINE (NV-CLIP) |
Visualization |
| All sources | Plotly/NetworkX charts |
Note: FHIR queries are performed via SQL on pre-ingested data in IRIS tables. The system does not make live FHIR REST API calls - documents are batch-loaded during setup and stored with their embeddings in IRIS vector columns.
Data Pipeline: Ingestion ā Storage ā Query
Note: Current implementation uses batch vectorization on initial data load. Vectors are stored in standard VECTOR columns and require manual re-vectorization when source documents change. See Future Enhancements for planned automatic sync capabilities.
IRIS Database Schema
Features
MCP Tools (10+ available)
FHIR & GraphRAG:
search_fhir_documents - Full-text search of clinical notes
get_document_details - Retrieve complete clinical notes by ID
search_knowledge_graph - Search medical entities (symptoms, conditions, medications)
hybrid_search - Combined vector + graph search with RRF fusion
get_entity_statistics - Knowledge graph statistics and insights
Medical Images: 6. search_medical_images - Semantic search over chest X-rays with NV-CLIP
Agent Memory: 7. remember_information - Store semantic memories (corrections, knowledge, preferences, feedback) 8. recall_information - Semantic search over agent memories 9. get_memory_stats - Memory system statistics
Visualizations: 10. plot_symptom_frequency - Chart of most common symptoms 11. plot_entity_distribution - Entity type distribution charts 12. plot_patient_timeline - Patient encounter timeline 13. plot_entity_network - Knowledge graph relationship visualization 14. visualize_graphrag_results - Interactive GraphRAG search results
Chat Interface Features
ā Multi-Modal Search - Search clinical text, medical images, and knowledge graph
ā Agent Memory - Persistent semantic memory with vector search
ā Medical Image Display - View chest X-rays with DICOM support
ā Execution Transparency - See which tools Claude calls and its reasoning
ā Interactive Charts - Generate visualizations from data
ā Conversation History - Multi-turn conversations with context
ā Memory Editor - Browse, search, add, and delete agent memories in sidebar
ā Error Handling - Graceful handling of API issues with detailed logs
ā Max Iterations Control - Prevents infinite loops (10 iteration limit)
Current Version: v2.14.0
Recent Features (v2.14.0):
ā Auto Memory Recall: Memories automatically recalled before each query to guide tool selection
ā Interactive Graph Viz: Force-directed, draggable graphs with
streamlit-agraphā Memory in Execution Log: See recalled memories in "Show Execution Details" pane
ā NetworkX-powered graph layouts with physics simulation
Previous Updates:
v2.13.0: Multi-LLM provider support (NIM > OpenAI > Bedrock), OneDrive backup
v2.12.0: Agent memory system with pure IRIS vector storage
v2.10.2: Fixed content processing errors, increased max iterations
v2.10.0: GraphRAG multi-modal search with RRF fusion
v2.0.0: AWS deployment with NVIDIA NIM integration
Configuration
Required Environment Variables
Config Files
config/fhir_graphrag_config.yaml- Local development configconfig/fhir_graphrag_config.aws.yaml- AWS production config (active)config/aws-config.yaml- AWS infrastructure settings
Project Structure
Technology Stack
AI/ML:
AWS Bedrock (Claude Sonnet 4.5)
NVIDIA NV-CLIP (1024-dim multimodal embeddings)
NVIDIA NIM (Inference Microservices)
Model Context Protocol (MCP)
Database & Vector Storage:
InterSystems IRIS Community Edition (AWS EC2)
Native VECTOR(DOUBLE, 1024) support
VECTOR_COSINE similarity search
Tables: ClinicalNoteVectors, MIMICCXRImages, Entities, EntityRelationships, AgentMemoryVectors
InterSystems IRIS Vector Packages:
iris-vector-rag- Production RAG framework with BYOT storage, GraphRAG pipelines, and CloudConfiguration APIiris-vector-graph- Graph-oriented vector toolkit for entity storage and relationship traversalintersystems-irispython- Native IRIS database driver
Infrastructure:
AWS EC2 g5.xlarge (NVIDIA A10G GPU)
Python 3.10+
Streamlit for UI
Docker for containerization
Key Libraries:
fhirpy- FHIR resource parsing and handlingboto3- AWS SDKstreamlit- Chat UIstreamlit-agraph- Interactive graph visualizationmcp- Model Context Protocol SDKpydicom- DICOM medical image processingnetworkx- Graph algorithms and layout
Example Queries
Try these in the chat interface:
FHIR Search:
"Find patients with chest pain"
"Search for diabetes cases"
"Show recent emergency visits"
GraphRAG:
"What medications treat hypertension?"
"Show me the relationship between conditions and procedures"
"What are the side effects of metformin?"
Medical Images:
"Show me chest X-rays of pneumonia"
"Find chest X-rays showing cardiomegaly"
"Search for lateral view chest X-rays"
Agent Memory:
"Remember that I prefer concise clinical summaries"
"What do you know about my preferences?"
"Recall any corrections I've given you about medical terminology"
Hybrid Search:
"Find treatment options for chronic pain" (combines vector + graph + image search)
Visualization:
"Show a chart of conditions by frequency"
"Visualize the knowledge graph for chest pain"
"Graph the entity relationships"
Backup
The project uses OneDrive for automatic cloud backup:
Backup includes all code, configs, and medical images (~195 MB). OneDrive automatically syncs to cloud.
Development
Running Tests
Debug Mode
Enable debug logging:
AWS Deployment
The system is deployed on AWS EC2 with:
Instance: g5.xlarge (NVIDIA A10G GPU)
Region: us-east-1
Database: InterSystems IRIS Community Edition
GPU Services: NVIDIA NIM for NV-CLIP embeddings
Data: 50 medical images, 51 clinical notes, 83 entities, 540 relationships
See docs/deployment-guide.md for detailed deployment instructions.
Troubleshooting
See docs/troubleshooting.md for common issues.
Common Issues:
AWS credentials not configured ā Set AWS_PROFILE or AWS env vars
IRIS connection failed ā Check IRIS_HOST and credentials
NV-CLIP not responding ā Check NVCLIP_BASE_URL and SSH tunnel
Medical images not found ā Verify image paths and DICOM support
Memory search returning 0 results ā Check embeddings with magnitude test
Max iterations reached ā Query may be too complex, try simplifying
Documentation
Core Documentation
Architecture Overview - System design and data flow
Deployment Guide - AWS deployment instructions
Troubleshooting Guide - Common issues and solutions
Current Session Docs
EMBEDDINGS_FIXED.md - Image and memory embeddings fix
MEMORY_SEARCH_BROWSE_FIX.md - Memory search UI fix
PROGRESS.md - Development history and achievements
TODO.md - Current tasks and roadmap
Historical Documentation
archive/ - Old implementations, scripts, and session docs
Future Enhancements
Automatic Vector Synchronization
Current State: Vectors are generated via batch processing during initial data load. When FHIR documents are updated in the repository, embeddings must be manually re-generated.
Planned Enhancement: Leverage IRIS EMBEDDING column type for automatic vector synchronization:
Benefits:
Automatic re-vectorization when
TextContentchangesNo manual batch re-processing required
Real-time sync between FHIR repository and vector store
Additional Planned Features
FHIR Subscription Hooks - Trigger vectorization on resource create/update events
Incremental Knowledge Graph Updates - Update entities/relationships without full rebuild
IRIS HealthShare Integration - Direct FHIR R4 repository connection
Vector Index Optimization - HNSW index tuning for larger datasets
Multi-tenant Support - Namespace isolation for multiple healthcare organizations
Contributing
This project is based on the FHIR-AI-Hackathon-Kit. The original tutorial content remains in the tutorial/ directory.
License
Inherits license from upstream FHIR-AI-Hackathon-Kit repository.
Acknowledgments
Original Project: FHIR-AI-Hackathon-Kit by gabriel-ing
InterSystems IRIS for the vector database platform
AWS Bedrock for Claude Sonnet 4.5 access
NVIDIA NIM for NV-CLIP multimodal embeddings
Model Context Protocol by Anthropic
MIMIC-CXR dataset for medical imaging data