Medical GraphRAG Assistant

A production-ready medical AI assistant platform built on Model Context Protocol (MCP), featuring GraphRAG multi-modal search, FHIR integration, NVIDIA NIM embeddings, and AWS Bedrock Claude Sonnet 4.5.

Originally forked from: FHIR-AI-Hackathon-Kit

Current Version: v2.14.0 (Auto Memory Recall & Interactive Graphs)

What This Is

An agentic medical chat platform with advanced capabilities:

• 🤖 Model Context Protocol (MCP) - Claude autonomously calls medical search tools

• 🧠 GraphRAG - Knowledge graph-based retrieval with entity and relationship extraction

• 🖼️ Medical Image Search - Semantic search over chest X-rays using NV-CLIP embeddings

• 💾 Agent Memory System - Persistent semantic memory with vector search

• 🏥 FHIR Integration - Full-text search of clinical documents

• ☁️ AWS Deployment - Production deployment on AWS EC2 with NVIDIA A10G GPU

• 📊 Interactive UI - Streamlit interface with execution transparency

• 🗄️ InterSystems IRIS - Vector database with native VECTOR(DOUBLE, 1024) support

Quick Start

1. Run the Streamlit Chat Interface

Visit http://localhost:8501 and start chatting!

2. Use as MCP Server (Claude Desktop, etc.)

Architecture

Features

MCP Tools (10+ available)

FHIR & GraphRAG:

1. search_fhir_documents - Full-text search of clinical notes

2. get_document_details - Retrieve complete clinical notes by ID

3. search_knowledge_graph - Search medical entities (symptoms, conditions, medications)

4. hybrid_search - Combined vector + graph search with RRF fusion

5. get_entity_statistics - Knowledge graph statistics and insights

Medical Images: 6. search_medical_images - Semantic search over chest X-rays with NV-CLIP

Agent Memory: 7. remember_information - Store semantic memories (corrections, knowledge, preferences, feedback) 8. recall_information - Semantic search over agent memories 9. get_memory_stats - Memory system statistics

Visualizations: 10. plot_symptom_frequency - Chart of most common symptoms 11. plot_entity_distribution - Entity type distribution charts 12. plot_patient_timeline - Patient encounter timeline 13. plot_entity_network - Knowledge graph relationship visualization 14. visualize_graphrag_results - Interactive GraphRAG search results

Chat Interface Features

• ✅ Multi-Modal Search - Search clinical text, medical images, and knowledge graph

• ✅ Agent Memory - Persistent semantic memory with vector search

• ✅ Medical Image Display - View chest X-rays with DICOM support

• ✅ Execution Transparency - See which tools Claude calls and its reasoning

• ✅ Interactive Charts - Generate visualizations from data

• ✅ Conversation History - Multi-turn conversations with context

• ✅ Memory Editor - Browse, search, add, and delete agent memories in sidebar

• ✅ Error Handling - Graceful handling of API issues with detailed logs

• ✅ Max Iterations Control - Prevents infinite loops (10 iteration limit)

Current Version: v2.14.0

Recent Features (v2.14.0):

• ✅ Auto Memory Recall: Memories automatically recalled before each query to guide tool selection

• ✅ Interactive Graph Viz: Force-directed, draggable graphs with streamlit-agraph

• ✅ Memory in Execution Log: See recalled memories in "Show Execution Details" pane

• ✅ NetworkX-powered graph layouts with physics simulation

Previous Updates:

• v2.13.0: Multi-LLM provider support (NIM > OpenAI > Bedrock), OneDrive backup

• v2.12.0: Agent memory system with pure IRIS vector storage

• v2.10.2: Fixed content processing errors, increased max iterations

• v2.10.0: GraphRAG multi-modal search with RRF fusion

• v2.0.0: AWS deployment with NVIDIA NIM integration

Configuration

Required Environment Variables

Config Files

• config/fhir_graphrag_config.yaml - Local development config

• config/fhir_graphrag_config.aws.yaml - AWS production config (active)

• config/aws-config.yaml - AWS infrastructure settings

Project Structure

Technology Stack

AI/ML:

• AWS Bedrock (Claude Sonnet 4.5)

• NVIDIA NV-CLIP (1024-dim multimodal embeddings)

• NVIDIA NIM (Inference Microservices)

• Model Context Protocol (MCP)

Database:

• InterSystems IRIS Community Edition (AWS EC2)

• Native VECTOR(DOUBLE, 1024) support

• VECTOR_COSINE similarity search

• Tables: ClinicalNoteVectors, MIMICCXRImages, Entities, EntityRelationships, AgentMemoryVectors

Infrastructure:

• AWS EC2 g5.xlarge (NVIDIA A10G GPU)

• Python 3.10+

• Streamlit for UI

• Docker for containerization

Key Libraries:

• intersystems-irispython - IRIS native client

• boto3 - AWS SDK

• streamlit - Chat UI

• mcp - Model Context Protocol SDK

• pydicom - DICOM medical image processing

• PIL - Image handling

Example Queries

Try these in the chat interface:

FHIR Search:

• "Find patients with chest pain"

• "Search for diabetes cases"

• "Show recent emergency visits"

GraphRAG:

• "What medications treat hypertension?"

• "Show me the relationship between conditions and procedures"

• "What are the side effects of metformin?"

Medical Images:

• "Show me chest X-rays of pneumonia"

• "Find chest X-rays showing cardiomegaly"

• "Search for lateral view chest X-rays"

Agent Memory:

• "Remember that I prefer concise clinical summaries"

• "What do you know about my preferences?"

• "Recall any corrections I've given you about medical terminology"

Hybrid Search:

• "Find treatment options for chronic pain" (combines vector + graph + image search)

Visualization:

• "Show a chart of conditions by frequency"

• "Visualize the knowledge graph for chest pain"

• "Graph the entity relationships"

Backup

The project uses OneDrive for automatic cloud backup:

Backup includes all code, configs, and medical images (~195 MB). OneDrive automatically syncs to cloud.

Development

Running Tests

Debug Mode

Enable debug logging:

AWS Deployment

The system is deployed on AWS EC2 with:

• Instance: g5.xlarge (NVIDIA A10G GPU)

• Region: us-east-1

• Database: InterSystems IRIS Community Edition

• GPU Services: NVIDIA NIM for NV-CLIP embeddings

• Data: 50 medical images, 51 clinical notes, 83 entities, 540 relationships

See docs/deployment-guide.md for detailed deployment instructions.

Troubleshooting

See docs/troubleshooting.md for common issues.

Common Issues:

• AWS credentials not configured → Set AWS_PROFILE or AWS env vars

• IRIS connection failed → Check IRIS_HOST and credentials

• NV-CLIP not responding → Check NVCLIP_BASE_URL and SSH tunnel

• Medical images not found → Verify image paths and DICOM support

• Memory search returning 0 results → Check embeddings with magnitude test

• Max iterations reached → Query may be too complex, try simplifying

Documentation

Core Documentation

• Architecture Overview - System design and data flow

• Deployment Guide - AWS deployment instructions

• Troubleshooting Guide - Common issues and solutions

Current Session Docs

• EMBEDDINGS_FIXED.md - Image and memory embeddings fix

• MEMORY_SEARCH_BROWSE_FIX.md - Memory search UI fix

• PROGRESS.md - Development history and achievements

• TODO.md - Current tasks and roadmap

Historical Documentation

• archive/ - Old implementations, scripts, and session docs

Contributing

This project is based on the FHIR-AI-Hackathon-Kit. The original tutorial content remains in the tutorial/ directory.

License

Inherits license from upstream FHIR-AI-Hackathon-Kit repository.

Acknowledgments

• Original Project: FHIR-AI-Hackathon-Kit by gabriel-ing

• InterSystems IRIS for the vector database platform

• AWS Bedrock for Claude Sonnet 4.5 access

• NVIDIA NIM for NV-CLIP multimodal embeddings

• Model Context Protocol by Anthropic

• MIMIC-CXR dataset for medical imaging data