# MCP 1.5 - Advanced Hammerspace Storage Management with AI
A production-ready Model Context Protocol (MCP) server for Hammerspace storage management with natural language interface, automatic file ingestion, and comprehensive monitoring capabilities.
## 🚀 Quick Start
### Prerequisites
- Python 3.8 or higher
- Linux system (Ubuntu 20.04+ recommended)
- Hammerspace storage cluster (HSTK CLI installed)
- Anthropic API key (for natural language UI)
- NVIDIA API key (optional, for AI features)
- Kubernetes cluster (for automatic file ingestion)
- Milvus vector database (for document embeddings)
### Installation
1. **Clone the repository**:
```bash
git clone https://github.com/mbloomhammerspace/mcp-1.5.git
cd mcp-1.5
```
2. **Create virtual environment**:
```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
3. **Configure environment variables**:
```bash
cp .env.example .env
nano .env
```
Set the following:
```bash
# Hammerspace Configuration
HS_ANVIL=10.200.120.90
# Anthropic API (for Web UI)
ANTHROPIC_API_KEY=your_anthropic_key_here
# NVIDIA API (optional)
NVIDIA_API_KEY=your_nvidia_key_here
```
4. **Start all services with one command**:
```bash
./start_all_services.sh start
```
Access Web UI at: `http://localhost:5000`
## 📋 Features
### 🤖 Natural Language Interface
- **Web-based UI** powered by Claude AI at `http://localhost:5000`
- **Direct MCP integration** for all Hammerspace operations
- **Real-time debug log viewer** at `/debug`
- **File ingest monitor dashboard** at `/monitor`
- **Action-oriented responses** - executes commands, reports results
### 📁 Advanced File Ingestion System
- **Multi-format support** - BMP, DOCX, HTML, JPEG, JSON, MD, PDF, PNG, PPTX, SH, TIFF, TXT, MP3
- **Real-time file monitoring** - Polling-based detection on NFS 4.2 mounts
- **Folder-based processing** - New folders trigger batch processing of all supported files
- **Kubernetes job deployment** - Automatic file ingestion jobs with ConfigMaps
- **Milvus vector database integration** - Automatic embedding generation and storage
- **Intelligent collection naming** - Collections named after folders or sequential numbering
- **Tag-based tier management** - Automatic tier0 promotion for embedding files
- **Event streaming** - Real-time monitoring of ingestion pipeline
### 📊 Comprehensive Monitoring UI
- **Real-time event streaming** - Watch files being ingested live
- **Event filtering** - Filter by event type, file pattern, or timestamp
- **Toast notifications** - Get notified when new files arrive
- **Status dashboard** - Monitor service health, file counts, CPU usage
- **Interactive UI** - Beautiful, modern interface for tracking ingestion pipelines
- **Multi-service monitoring** - Track all MCP servers and services
### 🔧 Persistent Service Management
- **Unified startup script** - Start all services with one command
- **Screen session persistence** - Services survive SSH disconnections
- **Auto-restart capability** - Services automatically restart if they crash
- **Systemd integration** - Optional auto-start on boot
- **Comprehensive logging** - Detailed logs for all services
- **Health monitoring** - Real-time service status and port monitoring
### API Endpoints
- **`/api/monitor/status`** - Get monitor service status
- **`/api/monitor/events`** - Query ingestion events with filters
- **`/api/monitor/events/stream`** - Server-Sent Events for real-time updates
- **`/api/chat`** - Natural language chat interface
- **`/api/tools`** - List available MCP tools
- **`/api/logs/stream`** - Stream debug logs in real-time
### Core MCP Tools
All tools use **real HSTK CLI commands** (no mock data):
- **`tag_directory_recursive`** - Tag files and directories recursively
- **`check_tagged_files_alignment`** - Check if tagged files are aligned to their objectives
- **`apply_objective_to_path`** - Apply Hammerspace objectives (e.g., "Place-on-tier0", "placeonvolumes")
- **`remove_objective_from_path`** - Remove objectives from paths
- **`list_objectives_for_path`** - List all objectives applied to a path
- **`ingest_new_files`** - Find new files by ctime/mtime, tag them, and place on Tier 1
- **`refresh_mounts`** - Refresh NFS mounts to resolve stale file handles
### Advanced Features
- **Automatic file monitoring** - Polling service that auto-tags new files with MD5 hash and MIME type
- **Automatic stale file handle recovery** - Detects and automatically refreshes mounts
- **Share-relative path support** - Use `/modelstore/dir` instead of `/mnt/se-lab/modelstore/dir`
- **Multi-tier support** - Tier 0 (high-performance) and Tier 1 (default storage)
- **Tag-based workflows** - Tag files and manage them as collections
- **Intelligent batching** - Groups file events (15 sec) or processes immediately if traffic is light
## 🆕 Advanced File Ingestion Workflow
### How It Works
1. **File Detection**: The file monitor polls NFS mounts every 5 seconds for new files
2. **Tagging**: New files are automatically tagged with:
- `user.ingestid=<md5hash>` - MD5 hash for deduplication
- `user.mimeid=<mimetype>` - MIME type for classification
- `user.embedding` - Tag for files requiring embedding
3. **Multi-format Processing**: All supported file types trigger Kubernetes ingestion jobs
4. **Collection Management**: Files are organized into Milvus collections:
- **Folder-based**: Collections named after folder (e.g., `cold_0011`)
- **Sequential**: Collections named `intel_1`, `intel_2`, etc.
5. **Tier Management**: Files tagged for embedding are automatically promoted to tier0
6. **Embedding Generation**: Files are processed into vector embeddings
7. **Storage**: Embeddings stored in Milvus for semantic search
8. **Tier Demotion**: After embedding, files are moved back to default tier
### Folder Processing
When a new folder is detected:
- All supported files in the folder are processed as a batch
- A single Milvus collection is created named after the folder
- Collection names are sanitized (e.g., `cold-0011` → `cold_0011`)
- All files in the folder are uploaded to the same collection
- Files are tagged with `embedding` and promoted to tier0 for processing
- After embedding, tier0 objective is removed
### Configuration
The file monitor supports:
- **Polling intervals**: 5 seconds (fast) during business hours, 30 seconds during retroactive hours
- **Retroactive tagging**: Configurable time windows (default: disabled for testing)
- **Path monitoring**: Monitors `/mnt/anvil/hub/` by default
- **File filtering**: Processes all supported file types (BMP, DOCX, HTML, JPEG, JSON, MD, PDF, PNG, PPTX, SH, TIFF, TXT, MP3)
- **Recursive folder tagging**: Tags entire folder hierarchies for 40x performance improvement
### Kubernetes Integration
- **Job Templates**: Uses `k8s-templates/ingest.yaml` for PDF processing
- **ConfigMaps**: File lists passed to jobs via Kubernetes ConfigMaps
- **PVC Integration**: Uses `hammerspace-hub-pvc` for persistent storage
- **Path Mapping**: Container paths mapped from `/mnt/anvil/hub/` to `/data/`
## 🎯 Usage
### Example Commands (Web UI)
Try these in the natural language interface:
```
Tag all files in /modelstore/nvidia-test-thurs as modelsetid=my-demo
Promote all modelsetid=my-demo tagged files to tier0
Check alignment status of files tagged with modelsetid=my-demo
Remove the Place-on-tier0 objective from /modelstore/nvidia-test-thurs
List all objectives for /modelstore/incoming-models
Refresh Hammerspace mounts
```
### 🆕 File Ingestion Commands
```
Start the file monitor daemon
Check file monitor status
View recent file ingestion events
Stop the file monitor
```
### Direct MCP Server Usage
Add to your MCP client configuration (e.g., Cursor IDE):
```json
{
"mcpServers": {
"hammerspace": {
"command": "python",
"args": ["/home/mike/mcp-1.5/src/aiq_hstk_mcp_server.py"],
"env": {
"PYTHONPATH": "/home/mike/mcp-1.5"
}
}
}
}
```
### Available MCP Tools
#### Tag Management
- **`tag_directory_recursive`** - Recursively tag all files in a directory
```json
{
"path": "/mnt/se-lab/modelstore/my-models",
"tag_name": "user.modelsetid",
"tag_value": "my-tag-value"
}
```
- **`check_tagged_files_alignment`** - Find and check alignment of tagged files
```json
{
"tag_name": "user.modelsetid",
"tag_value": "my-tag-value",
"share_path": "/mnt/se-lab/modelstore/"
}
```
#### Tier Management
- **`apply_objective_to_path`** - Apply objectives like tier promotion
```json
{
"objective_name": "Place-on-tier0",
"path": "/mnt/se-lab/modelstore/my-models"
}
```
- **`remove_objective_from_path`** - Remove objectives
```json
{
"objective_name": "Place-on-tier0",
"path": "/mnt/se-lab/modelstore/my-models"
}
```
#### File Ingestion
- **`ingest_new_files`** - Find new files, tag them, and place on Tier 1
```json
{
"path": "/mnt/se-lab/modelstore/",
"tag_name": "user.modelsetid",
"tag_value": "new-batch",
"age_minutes": 60
}
```
#### System Utilities
- **`refresh_mounts`** - Refresh NFS mounts to resolve stale file handles
```json
{
"mount_type": "all"
}
```
#### File Monitoring & Ingestion
- **`start_inotify_monitor`** - Start automated file monitoring service
```json
{}
```
Monitors all Hammerspace shares, automatically tags new files with:
- `user.ingestid=<md5hash>` - MD5 hash for deduplication
- `user.mimeid=<mimetype>` - MIME type for classification
Events are batched every 15 seconds or processed immediately if traffic is light.
- **`get_file_monitor_status`** - Get monitor status
```json
{}
```
Returns: running state, watched paths, pending events, files tagged, CPU usage
- **`get_file_ingest_events`** - Query file ingest/tagging events
```json
{
"limit": 100,
"event_type": "NEW_FILE",
"file_pattern": ".pdf",
"since_timestamp": "2025-10-09T20:00:00"
}
```
Returns recent file ingestion events with full metadata including:
- Event type (NEW_FILE or RETROACTIVE_TAG)
- File path and name
- MD5 hash (ingestid)
- MIME type (mimeid)
- File size and timestamp
**Use Cases:**
- Track which files have been ingested
- Build automated workflows based on file events
- Monitor ingestion pipelines
- Audit file processing
- **`stop_inotify_monitor`** - Stop monitoring service
```json
{}
```
- **`list_objectives_for_path`** - List all objectives for a path
```json
{
"path": "/mnt/se-lab/modelstore/my-models"
}
```
## 🔧 Architecture
### Components
1. **MCP Server** (`src/aiq_hstk_mcp_server.py`)
- Implements MCP protocol
- Executes Hammerspace HSTK CLI commands
- Handles automatic retry on stale file handles
- Uses real HSTK operations (no mock data)
2. **Web UI** (`web_ui/app.py`)
- Flask-based natural language interface
- Anthropic Claude integration for NL understanding
- MCP client for tool execution
- Real-time debug log streaming
3. **🆕 File Monitor** (`src/file_monitor.py`)
- Polling-based file detection for NFS 4.2 mounts
- Automatic tagging with MD5 and MIME type
- Folder-based batch processing
- Kubernetes job deployment for PDF ingestion
- Milvus collection management
4. **🆕 File Monitor Daemon** (`src/file_monitor_daemon.py`)
- Standalone daemon for running file monitor
- Background service for continuous monitoring
- Logging and error handling
5. **Mount Refresh Script** (`refresh_mounts.sh`)
- Unmounts and remounts Hammerspace NFS shares
- Resolves stale file handle errors
- Supports selective mount refresh
### Data Flow
```
User Input (NL) → Claude AI → MCP Tool Selection → HSTK CLI → Hammerspace
↓
MCP Server
↓
Real Operations
```
### 🆕 File Ingestion Data Flow
```
New File/Folder → File Monitor → Tagging → Kubernetes Job → PDF Processing → Milvus Collection
↓
Event Logging → Web UI Dashboard → Real-time Monitoring
```
## 🧪 Testing
### Web UI Tests
```bash
cd tests
./test_web_ui_curl.sh
python test_web_ui.py
```
### Single Command Test
```bash
cd tests
./test_single_question.sh "Tag all files in /modelstore/test as modelsetid=test123"
```
### 🆕 File Ingestion Tests
```bash
# Start file monitor daemon
python3 src/file_monitor_daemon.py
# Check monitor status
curl http://localhost:5000/api/monitor/status
# View ingestion events
curl http://localhost:5000/api/monitor/events
# Test with new files
cp test.pdf /mnt/anvil/hub/
```
## 🛠️ Service Management
### Unified Service Management
```bash
# Start all services with one command
./start_all_services.sh start
# Check service status
./start_all_services.sh status
# Stop all services
./start_all_services.sh stop
# Restart all services
./start_all_services.sh restart
```
### Individual Service Management
```bash
# View logs for specific services
./start_all_services.sh logs web-ui
./start_all_services.sh logs file-monitor
./start_all_services.sh logs hammerspace-mcp
./start_all_services.sh logs milvus-mcp
# Attach to service screen sessions
./start_all_services.sh attach web-ui
./start_all_services.sh attach file-monitor
# Clean up old logs
./start_all_services.sh cleanup
```
### Service URLs
Once started, services are available at:
- **Web UI**: http://localhost:5000
- **Web UI (LAN)**: http://10.0.0.236:5000
- **Hammerspace MCP**: stdio-based (no HTTP endpoint)
- **Milvus MCP**: http://localhost:9902/sse (if Milvus is running)
- **Kubernetes MCP**: http://localhost:9903/sse (not implemented)
### View Logs
```bash
# Web UI logs
tail -f logs/web_ui.log
# MCP Server logs
tail -f logs/aiq_hstk_mcp.log
# File monitor logs
tail -f logs/retroactive_fully_disabled.log
# Real-time streaming (in browser)
# Visit http://localhost:5000/debug
```
## 🐛 Troubleshooting
### Stale File Handle Errors
The MCP server automatically detects and retries operations when stale file handles occur. If issues persist:
```bash
# Manual mount refresh
./refresh_mounts.sh
# Or via Web UI
"Refresh Hammerspace mounts"
```
### Tag Operations Not Working
1. Ensure you're targeting a mounted directory
2. Verify the HSTK CLI is accessible: `/home/mike/hs-mcp-1.0/.venv/bin/hs --version`
3. Check mounts: `mount | grep hammerspace`
### 🆕 File Ingestion Issues
1. **Files not detected**: Check if file monitor is running
```bash
ps aux | grep file_monitor_daemon
```
2. **Kubernetes jobs failing**: Check pod status
```bash
kubectl get pods -l app=pdf-ingest
kubectl logs -l app=pdf-ingest
```
3. **Empty Milvus collections**: Check ingest service logs
```bash
kubectl logs -l app=ingestor-server
```
4. **NFS mount issues**: Verify mount points
```bash
mount | grep anvil
ls -la /mnt/anvil/hub/
```
### Common Tag Formats
Tags in Hammerspace use the format: `namespace.key=value`
Examples:
- `user.modelsetid=my-demo`
- `user.project=gtc-2025`
- `user.tier=critical`
## 📚 Documentation
### **🚀 Quick Start**
- **[Quick Reference Guide](QUICK_REFERENCE.md)** - Your go-to guide for common operations
- **[Service Management](SERVICE_MANAGEMENT.md)** - Complete service management guide
### **🏗️ Architecture & API**
- **[Architecture Documentation](ARCHITECTURE.md)** - Comprehensive system architecture
- **[API Documentation](API_DOCUMENTATION.md)** - Complete API reference for all MCP servers
### **🔗 Integration Guides**
- **[Integration Guide](docs/INTEGRATION_GUIDE.md)** - Connect to Cursor, Windsurf, NVIDIA Playground
### **📖 Feature Guides**
- [Complete Feature Documentation](docs/VOLUME_CANVAS_FEATURES.md)
- [Tag Search Guide](docs/TAG_SEARCH_GUIDE.md)
- [Tier Management Guide](docs/TIER_MANAGEMENT_GUIDE.md)
- [Tag to Objectives Workflow](docs/TAG_TO_OBJECTIVES_GUIDE.md)
- [Guides Index](docs/GUIDES_INDEX.md)
### **🧪 Testing Guides**
- [Web UI Testing Guide](web_ui/TESTING_GUIDE.md)
## 🔒 Production Deployment
### Security Considerations
- Run the Web UI behind a reverse proxy (nginx, Apache)
- Use HTTPS for production deployments
- Restrict access with authentication
- Review and limit MCP tool permissions
### Performance Tuning
- Adjust `max_files_to_check` for large filesystems (default: 500)
- Use specific `share_path` parameters to limit search scope
- Monitor log files for performance issues
## 📞 Support
For support and questions:
- Create an issue in the [GitHub repository](https://github.com/mbloomhammerspace/mcp-1.5)
- Check the documentation in the `docs/` folder
- Review the troubleshooting section above
## 🎉 Recent Updates
### 🆕 October 23, 2025 - Advanced Service Management + Multi-Format Support
- ✅ **NEW**: Unified service management script (`start_all_services.sh`) for all MCP services
- ✅ **NEW**: Screen session persistence - services survive SSH disconnections
- ✅ **NEW**: Auto-restart capability for crashed services
- ✅ **NEW**: Systemd integration for auto-start on boot
- ✅ **NEW**: Multi-format file support (BMP, DOCX, HTML, JPEG, JSON, MD, PDF, PNG, PPTX, SH, TIFF, TXT, MP3)
- ✅ **NEW**: Tag-based tier management with automatic tier0 promotion/demotion
- ✅ **NEW**: Recursive folder tagging for 40x performance improvement
- ✅ **NEW**: Enhanced event filtering and monitoring UI
- ✅ **NEW**: Comprehensive health monitoring and port conflict resolution
- ✅ **FIXED**: Unicode handling in logs and CLI output
- ✅ **FIXED**: NFS timing issues with retry mechanisms
- ✅ **FIXED**: Hammerspace CLI tag operations with fallback methods
- ✅ **FIXED**: Individual file vs folder-level tagging optimization
### 🆕 October 16, 2025 - Automatic File Ingestion + Kubernetes + MCP
- ✅ **NEW**: Complete automatic file ingestion system with Kubernetes integration
- ✅ **NEW**: Polling-based file monitoring for NFS 4.2 mounts (inotify not supported)
- ✅ **NEW**: Folder-based batch processing - new folders trigger collection creation
- ✅ **NEW**: Milvus vector database integration for PDF embeddings
- ✅ **NEW**: Kubernetes job deployment with ConfigMaps for file processing
- ✅ **NEW**: Intelligent collection naming (folder-based and sequential)
- ✅ **NEW**: Real-time event streaming and monitoring dashboard
- ✅ **NEW**: Standalone file monitor daemon for continuous operation
- ✅ **FIXED**: NFS compatibility issues with polling-only approach
- ✅ **FIXED**: Kubernetes PVC integration and path mapping
- ✅ **FIXED**: Milvus collection naming conventions (underscores vs hyphens)
- ✅ **FIXED**: File path handling in container environments
- ✅ **FIXED**: Job completion tracking and error handling
- ✅ **FIXED**: Retroactive tagging time window configuration
- ✅ **FIXED**: Threading and async operation coordination
### October 9, 2025 - File Ingestion Event System & Monitoring UI
- ✅ **NEW**: Agentic event consumption via `get_file_ingest_events` MCP tool
- ✅ **NEW**: Real-time monitoring dashboard at `/monitor` with live event streaming
- ✅ **NEW**: API endpoints for event querying and monitoring
- ✅ **FIXED**: Duplicate file processing - files now tagged exactly once
- ✅ **FIXED**: In-memory tracking prevents repeated tagging of same files
- ✅ Deduplication verification tools: `check_duplicates.sh` and `find-dup.sh`
- ✅ Full event metadata: timestamp, file path, MD5 hash, MIME type, file size
- ✅ Event filtering: by type, file pattern, or timestamp
- ✅ Toast notifications for new file arrivals
- ✅ Automated file monitoring service with MD5/MIME tagging
### Previous Updates
- ✅ Added automatic mount refresh on stale file handle errors
- ✅ Suppressed Anthropic API deprecation warnings in logs
- ✅ Added copy buttons to all chat messages
- ✅ Fixed PARTIALLY ALIGNED status detection
- ✅ Web-based natural language console with Claude AI
---
**MCP 1.5** - Production-ready Hammerspace storage management with natural language interface and automatic file ingestion.
