# **Disclaimer: this project and documentation were created with active usage of AI with basic code and content review and are in the "beta" state as a part of POC project. There can be errors in documentation.**
# Insight Digger MCP
Enterprise-grade Model Context Protocol (MCP) system for data analysis with Claude Desktop integration.
## Architecture Overview
This project provides a sophisticated **3-layer MCP architecture** designed for enterprise environments:
1. **MCP Bridge** ↔ **MCP Client Flask API** (Custom HTTP REST endpoints)
2. **MCP Client Flask API** ↔ **MCP Server subprocess** (Standard MCP protocol)
3. **MCP Server** ↔ **Backend Data API** (HTTP calls to enterprise backend)
### Key Enterprise Features
- **🔐 Dynamic JWT Authentication**: 14-day JWT tokens with session management
- **🧠 Intelligent Caching**: Parameter caching and auto-injection for efficient workflows
- **📋 Workflow Guidance**: LLM-optimized tool orchestration with conversation management
- **👥 Multi-User Support**: Centralized service with session isolation
- **🏢 Enterprise Integration**: Compatible with existing authentication and monitoring systems
## Setup Options
### Option 1: Claude Desktop Integration (Recommended)
**For end users who want to use Claude Desktop with Insight Digger:**
#### 1. Install the NPX Bridge
```bash
npx @sandsiv/data-narrator-mcp
```
#### 2. Configure Claude Desktop
Add to your Claude Desktop configuration file:
**macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
**Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
```json
{
"mcpServers": {
"data-narrator": {
"command": "npx",
"args": ["-y", "@sandsiv/data-narrator-mcp@1.0.0"],
"env": {
"MCP_CLIENT_URL": "https://your-mcp-service.com"
}
}
}
}
```
**Note:** The `MCP_CLIENT_URL` environment variable is optional. By default, the system will use Sandsiv's hosted MCP service. Only provide this variable if you're deploying your own version of the data-narrator-mcp service.
#### 3. Usage in Claude Desktop
1. **Authenticate first:** Use the `setup_authentication` tool with your API URL and JWT token
2. **Start analysis:** Begin with `list_sources` to see available data
3. **Follow the workflow:** The system guides you through multi-step analysis processes
### Option 2: Direct API Integration (For developers)
**For custom integrations or testing:**
#### 1. Start the MCP Services
```bash
# Install dependencies
./scripts/setup/install_dependencies.sh
# Activate virtual environment
source venv/bin/activate
# Start the Flask API service
npm run dev:flask
# OR
python src/python/scripts/start_flask_api.py
```
#### 2. Use the REST API
```bash
# Initialize session
curl -X POST http://localhost:5000/init \
-H "Content-Type: application/json" \
-d '{"session_id": "test-session", "apiUrl": "https://your-api.com", "jwtToken": "your-jwt"}'
# Get available tools
curl -X POST http://localhost:5000/tools \
-H "Content-Type: application/json" \
-d '{"session_id": "test-session"}'
# Call a tool
curl -X POST http://localhost:5000/call-tool \
-H "Content-Type: application/json" \
-d '{"session_id": "test-session", "tool": "list_sources", "params": {}}'
```
## Development Setup
### Prerequisites
- Python 3.8+
- Node.js 18+ (for NPX bridge)
- Access to Insight Digger backend API
### Local Development
```bash
# Clone the repository
git clone <repository-url>
cd insight_digger_mcp
# Install all dependencies
./scripts/setup/install_dependencies.sh --dev
# Activate virtual environment
source venv/bin/activate
# Run tests
npm test
# OR separately:
npm run test:python
npm run test:nodejs
```
### Testing the NPX Bridge Locally
```bash
# Start your MCP client service
npm run dev:flask
# In another terminal, test the bridge
npm run dev:bridge
# Use the MCP Inspector or Claude Desktop to test
```
## Authentication Flow
### JWT Token Management
- **Lifetime**: 14 days
- **Refresh**: Through the main platform web UI (outside MCP scope)
- **Validation**: Bridge handles expired tokens by requesting re-authentication
### Session Management
- **Single Session**: One active session per bridge instance
- **Session ID**: UUID generated for each bridge startup
- **Isolation**: Multiple Claude Desktop instances use separate sessions
## Tools & Workflow
### Available Analysis Tools
The system provides LLM-optimized tools for:
- 📊 **Data Source Discovery**: `list_sources`, `get_source_structure`
- ⚙️ **Analysis Configuration**: `prepare_analysis_configuration`
- 🚀 **Execution**: `execute_analysis_from_config`
- 📈 **Results**: Interactive dashboards and summaries
### Intelligent Caching
- **Parameter Injection**: Previously fetched data automatically included in subsequent calls
- **Workflow Memory**: System remembers source selections, configurations, and analysis state
- **Efficiency**: LLM doesn't need to repeat large data structures between steps
### Error Handling
- **Authentication Errors**: Clear guidance for JWT/URL validation failures
- **Tool Errors**: Contextual error messages from backend systems
- **Session Errors**: Automatic cleanup and re-authentication prompts
## Configuration
### Environment Variables
- `MCP_CLIENT_URL`: URL of the MCP Client Flask API service
- `INSIGHT_DIGGER_API_URL`: Backend API URL (configured in MCP server layer)
### Service Configuration
The MCP Server (`mcp_server.py`) connects to your backend API using configuration provided during the `/init` call.
## Documentation
- [`docs/mcp_bridge_implementation_guide.md`](docs/mcp_bridge_implementation_guide.md) - Detailed bridge architecture
- [`docs/integration_guide.md`](docs/integration_guide.md) - Integration patterns
- [`docs/mcp_client_development_plan.md`](docs/mcp_client_development_plan.md) - Client development guide
- [`docs/mcp_server_development_plan.md`](docs/mcp_server_development_plan.md) - Server development guide
## Production Deployment
### Service Deployment
```bash
# Install as systemd service (Linux)
sudo cp data-narrator-mcp.service /etc/systemd/system/
sudo systemctl enable data-narrator-mcp
sudo systemctl start data-narrator-mcp
```
### NPX Package Publishing
```bash
# Build and publish the bridge package
npm version patch
npm publish --access public
```
### Monitoring
- Service logs: `journalctl -u data-narrator-mcp -f`
- Bridge logs: Console output in Claude Desktop
- Session tracking: All sessions logged with UUIDs
## Security & Production Readiness
✅ **Status**: Ready for external publication
🔐 **Security**: Comprehensive credential validation implemented
📊 **Performance**: Optimized with session reuse and direct validation
### Security Features
- **Immediate credential validation** during `/init` endpoint
- **Session reuse optimization** - no redundant validation calls
- **Proper HTTP status codes** (401 for auth failures, 500 for server errors)
- **Input validation** for API URLs and JWT tokens
- **Resource efficiency** - MCP servers created only for valid credentials
- **5-second timeout** for validation requests
### Security Considerations
- **JWT Tokens**: Never logged or stored permanently
- **Session Isolation**: Proper cleanup prevents cross-session data leakage
- **HTTPS Required**: All production communications must use HTTPS
- **Enterprise Auth**: Integrates with existing authentication systems
- **Immediate Auth Feedback**: Invalid credentials rejected in <5 seconds
- **Resource Protection**: No MCP instances created for invalid credentials
See [SECURITY.md](SECURITY.md) for detailed security documentation.
## Support
For issues or questions:
1. Check the documentation in the `docs/` folder
2. Review the service logs for error details
3. Verify JWT token validity and API connectivity
4. Ensure MCP Client service is running and accessible
## License
MIT License - See LICENSE file for details.
