# MCP Brain Service Documentation
## Overview
The MCP Brain Service is a pure infrastructure service for storing and retrieving documents with semantic search, graph relationships, and hybrid search capabilities.
**Key Principle**: Brain service is infrastructure, not application logic. Apps prepare data, brain service stores and retrieves.
## Quick Links
### 🆕 New in v1.1.0
- **[How to Use](how-to-use.md)** - Complete API usage guide with all 10 endpoints
- **[Batch Endpoints Guide](BATCH_ENDPOINTS_GUIDE.md)** - Detailed guide for new batch endpoints
- **[Implementation Summary](IMPLEMENTATION_SUMMARY.md)** - Technical implementation details
- **[Deployment Guide](DEPLOYMENT_GUIDE.md)** - Deployment instructions
- **[Changelog](../CHANGELOG.md)** - Version history and changes
### Architecture & Design
- [Architecture Decision](./architecture-decision.md) - Why data preparation belongs in the app
- [Retriv Integration Plan](./retriv-integration-plan.md) - How to enhance queries with hybrid search
- [API Contracts](./api-contracts.md) - Storage and query API specifications
### Implementation
- [Implementation Checklist](./implementation-checklist.md) - Step-by-step guide to add Retriv
### Operations
- [QUICKSTART](../QUICKSTART.md) - Get started quickly
- [PRODUCTION](../PRODUCTION.md) - Production deployment guide
- [WARP](../WARP.md) - Development commands and tips
## What is MCP Brain Service?
A Python-based service that provides:
1. **Storage**: Store documents with embeddings and relationships
2. **Semantic Search**: Find similar documents using Jina embeddings
3. **Hybrid Search**: Combine keyword (BM25) + semantic search with Retriv
4. **Graph Queries**: Navigate relationships using Neo4j
5. **MCP Protocol**: WebSocket-based tool interface for agents
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ Application (Next.js) │
│ │
│ - Prepares data (enrichment, transformation) │
│ - Calls brain service for storage/retrieval │
└─────────────────────────────────────────────────────────────┘
↓ ↑
(store API) (query API)
↓ ↑
┌─────────────────────────────────────────────────────────────┐
│ Brain Service (brain.ft.tc) │
│ Pure Storage & Retrieval Infrastructure │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ Store API │ │ Query API │ │
│ │ - Store documents │ │ - Hybrid search │ │
│ │ - Batch storage │ │ - Semantic search │ │
│ └────────────────────┘ │ - Graph queries │ │
│ └────────────────────┘ │
│ ↓ ↑ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Storage & Retrieval Services │ │
│ │ - RetrivService (hybrid search) │ │
│ │ - Neo4jService (graph storage) │ │
│ │ - JinaService (embeddings) │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
```
## Key Concepts
### 1. Pure Infrastructure
Brain service is like a database - it doesn't understand your business logic:
```
✅ Good: App prepares character data → Brain service stores
❌ Bad: Brain service understands PayloadCMS structure
```
### 2. Document Format
All documents follow a standard format:
```typescript
{
id: string // Unique identifier
type: string // Document type (character, scene, etc.)
project_id: string // Project isolation
text: string // Searchable text (prepared by app)
metadata: object // Structured data (prepared by app)
relationships: array // Graph relationships (prepared by app)
}
```
### 3. Hybrid Search
Retriv combines two search methods:
- **BM25 (Keyword)**: Matches exact terms like "vest", "scene 3"
- **Embeddings (Semantic)**: Understands meaning like "clothing", "appearance"
- **Combined**: Better results than either alone
### 4. Project Isolation
All data is isolated by `project_id` - different projects don't see each other's data.
## Getting Started
### For Brain Service Development
1. Read [Retriv Integration Plan](./retriv-integration-plan.md)
2. Follow [Implementation Checklist](./implementation-checklist.md)
3. Run tests: `pytest`
4. Deploy: See [PRODUCTION.md](../PRODUCTION.md)
### For App Integration
1. Read [Architecture Decision](./architecture-decision.md)
2. Review [API Contracts](./api-contracts.md)
3. Implement data preparation in your app
4. Call brain service storage/query APIs
## API Overview
### Storage
```bash
# Store single document
POST /store
{
"id": "char_aladdin_proj123",
"type": "character",
"project_id": "proj123",
"text": "Aladdin: A street-smart young man...",
"metadata": {...},
"relationships": [...]
}
# Store multiple documents
POST /store/batch
{
"documents": [...]
}
```
### Query
```bash
# Hybrid search (recommended)
POST /query
{
"project_id": "proj123",
"query": "What does Aladdin wear in scene 3?",
"search_type": "hybrid",
"top_k": 5
}
# Character context
POST /query/character-context
{
"project_id": "proj123",
"character_name": "Aladdin",
"scene_number": 3
}
```
See [API Contracts](./api-contracts.md) for full details.
## MCP Tools
Brain service also exposes WebSocket-based MCP tools:
- `store_document` - Store a document
- `store_batch` - Store multiple documents
- `search` - Hybrid search
- `find_similar` - Find similar documents
- `get_relationships` - Get document relationships
## Technology Stack
- **FastAPI**: Web framework with WebSocket support
- **Neo4j**: Graph database for relationships
- **Jina AI**: Embedding generation
- **Retriv**: Hybrid search (BM25 + embeddings)
- **Pydantic**: Data validation
- **Pytest**: Testing framework
## Development Workflow
1. **Make changes** to brain service code
2. **Run tests**: `pytest`
3. **Test locally**: `python src/main.py`
4. **Deploy**: Follow deployment guide
## Testing
```bash
# Run all tests
pytest
# Run specific test type
pytest tests/unit/
pytest tests/integration/
pytest tests/performance/
# Run with coverage
pytest --cov=src
```
## Deployment
See [PRODUCTION.md](../PRODUCTION.md) for deployment instructions.
Quick deploy to Coolify:
1. Push to main branch
2. Coolify auto-deploys
3. Verify health: `curl https://brain.ft.tc/health`
## Monitoring
### Health Check
```bash
curl https://brain.ft.tc/health
```
Response:
```json
{
"status": "healthy",
"services": {
"neo4j": "connected",
"jina": "connected",
"retriv": "initialized"
}
}
```
### Logs
```bash
# View logs in Coolify dashboard
# Or SSH to server and check Docker logs
docker logs mcp-brain-service
```
## Troubleshooting
### Retriv Not Initializing
```bash
# Check if Retriv package installed
pip list | grep retriv
# Check data directory permissions
ls -la data/retriv_index/
# Check logs for errors
docker logs mcp-brain-service | grep -i retriv
```
### Slow Queries
```bash
# Check Retriv index size
du -sh data/retriv_index/
# Monitor query performance
# Add logging in retriv_service.py
```
### Neo4j Connection Issues
```bash
# Check Neo4j connection
curl http://neo4j.ft.tc:7474
# Verify credentials in .env
echo $NEO4J_URI
echo $NEO4J_USER
```
## Contributing
1. Create feature branch
2. Make changes
3. Add tests
4. Update documentation
5. Submit PR
## Support
- **Issues**: Create GitHub issue
- **Questions**: Check documentation first
- **Urgent**: Contact team lead
## Roadmap
### ✅ Completed (v1.1.0) - January 2025
- ✅ Neo4j storage
- ✅ Jina embeddings v4
- ✅ MCP protocol
- ✅ REST API routes (10 endpoints)
- ✅ Authentication (API key)
- ✅ Batch node creation (up to 50 nodes)
- ✅ Semantic duplicate detection
- ✅ AI-powered department context aggregation (OpenRouter LLM)
- ✅ Content coverage analysis with LLM
- ✅ Comprehensive documentation
### In Progress (v1.2)
- 🚧 Retriv hybrid search
- 🚧 Rate limiting
- 🚧 Prometheus metrics
- 🚧 Redis caching layer
### Future (v2.0)
- [ ] Multi-modal search (images + text)
- [ ] Real-time updates via WebSocket
- [ ] Advanced analytics dashboard
- [ ] Multi-tenancy support
- [ ] GraphQL API
## License
[Your License Here]
## Contact
[Your Contact Info Here]
