MCP Memory Service

# Cloudflare Backend Setup Guide ## Overview The MCP Memory Service supports native Cloudflare integration using Vectorize for vector storage, D1 for metadata, and optional R2 for large content. This provides: - **Vectorize**: Vector database for semantic search (768-dimensional embeddings) - **D1**: SQLite database for metadata storage - **Workers AI**: Embedding generation (@cf/baai/bge-base-en-v1.5) - **R2** (optional): Object storage for large content This setup provides global distribution, automatic scaling, and cost-effective pay-per-use pricing. ## 🚀 Quick Start For users who want to get started immediately: ### Prerequisites 1. **Cloudflare Account**: You need a Cloudflare account with Workers/D1/Vectorize access 2. **API Token**: Create an API token with these permissions: - **Vectorize Edit** (for creating and managing vector indexes) - **D1 Edit** (for creating and managing databases) - **R2 Edit** (optional, for large content storage) - **Workers AI Read** (for embedding generation) ### Quick Setup Commands ```bash # 1. Install dependencies pip install httpx>=0.24.0 # 2. Create Cloudflare resources (requires wrangler CLI) wrangler vectorize create mcp-memory-index --dimensions=768 --metric=cosine wrangler d1 create mcp-memory-db wrangler r2 bucket create mcp-memory-content # Optional # 3. Configure environment export MCP_MEMORY_STORAGE_BACKEND=cloudflare export CLOUDFLARE_API_TOKEN="your-api-token" export CLOUDFLARE_ACCOUNT_ID="your-account-id" export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index" export CLOUDFLARE_D1_DATABASE_ID="your-d1-database-id" export CLOUDFLARE_R2_BUCKET="mcp-memory-content" # Optional # 4. Test and start python -m src.mcp_memory_service.server # Alternative startup methods: # uv run memory server # Modern CLI (recommended) # python scripts/run_memory_server.py # Direct script execution ``` > **⚠️ Important**: Cloudflare backend uses Workers AI for embedding generation, so do NOT use `scripts/memory_offline.py` which sets offline mode. Use the standard startup methods above instead. ## Prerequisites 1. **Cloudflare Account**: Sign up at [cloudflare.com](https://www.cloudflare.com/) 2. **Cloudflare Services**: Access to Vectorize, D1, and optionally R2 3. **API Token**: With appropriate permissions ## Step 1: Create Cloudflare Resources ### 1.1 Create Vectorize Index ```bash # Install Wrangler CLI npm install -g wrangler # Login to Cloudflare wrangler login # Create Vectorize index (768 dimensions for BGE embeddings) wrangler vectorize create mcp-memory-index --dimensions=768 --metric=cosine ``` ### 1.2 Create D1 Database ```bash # Create D1 database wrangler d1 create mcp-memory-db # Note the database ID from the output ``` ### 1.3 Create R2 Bucket (Optional) ```bash # Create R2 bucket for large content storage wrangler r2 bucket create mcp-memory-content ``` ## Step 2: Configure API Token ### 2.1 Create API Token 1. Go to [Cloudflare Dashboard → My Profile → API Tokens](https://dash.cloudflare.com/profile/api-tokens) 2. Click "Create Token" 3. Use "Custom Token" template 4. Configure permissions: - **Account**: `Read` (to access account resources) - **Vectorize**: `Edit` (to manage vector operations) - **D1**: `Edit` (to manage database operations) - **R2**: `Edit` (if using R2 for large content) - **Workers AI**: `Read` (for embedding generation) ### 2.2 Get Account ID 1. Go to [Cloudflare Dashboard](https://dash.cloudflare.com/) 2. Select your domain or go to overview 3. Copy the Account ID from the right sidebar ### 2.3 Manual Resource Creation (Alternative) If you prefer manual creation via the Cloudflare Dashboard or encounter authentication issues: **Create Vectorize Index via Dashboard:** 1. Go to [Cloudflare Dashboard → Vectorize](https://dash.cloudflare.com/vectorize) 2. Click "Create Index" 3. Name: `mcp-memory-index` 4. Dimensions: `768` 5. Metric: `cosine` **Create D1 Database via Dashboard:** 1. Go to [Cloudflare Dashboard → D1](https://dash.cloudflare.com/d1) 2. Click "Create Database" 3. Name: `mcp-memory-db` 4. Copy the Database ID from the overview page **Create R2 Bucket via Dashboard (Optional):** 1. Go to [Cloudflare Dashboard → R2](https://dash.cloudflare.com/r2) 2. Click "Create Bucket" 3. Name: `mcp-memory-content` 4. Choose region closest to your location **Alternative API Creation:** ```bash # Create Vectorize index via API curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/vectorize/indexes" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "mcp-memory-index", "config": { "dimensions": 768, "metric": "cosine" } }' # Create D1 database via API curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/d1/database" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "mcp-memory-db" }' # Create R2 bucket via API (optional) curl -X POST "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/r2/buckets" \ -H "Authorization: Bearer YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "mcp-memory-content" }' ``` ## Step 3: Configure Environment Variables Set the following environment variables: ```bash # Required Configuration export MCP_MEMORY_STORAGE_BACKEND=cloudflare export CLOUDFLARE_API_TOKEN="your-api-token-here" export CLOUDFLARE_ACCOUNT_ID="your-account-id-here" export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index" export CLOUDFLARE_D1_DATABASE_ID="your-d1-database-id" # Optional Configuration export CLOUDFLARE_R2_BUCKET="mcp-memory-content" # For large content export CLOUDFLARE_EMBEDDING_MODEL="@cf/baai/bge-base-en-v1.5" # Default export CLOUDFLARE_LARGE_CONTENT_THRESHOLD="1048576" # 1MB threshold export CLOUDFLARE_MAX_RETRIES="3" # API retry attempts export CLOUDFLARE_BASE_DELAY="1.0" # Retry delay in seconds ``` ### Configuration File Example Create a `.env` file in your project root: ```env # Cloudflare Backend Configuration MCP_MEMORY_STORAGE_BACKEND=cloudflare # Required Cloudflare Settings CLOUDFLARE_API_TOKEN=your-api-token-here CLOUDFLARE_ACCOUNT_ID=your-account-id-here CLOUDFLARE_VECTORIZE_INDEX=mcp-memory-index CLOUDFLARE_D1_DATABASE_ID=your-d1-database-id # Optional Settings CLOUDFLARE_R2_BUCKET=mcp-memory-content CLOUDFLARE_EMBEDDING_MODEL=@cf/baai/bge-base-en-v1.5 CLOUDFLARE_LARGE_CONTENT_THRESHOLD=1048576 CLOUDFLARE_MAX_RETRIES=3 CLOUDFLARE_BASE_DELAY=1.0 # Logging LOG_LEVEL=INFO ``` ## Step 4: Install Dependencies The Cloudflare backend requires additional dependencies: ```bash # Install additional requirements pip install -r requirements-cloudflare.txt # Or install manually pip install httpx>=0.24.0 ``` ## Step 5: Initialize and Test ### 5.1 Start the Service ```bash # Start MCP Memory Service with Cloudflare backend python -m src.mcp_memory_service.server ``` ### 5.2 Verify Configuration The service will automatically: 1. Initialize the D1 database schema 2. Verify access to the Vectorize index 3. Check R2 bucket access (if configured) Look for these success messages in the logs: ``` INFO:mcp_memory_service.config:Using Cloudflare backend with: INFO:mcp_memory_service.config: Vectorize Index: mcp-memory-index INFO:mcp_memory_service.config: D1 Database: your-d1-database-id INFO:mcp_memory_service.server:Created Cloudflare storage with Vectorize index: mcp-memory-index INFO:mcp_memory_service.storage.cloudflare:Cloudflare storage backend initialized successfully ``` ### 5.3 Test Basic Operations **Option A: Comprehensive Test Suite** ```bash # Run comprehensive automated tests python scripts/test_cloudflare_backend.py ``` **Option B: Manual API Testing** ```bash # Store a test memory curl -X POST http://localhost:8000/api/memories \ -H "Content-Type: application/json" \ -d '{ "content": "This is a test memory for Cloudflare backend", "tags": ["test", "cloudflare"] }' # Search memories curl -X POST http://localhost:8000/api/memories/search \ -H "Content-Type: application/json" \ -d '{ "query": "test memory", "n_results": 5 }' # Get statistics curl http://localhost:8000/api/stats ``` **Option C: Automated Resource Setup** ```bash # Set up Cloudflare resources automatically python scripts/setup_cloudflare_resources.py ``` ## Architecture Details ### Data Flow 1. **Content Storage**: - Small content (<1MB): Stored directly in D1 - Large content (>1MB): Stored in R2, referenced in D1 2. **Vector Processing**: - Content → Workers AI → Embedding Vector - Vector stored in Vectorize with metadata - Semantic search via Vectorize similarity 3. **Metadata Management**: - Memory metadata stored in D1 SQLite - Tags stored in relational tables - Full ACID compliance for data integrity ### Performance Optimizations - **Connection Pooling**: Reused HTTP connections - **Embedding Caching**: 1000-entry LRU cache - **Batch Operations**: Bulk vector operations - **Smart Retries**: Exponential backoff for rate limits - **Async Operations**: Non-blocking I/O throughout ### Security Features - **API Key Security**: Never logged or exposed - **Input Validation**: SQL injection prevention - **Rate Limiting**: Built-in protection - **Secure Headers**: Proper HTTP security ## Migration from Other Backends ### From SQLite-vec ```bash # Export existing data python scripts/export_sqlite_vec.py --output cloudflare_export.json # Switch to Cloudflare backend export MCP_MEMORY_STORAGE_BACKEND=cloudflare # Import data python scripts/import_to_cloudflare.py --input cloudflare_export.json ``` ### From ChromaDB ```bash # Export ChromaDB data python scripts/export_chroma.py --output cloudflare_export.json # Switch to Cloudflare backend export MCP_MEMORY_STORAGE_BACKEND=cloudflare # Import data python scripts/import_to_cloudflare.py --input cloudflare_export.json ``` ## Troubleshooting ### Common Issues #### 1. Authentication Errors (401) ``` ERROR: Missing required environment variables for Cloudflare backend: CLOUDFLARE_API_TOKEN ERROR: Unauthorized - Invalid API token ``` **Solution**: - Verify all required environment variables are set - Check API token has correct permissions (Vectorize:Edit, D1:Edit, Workers AI:Read) - Ensure token is not expired - Verify account ID is correct #### 2. Resource Not Found (404) ``` ValueError: Vectorize index 'mcp-memory-index' not found ValueError: D1 database not found ``` **Solution**: - Create the Vectorize index or verify the index name is correct - Check that resources were created in the correct account - Confirm resource IDs/names match exactly - Verify resource names match exactly #### 3. Vector Storage Errors (400) ``` ValueError: Failed to store vector data HTTP 400: Invalid vector data format ``` **Solution**: - Check vector dimensions (must be 768) - Verify NDJSON format for vector data - Ensure metadata values are properly serialized - Validate input data types #### 4. D1 Database Access Issues ``` ValueError: Failed to initialize D1 schema HTTP 403: Insufficient permissions ``` **Solution**: - Verify D1 database ID and API token permissions - Ensure database exists and is accessible - Check API token has D1:Edit permissions #### 5. API Rate Limits (429) ``` Rate limited after 3 retries HTTP 429: Too Many Requests ``` **Solution**: - Increase `CLOUDFLARE_MAX_RETRIES` or `CLOUDFLARE_BASE_DELAY` for more conservative retry behavior - Implement exponential backoff (already included) - Monitor API usage through Cloudflare dashboard - Consider implementing request caching for high-volume usage ### Debug Mode Enable detailed logging: ```bash export LOG_LEVEL=DEBUG python -m src.mcp_memory_service.server --debug ``` ### Health Check ```bash # Check backend health curl http://localhost:8000/api/health # Get detailed statistics curl http://localhost:8000/api/stats ``` ## Limitations ### Current Limitations - **Embedding Model**: Fixed to Workers AI BGE model (768 dimensions) - **Content Size**: R2 storage recommended for content >1MB - **Rate Limits**: Subject to Cloudflare service limits - **Region**: Embedding generation uses Cloudflare's global network ### Planned Improvements - **Local Embedding Fallback**: For offline or restricted environments - **Custom Embedding Models**: Support for other embedding models - **Enhanced Caching**: Multi-level caching strategy - **Batch Import Tools**: Efficient migration utilities ## 🔄 Multi-Machine Bidirectional Sync **New in v6.13.7**: Cloudflare backend now supports seamless bidirectional sync between multiple machines, making it ideal for distributed teams or as a replacement for failed centralized servers. ### Use Cases 1. **Failed Server Recovery**: Replace a failed narrowbox/central server with Cloudflare 2. **Multi-Machine Development**: Sync memories across multiple development machines 3. **Team Collaboration**: Share memory context across team members 4. **Backup Strategy**: Cloudflare as primary with local sqlite_vec as backup ### Architecture ``` ┌─────────────────┐ ┌─────────────────┐ │ Machine A │ │ Machine B │ │ │ │ │ │ Claude Desktop │ │ Claude Desktop │ │ ↕ │ │ ↕ │ │ sqlite_vec │ │ sqlite_vec │ │ (backup) │ │ (backup) │ └─────────┬───────┘ └─────────┬───────┘ │ │ └─────────┬────────────┘ ↕ ┌─────────────────┐ │ Cloudflare │ │ │ │ D1 Database │ │ Vectorize Index │ │ Workers AI │ └─────────────────┘ ``` ### Setup Process #### 1. Initial Migration (from failed server) ```bash # Export memories from existing machine memory export /path/to/export.json # Set up Cloudflare environment export CLOUDFLARE_API_TOKEN="your-token" export CLOUDFLARE_ACCOUNT_ID="your-account" export CLOUDFLARE_D1_DATABASE_ID="your-d1-id" export CLOUDFLARE_VECTORIZE_INDEX="mcp-memory-index" export MCP_MEMORY_STORAGE_BACKEND="cloudflare" # Import to Cloudflare python scripts/import_to_cloudflare.py /path/to/export.json ``` #### 2. Configure Each Machine **Claude Desktop Configuration** (`claude_desktop_config.json`): ```json { "mcpServers": { "memory": { "command": "/path/to/memory", "args": ["server"], "env": { "MCP_MEMORY_STORAGE_BACKEND": "cloudflare", "MCP_MEMORY_SQLITE_PATH": "/local/backup/path/sqlite_vec.db", "CLOUDFLARE_API_TOKEN": "your-token", "CLOUDFLARE_ACCOUNT_ID": "your-account", "CLOUDFLARE_D1_DATABASE_ID": "your-d1-id", "CLOUDFLARE_VECTORIZE_INDEX": "mcp-memory-index" } } } } ``` #### 3. Verification Testing Test bidirectional sync by storing and retrieving memories from each machine: ```python # Test script for verification import asyncio from mcp_memory_service.storage.cloudflare import CloudflareStorage async def test_sync(): storage = CloudflareStorage(...) await storage.initialize() # Store test memory test_memory = Memory(content="Test from Machine A", tags=["sync-test"]) success, message = await storage.store(test_memory) # Verify from other machine results = await storage.retrieve("Test from Machine A") print(f"Sync verified: {len(results)} results found") ``` ### Important Notes - **v6.13.7 Required**: This version fixes the critical Vectorize ID length issue - **Breaking Change**: Vector IDs changed format from v6.13.6 (removed "mem_" prefix) - **Backup Strategy**: Local sqlite_vec files are maintained for fallback - **Migration Time**: Allow extra time for initial memory migration to Cloudflare ### Troubleshooting Sync Issues #### Vector ID Length Error (Fixed in v6.13.7) ``` Error: "id too long; max is 64 bytes, got 68 bytes" ``` **Solution**: Update to v6.13.7 or later #### Environment Variable Issues **Problem**: Memories not syncing between machines **Solution**: - Verify identical environment variables on all machines - Check Claude Desktop configuration matches exactly - Restart Claude Desktop after config changes #### Sync Verification ```bash # Check memory count on each machine memory status # Test cross-machine visibility memory retrieve "test query from other machine" ``` ## Support For issues and questions: 1. **Documentation**: Check this guide and API documentation 2. **GitHub Issues**: Report bugs at the project repository 3. **Cloudflare Support**: For Cloudflare service-specific issues 4. **Community**: Join the project Discord/community channels ## Performance Benchmarks ### Typical Performance - **Storage**: ~200ms per memory (including embedding generation) - **Search**: ~100ms for semantic search (5 results) - **Batch Operations**: ~50ms per memory in batches of 100 - **Global Latency**: <100ms from most global locations ### Optimization Tips 1. **Batch Operations**: Use bulk operations when possible 2. **Content Strategy**: Use R2 for large content 3. **Caching**: Enable embedding caching 4. **Connection Pooling**: Reuse HTTP connections 5. **Regional Deployment**: Deploy close to your users