Document Extractor MCP Server

# Document Extractor MCP Server A Model Context Protocol (MCP) server that extracts document content from Microsoft Learn and GitHub URLs, storing them in PocketBase for easy retrieval and search. ## Features ✅ **Latest MCP SDK Features (v1.12.0+)** - Modern `McpServer` architecture with enhanced capabilities - Multiple transport protocols: STDIO, Streamable HTTP, SSE - Dynamic tool management with lazy loading - Session management for stateful connections - Server-Sent Events support with backwards compatibility - Real-time server statistics and metrics ✅ **Content Extraction** - Microsoft Learn articles with rich metadata - GitHub files (README, documentation, code files) - Intelligent content parsing and cleaning - Duplicate detection and updates ✅ **PocketBase Integration** - Persistent document storage - Full-text search capabilities - Metadata preservation - CRUD operations ✅ **Advanced Server Features** - Multiple transport modes (STDIO/HTTP) - Health check and info endpoints - Read-only mode support - Enhanced error handling and debugging - Resource endpoints for server metrics ✅ **Rich Metadata** - Word counts and content statistics - Source attribution and URLs - Extraction timestamps - Content headers and descriptions ## Requirements - Node.js 18+ with ES modules support - PocketBase server running - Network access for content extraction ## Installation ### 1. Install Dependencies ```powershell # Navigate to the project directory cd c:\powershell_scripts\pocketbase_document_mcp\document-extractor-mcp # Install dependencies npm install ``` ### 2. PocketBase Setup The MCP server supports **both local and remote PocketBase instances**. Choose the setup that best fits your needs: #### Option A: Local PocketBase Instance 1. **Download and install PocketBase:** ```powershell # Download from https://pocketbase.io/docs/ # Extract the executable to your preferred directory ``` 2. **Start local PocketBase server:** ```powershell # Run from the directory containing pocketbase.exe .\pocketbase.exe serve # Or specify custom port and data directory .\pocketbase.exe serve --http="127.0.0.1:8090" --dir="./pb_data" ``` 3. **Set up admin account:** - Access PocketBase Admin UI at http://127.0.0.1:8090/_/ - Create your admin account - Note the email/password for configuration #### Option B: Remote PocketBase Instance 1. **Deploy PocketBase to your preferred hosting:** - Railway, Fly.io, DigitalOcean, AWS, etc. - Follow your hosting provider's deployment guide - Ensure HTTPS is enabled for production 2. **Configure your remote instance:** - Set up admin account through the web interface - Configure CORS settings if needed - Note the full URL (e.g., https://your-pb-instance.com) #### Option C: Docker PocketBase 1. **Using Docker Compose:** ```yaml version: '3.8' services: pocketbase: image: ghcr.io/muchobien/pocketbase:latest ports: - "8090:8090" volumes: - ./pb_data:/pb/pb_data ``` 2. **Collection Management (Automatic for all setups):** - The server will automatically create the required `documents` collection on startup - If `AUTO_CREATE_COLLECTION=true` (default), no manual setup needed - Use the `ensure_collection` tool to manually verify/create collections - Use the `collection_info` tool to check collection status 3. **Manual Collection Setup (if needed):** - Access PocketBase Admin UI - Create a new collection named `documents` - Add these fields: ``` title (Text, required) content (Text, required) metadata (JSON, required) created (Date, auto-generated) updated (Date, optional) ``` ### 3. Environment Configuration Create a `.env` file in the project root. The server supports both **local** and **remote** PocketBase instances: #### For Local PocketBase Instance: ```env # PocketBase Configuration - Local POCKETBASE_URL=http://127.0.0.1:8090 POCKETBASE_ADMIN_EMAIL=admin@example.com POCKETBASE_ADMIN_PASSWORD=your-secure-password # Collection Settings DOCUMENTS_COLLECTION=documents # Transport Configuration TRANSPORT_MODE=stdio HTTP_PORT=3000 # Development Settings DEBUG=true NODE_ENV=development READ_ONLY_MODE=false # Collection Management ✨ New! AUTO_CREATE_COLLECTION=true ``` #### For Remote PocketBase Instance: ```env # PocketBase Configuration - Remote POCKETBASE_URL=https://your-pocketbase-instance.com POCKETBASE_ADMIN_EMAIL=admin@yourdomain.com POCKETBASE_ADMIN_PASSWORD=your-secure-password # Collection Settings DOCUMENTS_COLLECTION=documents # Transport Configuration TRANSPORT_MODE=stdio HTTP_PORT=3000 # Production Settings DEBUG=false NODE_ENV=production READ_ONLY_MODE=false # Collection Management AUTO_CREATE_COLLECTION=true ``` #### For Dockerized PocketBase: ```env # PocketBase Configuration - Docker POCKETBASE_URL=http://pocketbase:8090 POCKETBASE_ADMIN_EMAIL=admin@localhost POCKETBASE_ADMIN_PASSWORD=admin123 # Collection Settings DOCUMENTS_COLLECTION=documents # Transport Configuration TRANSPORT_MODE=stdio HTTP_PORT=3000 # Container Settings DEBUG=false NODE_ENV=production READ_ONLY_MODE=false # Collection Management AUTO_CREATE_COLLECTION=true ``` ## Usage ### Starting the Server The server supports multiple transport modes: ```powershell # STDIO mode (default) - for Claude Desktop and CLI clients npm start # or explicitly npm run start:stdio # HTTP mode - for web clients and testing npm run start:http # Development modes with debug logging npm run dev # STDIO mode with debugging npm run dev:http # HTTP mode with debugging npm run dev:stdio # STDIO mode with debugging # Test the setup npm run test ``` ### Transport Modes #### STDIO Mode (Default) Perfect for Claude Desktop and command-line MCP clients: ```powershell npm start ``` #### HTTP Mode Enables web-based clients and testing with multiple protocols: ```powershell npm run start:http ``` Available endpoints in HTTP mode: - `POST /mcp` - Streamable HTTP transport (modern protocol 2025-03-26) - `GET /sse` - Server-Sent Events transport (legacy protocol 2024-11-05) - `POST /messages` - SSE message endpoint - `GET /health` - Health check endpoint - `GET /info` - Server information endpoint ### Available Tools #### 1. `extract_document` Extract and store content from URLs. **Parameters:** - `url` (string, required): Microsoft Learn or GitHub URL **Example:** ```json { "url": "https://learn.microsoft.com/en-us/azure/cognitive-services/openai/" } ``` #### 2. `list_documents` List stored documents with pagination. **Parameters:** - `limit` (number, optional): Max results per page (1-100, default: 20) - `page` (number, optional): Page number (default: 1) #### 3. `search_documents` Search documents by title or content. **Parameters:** - `query` (string, required): Search query - `limit` (number, optional): Max results (1-100, default: 50) #### 4. `get_document` Retrieve a specific document by ID. **Parameters:** - `id` (string, required): Document ID #### 5. `delete_document` Delete a document by ID. **Parameters:** - `id` (string, required): Document ID to delete #### 6. `ensure_collection` ✨ New! Check if the documents collection exists and create it if needed. **Parameters:** None **Description:** Automatically verifies the documents collection exists in PocketBase. If not found, creates the collection with the proper schema including all required fields and indexes. #### 7. `collection_info` ✨ New! Get detailed information about the documents collection including statistics. **Parameters:** None **Description:** Returns comprehensive collection information including schema details, record counts, indexes, and timestamps. ### Available Resources #### 1. `stats://server` Real-time server statistics and metrics. **Content:** - Total document count - Server information (name, version, uptime) - Memory usage statistics - Environment information - Read-only mode status ### Dynamic Tool Management The server supports dynamic tool management with lazy loading: ```javascript // Tools can be dynamically enabled/disabled if (process.env.READ_ONLY_MODE === 'true') { // Write operations are disabled in read-only mode deleteDocumentTool.disable(); extractDocumentTool.disable(); } // Tools can be re-enabled at runtime tool.enable(); ``` ### Session Management In HTTP mode, the server supports session management: - **Streamable HTTP**: Modern session management with automatic session ID generation - **SSE (Legacy)**: Backwards compatible session handling - **Session persistence**: Sessions are maintained across requests - **Automatic cleanup**: Sessions are cleaned up when connections close ## Supported Sources ### Microsoft Learn - Full article extraction - Metadata preservation (description, keywords, author) - Section headers extraction - Content cleaning and formatting **Example URLs:** - `https://learn.microsoft.com/en-us/azure/cognitive-services/openai/` - `https://learn.microsoft.com/en-us/dotnet/core/introduction` ### GitHub - File content extraction (README, docs, code) - Repository metadata - Branch handling (main/master fallback) - File type detection **Supported URL formats:** - `https://github.com/owner/repo` (assumes README.md) - `https://github.com/owner/repo/blob/main/file.md` - `https://raw.githubusercontent.com/owner/repo/main/file.md` ## Configuration Options ### Environment Variables | Variable | Description | Default | |----------|-------------|---------| | `POCKETBASE_URL` | PocketBase server URL | `http://127.0.0.1:8090` | | `POCKETBASE_ADMIN_EMAIL` | Admin email for authentication | Required | | `POCKETBASE_ADMIN_PASSWORD` | Admin password | Required | | `DOCUMENTS_COLLECTION` | Collection name for documents | `documents` | | `DEBUG` | Enable debug logging | `false` | | `NODE_ENV` | Environment mode | `development` | | `READ_ONLY_MODE` | Disable write operations | `false` | | `AUTO_CREATE_COLLECTION` | Auto-create collections on startup | `true` | ### Debug Mode Enable detailed logging: ```powershell $env:DEBUG="true"; node server.js ``` Debug logs include: - Authentication status - Content extraction details - Database operations - Error context ## Error Handling The server implements comprehensive error handling: - **Network errors**: Timeout and connection issues - **Authentication errors**: PocketBase connection problems - **Validation errors**: Invalid input parameters - **Content errors**: Extraction failures - **Database errors**: Storage and retrieval issues All errors are returned as structured MCP responses with appropriate error codes. ## Development ### Scripts ```powershell # Start in development mode npm run dev # Start in production mode npm start # Install dependencies npm run install-deps ``` ### Testing the Server ```powershell # Test basic functionality $env:DEBUG="true"; node server.js # In another terminal, you can test with MCP tools or: # Use Claude Desktop with MCP configuration # Use other MCP-compatible clients ``` ## Troubleshooting ### Common Issues 1. **Authentication Failed** - Verify PocketBase is running: `http://127.0.0.1:8090` - Check admin credentials in `.env` - Ensure admin user exists in PocketBase 2. **Content Extraction Errors** - Check network connectivity - Verify URL accessibility - Review debug logs for details 3. **Collection Not Found** - Use the `ensure_collection` tool to automatically create the collection - Check collection name in environment variables - Verify `AUTO_CREATE_COLLECTION` is enabled - Check collection permissions 4. **Module Import Errors** - Ensure `"type": "module"` in package.json - Use Node.js 18+ with ES modules support - Check all dependencies are installed ### Debug Information Enable debug mode to see detailed logs: ```powershell $env:DEBUG="true"; node server.js ``` ### PocketBase Collection Schema If you need to recreate the collection, use this schema: ```javascript { "name": "documents", "type": "base", "schema": [ { "name": "title", "type": "text", "required": true, "options": { "max": 255 } }, { "name": "content", "type": "text", "required": true }, { "name": "metadata", "type": "json", "required": true }, { "name": "created", "type": "date", "required": false }, { "name": "updated", "type": "date", "required": false } ] } ``` ## MCP Client Configuration ### Claude Desktop Configuration Add this to your Claude Desktop MCP settings: ```json { "mcpServers": { "document-extractor": { "command": "node", "args": ["c:\\powershell_scripts\\pocketbase_document_mcp\\document-extractor-mcp\\server.js"], "env": { "POCKETBASE_URL": "http://127.0.0.1:8090", "POCKETBASE_ADMIN_EMAIL": "your-admin@example.com", "POCKETBASE_ADMIN_PASSWORD": "your-password", "DEBUG": "false" } } } } ``` ## License MIT License - see LICENSE file for details. ## Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Add tests if applicable 5. Submit a pull request ## Changelog ### v1.1.0 ✨ Latest Update - **Latest MCP SDK v1.13.1+**: Upgraded to the newest Model Context Protocol SDK - **Latest PocketBase SDK v0.26.1+**: Updated to the latest PocketBase features - **Collection Management Tools**: Added `ensure_collection` and `collection_info` tools - **Auto-Collection Creation**: Automatic database schema setup on startup - **Enhanced Lazy Loading**: Improved dynamic tool management - **Latest SSE Features**: Modern Server-Sent Events implementation - **Improved Error Handling**: Better collection management error recovery - **Enhanced Documentation**: Comprehensive usage examples and troubleshooting ### v1.0.0 - Updated to latest Anthropic MCP SDK - Added comprehensive error handling - Implemented input validation with Zod - Enhanced metadata extraction - Added debug logging - Improved documentation - Added PocketBase integration - Support for Microsoft Learn and GitHub ## Deployment ### Smithery Deployment This MCP server supports deployment on [Smithery](https://smithery.ai), a platform for hosting MCP servers. #### TypeScript Deploy (Recommended) The fastest way to deploy this server on Smithery: 1. **Fork or Clone** this repository to your GitHub account 2. **Connect GitHub** to Smithery (or claim your server if already listed) 3. **Navigate** to the Deployments tab on your server page 4. **Click Deploy** - Smithery will automatically build and host your server The `smithery.yaml` file is already configured for TypeScript/Node.js deployment. **Note**: Despite being called "TypeScript Deploy", this method works perfectly for Node.js projects with ES modules. #### Custom Deploy (Docker) For advanced deployment with full Docker control: 1. **Replace smithery.yaml** with the container configuration: ```bash cp smithery-container.yaml smithery.yaml ``` 2. **Push to GitHub** with the updated configuration 3. **Deploy** via Smithery's Deployments tab The `Dockerfile` is optimized for production deployment with security best practices. #### Configuration When deploying on Smithery, you'll configure: - **PocketBase URL**: Your PocketBase instance URL - **Admin Credentials**: Email and password for PocketBase admin - **Collection Settings**: Default collection name and auto-creation - **Debug Mode**: Enable detailed logging (optional) #### Best Practices for Smithery - **Tool Discovery**: All tools are available without authentication for discovery - **Lazy Authentication**: API validation occurs only when tools are invoked - **Environment Variables**: Configuration is handled via Smithery's config schema - **Health Checks**: Built-in health monitoring at `/health` endpoint