Marcus Local MCP Server

# 📚 Marcus Local MCP Server A Model Context Protocol (MCP) server that indexes documentation sites and local code repositories for semantic search by AI assistants.     ## 🎯 What Is This? This is a **local MCP server** that enables AI assistants (Cursor, Claude Desktop, ChatGPT) to semantically search through: - **Documentation websites** - Crawled and indexed from any docs site - **Local code repositories** - All text files from your projects It uses OpenAI embeddings to create a vector database (ChromaDB) that AI assistants can query through the Model Context Protocol. **Think of it as:** Giving your AI assistant instant access to searchable documentation and your entire codebase. ## 📋 How It Works ``` ┌─────────────────┐ │ AI Assistant │ (Cursor, Claude, ChatGPT, etc.) │ (via MCP) │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ MCP Server │ (Python - stdio) │ main.py │ └────────┬────────┘ │ ▼ ┌─────────────────┐ ┌──────────────┐ │ ChromaDB │◄─────┤ OpenAI │ │ (Vector Store) │ │ Embeddings │ └────────┬────────┘ └──────────────┘ │ ▼ ┌──────────────────────┐ │ Indexed Sources │ │ • Documentation │ │ - Moca Network │ │ - Your Docs │ │ • Repositories │ │ - Your Codebase │ │ - Local Projects │ └──────────────────────┘ ``` **The Flow:** 1. **Index** - Crawl docs OR read local repo files 2. **Chunk** - Split content into 800-token chunks 3. **Embed** - Create OpenAI embeddings (batched for speed) 4. **Store** - Save in ChromaDB vector database 5. **Search** - AI assistant queries via MCP protocol 6. **Retrieve** - Return relevant chunks from docs/code ## 🚀 How to Run It ### 1. Setup ```bash # Clone repository git clone <your-repo> cd crawl4ai_test # Install Node.js dependencies npm install # Setup Python virtual environment python3 -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate # Install Python dependencies pip install -r mcp-docs-server/requirements.txt # Install Crawl4AI pip install -U crawl4ai crawl4ai-setup ``` ### 2. Configure Create `.env` file in `mcp-docs-server/`: ```bash OPENAI_API_KEY=your_openai_api_key_here EMBEDDING_MODEL=text-embedding-3-small DEFAULT_RESULTS=5 ``` ### 3. Run the Web UI ```bash # Start Next.js server npm run dev # Open browser open http://localhost:3030 ``` ### 4. Connect to Cursor/Claude Add to your AI assistant config: **For Cursor** (`~/.cursor/mcp.json` or project config): ```json { "mcpServers": { "marcus-mcp-server": { "command": "/path/to/your/venv/bin/python3", "args": ["/path/to/crawl4ai_test/mcp-docs-server/server/main.py"] } } } ``` **For Claude Desktop** (`~/Library/Application Support/Claude/claude_desktop_config.json`): ```json { "mcpServers": { "marcus-docs": { "command": "/path/to/your/venv/bin/python", "args": ["/path/to/crawl4ai_test/mcp-docs-server/server/main.py"] } } } ``` ## 📖 How to Use It ### Adding Documentation **Via Web UI:** 1. Go to http://localhost:3030 2. Click **"Add New Docs"** 3. Enter: - **URL**: `https://docs.example.com` - **Source Name**: `Example Docs` - **Max Pages**: `50` (or unlimited) 4. Click **"Start Indexing"** 5. Wait for completion **Via Command Line:** ```bash cd mcp-docs-server source ../venv/bin/activate python scripts/crawler.py https://docs.example.com "Example Docs" 50 python scripts/indexer_multi.py "Example Docs" ``` ### Adding Repositories **Via Web UI:** 1. Go to http://localhost:3030 2. Click **"Add Repository"** 3. Enter: - **Repository Path**: Drag-and-drop folder OR paste path - **Source Name**: Auto-generated from folder name 4. Click **"Start Indexing"** 5. Watch live progress **What gets indexed:** - ✅ All text files (`.js`, `.py`, `.md`, `.tsx`, `.json`, `.css`, etc.) - ✅ Auto-skips: `node_modules`, `.git`, `venv`, `build`, `.next`, etc. - ✅ Batched embeddings (50-100x faster) **Via Command Line:** ```bash cd mcp-docs-server source ../venv/bin/activate python scripts/repo_indexer.py "/path/to/your/repo" "My Project" ``` ### Searching **From Web UI:** 1. Enter query: `"How do I initialize the SDK?"` 2. Select source (Docs, Repos, or All) 3. Click **"Search Documentation"** 4. View results **From AI Assistant:** Search all sources: ``` @marcus-mcp-server search for "authentication flow" ``` Filter by specific source: ``` @marcus-mcp-server search for "BorrowInterface component" with source="Credo Protocol" ``` Example usage in Cursor: ``` User: Using my marcus-mcp-server, show me how authentication is implemented in the Credo Protocol repository AI: [Searches indexed repository and returns relevant code chunks] ``` **Pro Tip:** Always filter by source name to get focused results and save context tokens. ### Managing Sources **View Sources:** - See all indexed docs and repos on the main page - Filter by "Docs" or "Repos" tabs - Expand to see individual pages/files **Delete Sources:** - Click trash icon next to any source - Confirm deletion - Source and all chunks are removed ## 📁 Project Structure ``` crawl4ai_test/ ├── pages/ # Next.js UI │ ├── index.js # Main page (search + sources) │ ├── add.js # Add documentation │ ├── add-repo.js # Add repository │ └── api/ # API routes │ ├── mcp-search.js # Search endpoint │ ├── mcp-info.js # Get index info │ ├── add-docs-crawl.js # Crawl docs │ ├── add-docs-index.js # Index docs │ ├── add-repo-index.js # Index repository │ └── mcp-delete-source.js # Delete source ├── components/ │ ├── ui/ # shadcn/ui components │ └── home/ # Page components ├── mcp-docs-server/ # MCP Server │ ├── server/ │ │ └── main.py # MCP server (stdio) │ ├── scripts/ │ │ ├── crawler.py # Crawl docs with Crawl4AI │ │ ├── indexer_multi.py # Index docs │ │ ├── repo_indexer.py # Index repositories │ │ ├── get_source_pages.py # Get pages/files │ │ ├── search.py # Search │ │ └── delete_source.py # Delete sources │ ├── data/ │ │ ├── chroma_db/ # Vector database │ │ ├── chunks/ # Metadata │ │ └── raw/ # Crawled JSON │ └── requirements.txt └── venv/ # Python environment ``` ## 🎨 Built With - **Frontend**: Next.js 15 + shadcn/ui + Tailwind CSS - **Backend**: Python 3.13 + MCP Protocol - **Crawler**: Crawl4AI - **Vector DB**: ChromaDB - **Embeddings**: OpenAI (text-embedding-3-small) --- **Status**: ✅ Fully Operational | 🤖 MCP Ready | 🔍 Search Enabled