Crawl4AI RAG MCP Server

A Retrieval-Augmented Generation (RAG) MCP server built with Python that enables AI assistants to crawl, index, and retrieve information from technical documentation using semantic search.

The server integrates Crawl4AI for web crawling, Supabase (pgvector) for vector storage, OpenAI embeddings for semantic retrieval, and Neo4j for repository knowledge graph validation.

Features

• Model Context Protocol (MCP) server

• Intelligent web crawling with Crawl4AI

• Recursive website indexing

• Automatic document chunking

• OpenAI embedding generation

• Supabase pgvector vector database

• Semantic document retrieval

• Optional Hybrid Search

• Optional Contextual Embeddings

• Optional Agentic RAG

• Optional Cross-Encoder Reranking

• Neo4j Knowledge Graph integration

• AI code hallucination detection

• Repository structure validation

• Docker support

• SSE & STDIO transport support

Related MCP server: Documentation MCP Server

Tech Stack

Architecture

                    AI Assistant
                         │
                         ▼
                  MCP Server (Python)
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
   Crawl4AI         Knowledge Graph     RAG Pipeline
        │              (Neo4j)             │
        ▼                                  ▼
 Crawl Websites                    Document Chunking
                                          │
                                          ▼
                                 OpenAI Embeddings
                                          │
                                          ▼
                              Supabase (pgvector)
                                          │
                                          ▼
                                  Semantic Search
                                          │
                                          ▼
                                  Generated Response

MCP Tools

Crawling

• Crawl a single page

• Crawl complete documentation websites

• Recursive crawling

• Sitemap crawling

Retrieval

• Semantic RAG search

• Source filtering

• Hybrid retrieval

• Context-aware retrieval

Knowledge Graph

• Parse GitHub repositories

• Validate AI-generated Python code

• Detect hallucinated imports

• Detect invalid methods/classes

• Query repository graph

Project Structure

src/
├── crawl4ai_mcp.py
├── tools/
├── knowledge_graphs/
├── rag/
├── utils/

public/

docker/

.env
README.md

Installation

Clone the repository

git clone <repository-url>
cd crawl4ai-rag-mcp-server

Install dependencies

pip install -r requirements.txt

Environment Variables

Create a .env file.

OPENAI_API_KEY=

SUPABASE_URL=
SUPABASE_SERVICE_KEY=

NEO4J_URI=
NEO4J_USER=
NEO4J_PASSWORD=

Running

Using Python

python src/crawl4ai_mcp.py

Using Docker

docker build -t crawl4ai-rag .
docker run --env-file .env -p 8051:8051 crawl4ai-rag

Retrieval Pipeline

1. Crawl technical documentation

2. Clean extracted content

3. Split documents into chunks

4. Generate embeddings

5. Store vectors in pgvector

6. Perform semantic similarity search

7. Inject retrieved context into prompts

8. Generate grounded responses

Knowledge Graph Pipeline

• Parse GitHub repositories

• Extract classes

• Extract methods

• Extract imports

• Build Neo4j graph

• Validate AI-generated code

• Detect hallucinated APIs

Future Improvements

• Multiple embedding model support

• Local embedding models

• Incremental indexing

• Authentication

• Citation support

• Multi-user support

• Document upload

• PDF ingestion

• Monitoring & Observability