# Crawl4AI RAG MCP Server

A Retrieval-Augmented Generation (RAG) MCP server built with Python that enables AI assistants to crawl, index, and retrieve information from technical documentation using semantic search.

The server integrates Crawl4AI for web crawling, Supabase (pgvector) for vector storage, OpenAI embeddings for semantic retrieval, and Neo4j for repository knowledge graph validation.

---

## Features

- Model Context Protocol (MCP) server
- Intelligent web crawling with Crawl4AI
- Recursive website indexing
- Automatic document chunking
- OpenAI embedding generation
- Supabase pgvector vector database
- Semantic document retrieval
- Optional Hybrid Search
- Optional Contextual Embeddings
- Optional Agentic RAG
- Optional Cross-Encoder Reranking
- Neo4j Knowledge Graph integration
- AI code hallucination detection
- Repository structure validation
- Docker support
- SSE & STDIO transport support

---

## Tech Stack

| Category | Technologies |
|----------|--------------|
| Language | Python |
| AI | OpenAI API |
| Protocol | Model Context Protocol (MCP) |
| Web Crawling | Crawl4AI |
| Vector Database | Supabase + pgvector |
| Knowledge Graph | Neo4j |
| Containerization | Docker |
| Retrieval | RAG |

---

# Architecture

```
                    AI Assistant
                         │
                         ▼
                  MCP Server (Python)
                         │
        ┌────────────────┼────────────────┐
        │                │                │
        ▼                ▼                ▼
   Crawl4AI         Knowledge Graph     RAG Pipeline
        │              (Neo4j)             │
        ▼                                  ▼
 Crawl Websites                    Document Chunking
                                          │
                                          ▼
                                 OpenAI Embeddings
                                          │
                                          ▼
                              Supabase (pgvector)
                                          │
                                          ▼
                                  Semantic Search
                                          │
                                          ▼
                                  Generated Response
```

---

# MCP Tools

### Crawling

- Crawl a single page
- Crawl complete documentation websites
- Recursive crawling
- Sitemap crawling

### Retrieval

- Semantic RAG search
- Source filtering
- Hybrid retrieval
- Context-aware retrieval

### Knowledge Graph

- Parse GitHub repositories
- Validate AI-generated Python code
- Detect hallucinated imports
- Detect invalid methods/classes
- Query repository graph

---

# Project Structure

```
src/
├── crawl4ai_mcp.py
├── tools/
├── knowledge_graphs/
├── rag/
├── utils/

public/

docker/

.env
README.md
```

---

# Installation

Clone the repository

```bash
git clone <repository-url>
cd crawl4ai-rag-mcp-server
```

Install dependencies

```bash
pip install -r requirements.txt
```

---

# Environment Variables

Create a `.env` file.

```env
OPENAI_API_KEY=

SUPABASE_URL=
SUPABASE_SERVICE_KEY=

NEO4J_URI=
NEO4J_USER=
NEO4J_PASSWORD=
```

---

# Running

Using Python

```bash
python src/crawl4ai_mcp.py
```

Using Docker

```bash
docker build -t crawl4ai-rag .
docker run --env-file .env -p 8051:8051 crawl4ai-rag
```

---

# Retrieval Pipeline

1. Crawl technical documentation
2. Clean extracted content
3. Split documents into chunks
4. Generate embeddings
5. Store vectors in pgvector
6. Perform semantic similarity search
7. Inject retrieved context into prompts
8. Generate grounded responses

---

# Knowledge Graph Pipeline

- Parse GitHub repositories
- Extract classes
- Extract methods
- Extract imports
- Build Neo4j graph
- Validate AI-generated code
- Detect hallucinated APIs

---

# Future Improvements

- Multiple embedding model support
- Local embedding models
- Incremental indexing
- Authentication
- Citation support
- Multi-user support
- Document upload
- PDF ingestion
- Monitoring & Observability

---