# Documentation Scraper & MCP Server

A comprehensive, domain-agnostic documentation scraping and AI integration toolkit. Scrape any documentation website, create structured databases, and integrate with Claude Desktop via MCP (Model Context Protocol) for seamless AI-powered documentation assistance.

## 🚀 Features

### Core Functionality
- **🌐 Universal Documentation Scraper**: Works with any documentation website
- **📊 Structured Database**: SQLite database with full-text search capabilities  
- **🤖 MCP Server Integration**: Native Claude Desktop integration via Model Context Protocol
- **📝 LLM-Optimized Output**: Ready-to-use context files for AI applications
- **⚙️ Configuration-Driven**: Single config file controls all settings

### Advanced Tools
- **🔍 Query Interface**: Command-line tool for searching and analyzing scraped content
- **🛠️ Debug Suite**: Comprehensive debugging tools for testing and validation
- **📋 Auto-Configuration**: Automatic MCP setup file generation
- **📈 Progress Tracking**: Detailed logging and error handling
- **💾 Resumable Crawls**: Smart caching for interrupted crawls

## 📋 Prerequisites

- Python 3.8 or higher
- Internet connection
- ~500MB free disk space per documentation site

## 🛠️ Quick Start

### 1. Installation

```bash
# Clone the repository
git clone <repository-url>
cd documentation-scraper

# Install dependencies
pip install -r requirements.txt
```

### 2. Configure Your Target

Edit `config.py` to set your documentation site:

```python
SCRAPER_CONFIG = {
    "base_url": "https://docs.example.com/",  # Your documentation site
    "output_dir": "docs_db",
    "max_pages": 200,
    # ... other settings
}
```

### 3. Run the Scraper

```bash
python docs_scraper.py
```

### 4. Query Your Documentation

```bash
# Search for content
python query_docs.py --search "tutorial"

# Browse by section
python query_docs.py --section "getting-started"

# Get statistics
python query_docs.py --stats
```

### 5. Set Up Claude Integration

```bash
# Generate MCP configuration files
python utils/gen_mcp.py

# Follow the instructions to add to Claude Desktop
```

## 🏗️ Project Structure

```
📁 documentation-scraper/
├── 📄 config.py                    # Central configuration file
├── 🕷️ docs_scraper.py              # Main scraper script
├── 🔍 query_docs.py                # Query and analysis tool
├── 🤖 mcp_docs_server.py           # MCP server for Claude integration
├── 📋 requirements.txt             # Python dependencies
├── 📁 utils/                       # Debug and utility tools
│   ├── 🛠️ gen_mcp.py               # Generate MCP config files
│   ├── 🧪 debug_scraper.py         # Test scraper functionality
│   ├── 🔧 debug_mcp_server.py      # Debug MCP server
│   ├── 🎯 debug_mcp_client.py      # Test MCP tools directly
│   ├── 📡 debug_mcp_server_protocol.py # Test MCP via JSON-RPC
│   └── 🌐 debug_site_content.py    # Debug content extraction
├── 📁 docs_db/                     # Generated documentation database
│   ├── 📊 documentation.db         # SQLite database
│   ├── 📄 documentation.json       # JSON export
│   ├── 📋 scrape_summary.json      # Statistics
│   └── 📁 llm_context/             # LLM-ready context files
└── 📁 mcp/                         # Generated MCP configuration
    ├── 🔧 run_mcp_server.bat       # Windows launcher script
    └── ⚙️ claude_mcp_config.json   # Claude Desktop config
```

## ⚙️ Configuration

### Main Configuration (`config.py`)

The entire system is controlled by a single configuration file:

```python
# Basic scraping settings
SCRAPER_CONFIG = {
    "base_url": "https://docs.example.com/",
    "output_dir": "docs_db",
    "max_depth": 3,
    "max_pages": 200,
    "delay_between_requests": 0.5,
}

# URL filtering rules
URL_FILTER_CONFIG = {
    "skip_patterns": [r'/api/', r'\.pdf$'],
    "allowed_domains": ["docs.example.com"],
}

# MCP server settings
MCP_CONFIG = {
    "server_name": "docs-server",
    "default_search_limit": 10,
    "max_search_limit": 50,
}
```

### Environment Overrides

You can override any setting with environment variables:

```bash
export DOCS_DB_PATH="/custom/path/documentation.db"
export DOCS_BASE_URL="https://different-docs.com/"
python mcp_docs_server.py
```

## 🤖 Claude Desktop Integration

### Automatic Setup

1. **Generate configuration files**:
   ```bash
   python utils/gen_mcp.py
   ```

2. **Copy the generated config** to Claude Desktop:
   - **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
   - **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`

3. **Restart Claude Desktop**

### Manual Setup

If you prefer manual setup, add this to your Claude Desktop config:

```json
{
  "mcpServers": {
    "docs": {
      "command": "python",
      "args": ["path/to/mcp_docs_server.py"],
      "cwd": "path/to/project",
      "env": {
        "DOCS_DB_PATH": "path/to/docs_db/documentation.db"
      }
    }
  }
}
```

### Available MCP Tools

Once connected, Claude can use these tools:

- **🔍 search_documentation**: Search for content across all documentation
- **📚 get_documentation_sections**: List all available sections  
- **📄 get_page_content**: Get full content of specific pages
- **🗂️ browse_section**: Browse pages within a section
- **📊 get_documentation_stats**: Get database statistics

## 🔧 Command Line Tools

### Documentation Scraper

```bash
# Basic scraping
python docs_scraper.py

# Override config settings
python docs_scraper.py  # Settings from config.py
```

### Query Tool

```bash
# Search for content
python query_docs.py --search "authentication guide"

# Browse specific sections  
python query_docs.py --section "api-reference"

# Get database statistics
python query_docs.py --stats

# List all sections
python query_docs.py --list-sections

# Export section to file
python query_docs.py --export-section "tutorials" --format markdown > tutorials.md

# Use custom database
python query_docs.py --db "custom/path/docs.db" --search "example"
```

### Debug Tools

```bash
# Test scraper functionality
python utils/debug_scraper.py

# Test MCP server
python utils/debug_mcp_server.py

# Test MCP tools directly
python utils/debug_mcp_client.py

# Test MCP protocol
python utils/debug_mcp_server_protocol.py

# Debug content extraction
python utils/debug_site_content.py

# Generate MCP config files
python utils/gen_mcp.py
```

## 📊 Database Schema

### Pages Table
```sql
CREATE TABLE pages (
    id INTEGER PRIMARY KEY,
    url TEXT UNIQUE NOT NULL,
    title TEXT,
    content TEXT,
    markdown TEXT,
    word_count INTEGER,
    section TEXT,
    subsection TEXT,
    scraped_at TIMESTAMP,
    metadata TEXT
);
```

### Full-Text Search
```sql
-- Search using FTS5
SELECT * FROM pages_fts WHERE pages_fts MATCH 'your search term';

-- Or use the query tool
python query_docs.py --search "your search term"
```

## 🎯 Example Use Cases

### 1. Documentation Analysis
```bash
# Get overview of documentation
python query_docs.py --stats

# Find all tutorial content
python query_docs.py --search "tutorial guide example"

# Export specific sections
python query_docs.py --export-section "getting-started" > onboarding.md
```

### 2. AI Integration with Claude
```python
# Once MCP is set up, ask Claude:
# "Search the documentation for authentication examples"
# "What sections are available in the documentation?"
# "Show me the content for the API reference page"
```

### 3. Custom Applications
```python
import sqlite3

# Connect to your scraped documentation
conn = sqlite3.connect('docs_db/documentation.db')

# Query for specific content
results = conn.execute("""
    SELECT title, url, markdown 
    FROM pages 
    WHERE section = 'tutorials' 
    AND word_count > 500
    ORDER BY word_count DESC
""").fetchall()

# Build your own tools on top of the structured data
```

## 🔍 Debugging and Testing

### Test Scraper Before Full Run
```bash
python utils/debug_scraper.py
```

### Validate Content Extraction  
```bash
python utils/debug_site_content.py
```

### Test MCP Integration
```bash
# Test server functionality
python utils/debug_mcp_server.py

# Test tools directly
python utils/debug_mcp_client.py

# Test JSON-RPC protocol
python utils/debug_mcp_server_protocol.py
```

## 📈 Performance and Optimization

### Scraping Performance
- **Start small**: Use `max_pages=50` for testing
- **Adjust depth**: `max_depth=2` covers most content efficiently  
- **Rate limiting**: Increase `delay_between_requests` if getting blocked
- **Caching**: Enabled by default for resumable crawls

### Database Performance
- **Full-text search**: Automatic FTS5 index for fast searching
- **Indexing**: Optimized indexes on URL and section columns
- **Word counts**: Pre-calculated for quick statistics

### MCP Performance  
- **Configurable limits**: Set appropriate search and section limits
- **Snippet length**: Adjust snippet size for optimal response times
- **Connection pooling**: Efficient database connections

## 🌐 Supported Documentation Sites

This scraper works with most documentation websites including:

- **Static sites**: Hugo, Jekyll, MkDocs, Docusaurus
- **Documentation platforms**: GitBook, Notion, Confluence  
- **API docs**: Swagger/OpenAPI documentation
- **Wiki-style**: MediaWiki, TiddlyWiki
- **Custom sites**: Any site with consistent HTML structure

### Site-Specific Configuration

Customize URL filtering and content extraction for your target site:

```python
URL_FILTER_CONFIG = {
    "skip_patterns": [
        r'/api/',           # Skip API endpoint docs
        r'/edit/',          # Skip edit pages  
        r'\.pdf$',          # Skip PDF files
    ],
    "allowed_domains": ["docs.yoursite.com"],
}

CONTENT_FILTER_CONFIG = {
    "remove_patterns": [
        r'Edit this page.*?\n',      # Remove edit links
        r'Was this helpful\?.*?\n',  # Remove feedback sections
    ],
}
```

## 🤝 Contributing

We welcome contributions! Here are some areas where you can help:

- **New export formats**: PDF, EPUB, Word documents
- **Enhanced content filtering**: Better noise removal
- **Additional debug tools**: More comprehensive testing
- **Documentation**: Improve guides and examples
- **Performance optimizations**: Faster scraping and querying

## ⚠️ Responsible Usage

- **Respect robots.txt**: Check the target site's robots.txt file
- **Rate limiting**: Use appropriate delays between requests
- **Terms of service**: Respect the documentation site's terms
- **Fair use**: Use for educational, research, or personal purposes
- **Attribution**: Credit the original documentation source

## 📄 License

This project is provided as-is for educational and research purposes. Please respect the terms of service and licensing of the documentation sites you scrape.

---

## 🎉 Getting Started Examples

### Example 1: Scrape Python Documentation
```python
# config.py
SCRAPER_CONFIG = {
    "base_url": "https://docs.python.org/3/",
    "max_pages": 500,
    "max_depth": 3,
}
```

### Example 2: Scrape API Documentation
```python
# config.py  
SCRAPER_CONFIG = {
    "base_url": "https://api-docs.example.com/",
    "max_pages": 200,
}

URL_FILTER_CONFIG = {
    "skip_patterns": [r'/changelog/', r'/releases/'],
}
```

### Example 3: Corporate Documentation
```python
# config.py
SCRAPER_CONFIG = {
    "base_url": "https://internal-docs.company.com/",
    "output_dir": "company_docs",
}

MCP_CONFIG = {
    "server_name": "company-docs-server",
    "docs_display_name": "Company Internal Docs",
}
```

---

**Happy Documenting! 📚✨**

For questions, issues, or feature requests, please check the debug logs first, then create an issue with relevant details.

---

## 🙏 Attribution

This project is powered by **[Crawl4AI](https://github.com/unclecode/crawl4ai)** - an amazing open-source LLM-friendly web crawler and scraper.

<a href="https://github.com/unclecode/crawl4ai">
  <img src="https://img.shields.io/badge/Powered%20by-Crawl4AI-blue?style=flat-square" alt="Powered by Crawl4AI"/>
</a>

Crawl4AI enables the intelligent web scraping capabilities that make this documentation toolkit possible. A huge thanks to [@unclecode](https://github.com/unclecode) and the Crawl4AI community for building such an incredible tool! 🚀

**Check out Crawl4AI:**
- **Repository**: https://github.com/unclecode/crawl4ai
- **Documentation**: https://crawl4ai.com
- **Discord Community**: https://discord.gg/jP8KfhDhyN

## 📄 License