# Site Crawler MCP

A powerful Model Context Protocol (MCP) server for crawling websites and extracting assets including images and SEO metadata. Built for e-commerce sites and general web crawling needs.

## Features

- **Comprehensive website analysis**: 12 different extraction modes for complete website insights
- **Multi-mode crawling**: Extract multiple data types in a single pass
- **Smart extraction**: Advanced pattern matching for accurate data extraction
- **Performance optimized**: Concurrent crawling with rate limiting
- **Security analysis**: HTTPS, security headers, SSL/TLS information
- **SEO analysis**: Complete SEO audit including meta tags, structured data, and more
- **Legal compliance**: KVKK, GDPR, privacy policy detection
- **Business intelligence**: Brand info, references, contact details extraction

## Installation

### From PyPI (when published)
```bash
pip install site-crawler-mcp
```

### From Source (Development)

#### Using uv (Recommended)
```bash
# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment with Python 3.12
uv venv --python 3.12
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies and package
uv sync
```

#### Using pip
```bash
# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment (recommended)
python -m venv venv

# Activate virtual environment
# On Windows:
venv\Scripts\activate
# On Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install package in development mode
pip install -e .
```

## Usage

### As an MCP Server

Add to your MCP configuration file:
- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Linux**: `~/.config/Claude/claude_desktop_config.json`

#### Using uvx (Recommended)
```json
{
  "mcpServers": {
    "site-crawler": {
      "command": "uvx",
      "args": ["--from", "/path/to/site-crawler-mcp", "site-crawler-mcp"]
    }
  }
}
```

#### Using uv run
```json
{
  "mcpServers": {
    "site-crawler": {
      "command": "uv",
      "args": ["run", "site_crawler"],
      "cwd": "/path/to/site-crawler-mcp"
    }
  }
}
```

#### Using python directly
```json
{
  "mcpServers": {
    "site-crawler": {
      "command": "python",
      "args": ["-m", "site_crawler.server"],
      "cwd": "/path/to/site-crawler-mcp/src",
      "env": {
        "PYTHONPATH": "/path/to/site-crawler-mcp/src"
      }
    }
  }
}
```

**Note**: Replace `/path/to/site-crawler-mcp` with your actual project path. On Windows, use backslashes and drive letters (e.g., `C:\\Users\\YourName\\site-crawler-mcp`).

### Available Tools

#### `site_crawlAssets`

Crawl a website and extract various assets based on specified modes.

**Parameters:**
- `url` (string, required): The URL to start crawling from
- `modes` (array, required): Array of extraction modes (see below)
- `depth` (number, optional): Crawling depth (default: 1)
- `max_pages` (number, optional): Maximum pages to crawl (default: 50)

**Available Modes:**
- `images`: Extract all images with metadata (alt text, dimensions, format)
- `meta`: Basic SEO metadata (title, description, H1 tags)
- `brand`: Company branding information (logo, name, about pages)
- `seo`: Comprehensive SEO analysis (meta tags, structured data, open graph)
- `performance`: Page load metrics and performance indicators
- `security`: Security headers and HTTPS configuration
- `compliance`: Accessibility and regulatory compliance checks
- `infrastructure`: Server technology and CDN detection
- `legal`: Privacy policies, terms, KVKK compliance
- `careers`: Job opportunities and career pages
- `references`: Client testimonials and case studies
- `contact`: Contact information (email, phone, social media, address)

**Example Requests:**

1. Basic image extraction:
```json
{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["images"],
    "depth": 1
  }
}
```

2. Full SEO and security audit:
```json
{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["seo", "security", "performance"],
    "depth": 2
  }
}
```

3. Business intelligence gathering:
```json
{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["brand", "contact", "references", "careers"],
    "depth": 3
  }
}
```

4. Legal compliance check:
```json
{
  "tool": "site_crawlAssets",
  "arguments": {
    "url": "https://example.com",
    "modes": ["legal", "compliance"],
    "depth": 2
  }
}
```

## Development

### Requirements

- Python 3.10+
- BeautifulSoup4
- aiohttp
- MCP SDK
- uv (recommended for development)

### Setup Development Environment

#### Using uv (Recommended)
```bash
# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment with Python 3.12
uv venv --python 3.12
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies and package
uv sync
```

#### Using pip
```bash
# Clone the repository
git clone https://github.com/AndacGuven/site-crawler-mcp.git
cd site-crawler-mcp

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install in development mode
pip install -e .
```

### Running the Server

#### Using uv
```bash
# Run the MCP server
uv run site_crawler
# or
uv run site-crawler-mcp
# or
uv run python -m site_crawler.server
```

#### Using python directly
```bash
python -m site_crawler.server
```

### Running Tests

```bash
# Using uv
uv run pytest tests/

# Using pip
pytest tests/
```

### Project Structure

```
site-crawler-mcp/
├── README.md
├── requirements.txt
├── pyproject.toml
├── src/
│   └── site_crawler/
│       ├── __init__.py
│       ├── server.py
│       ├── crawler.py
│       └── utils.py
└── tests/
    ├── __init__.py
    └── test_crawler.py
```

## Configuration

### Environment Variables

- `CRAWLER_MAX_CONCURRENT`: Maximum concurrent requests (default: 5)
- `CRAWLER_TIMEOUT`: Request timeout in seconds (default: 30)
- `CRAWLER_USER_AGENT`: Custom user agent string

### Rate Limiting

The crawler respects `robots.txt` and implements polite crawling:
- 1-2 second delay between requests to the same domain
- Maximum 5 concurrent requests
- Automatic retry with exponential backoff

## Use Cases

### E-commerce Analysis
Extract product images, pricing, and brand information:
```
"Analyze the e-commerce site example.com for product images, brand info, and contact details"
```

### SEO and Performance Audit
Comprehensive SEO and performance analysis:
```
"Perform a full SEO audit of example.com including performance metrics and structured data"
```

### Security Assessment
Check security headers and HTTPS configuration:
```
"Analyze the security posture of example.com including headers and SSL configuration"
```

### Legal Compliance Check
Verify KVKK/GDPR compliance and privacy policies:
```
"Check example.com for KVKK compliance, privacy policies, and data protection measures"
```

### Business Intelligence
Gather company information and references:
```
"Extract business information from example.com including company details, references, and career opportunities"
```

### Contact Information Extraction
Find all contact details:
```
"Find all contact information on example.com including emails, phones, social media, and addresses"
```

## Performance Considerations

- Images smaller than 50KB are filtered out by default
- Concurrent crawling limited to 5 pages simultaneously
- Memory-efficient streaming for large sites
- Automatic deduplication of URLs

## Error Handling

The crawler handles various error scenarios gracefully:
- Network timeouts
- Invalid URLs
- Rate limiting (429 responses)
- JavaScript-heavy sites (graceful degradation)
- Memory limits

## Contributing

Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) for details on our code of conduct and the process for submitting pull requests.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

- Built with [MCP SDK](https://github.com/modelcontextprotocol/sdk)
- Inspired by the need for better e-commerce crawling tools
- Thanks to the open-source community

## Support

For issues and feature requests, please use the [GitHub issue tracker](https://github.com/AndacGuven/site-crawler-mcp/issues).