# cloudflare-crawl-mcp

MCP server for [Cloudflare Browser Rendering Crawl API](https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/). Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

Built with [FastMCP](https://github.com/jlowin/fastmcp) and [uv](https://github.com/astral-sh/uv).

## Tools

| Tool | Description |
|------|-------------|
| `scrape_url` | Fetch a single page as Markdown. The primary tool — use this when you know the URL. |
| `map_url` | Discover URLs on a site without fetching content. Use to find the right page first. |
| `crawl_url` | Crawl multiple pages and return all content as Markdown. |

**Typical workflow:** `map_url` to find pages → `scrape_url` to read the right one.

## Prerequisites

1. A Cloudflare account with [Browser Rendering](https://developers.cloudflare.com/browser-rendering/) enabled
2. An API token with **Account > Browser Rendering > Edit** permission ([create one here](https://dash.cloudflare.com/profile/api-tokens))
3. [uv](https://docs.astral.sh/uv/getting-started/installation/) installed

## Configuration

| Variable | Required | Description |
|----------|----------|-------------|
| `CF_API_TOKEN` | Yes | Cloudflare API token |
| `CF_ACCOUNT_ID` | Yes | Cloudflare Account ID |
| `CF_RATE_LIMIT` | No | API requests per minute (default: `6` for Free, set to `600` for Paid) |

## Setup

### Claude Code

```bash
claude mcp add cloudflare-crawl \
  -e CF_API_TOKEN=your_api_token \
  -e CF_ACCOUNT_ID=your_account_id \
  -- uv run --directory /path/to/cloudflare-crawl-mcp python server.py
```

### Codex

```bash
codex mcp add cloudflare-crawl \
  -- env CF_API_TOKEN="your_api_token" CF_ACCOUNT_ID="your_account_id" \
  uv run --directory /path/to/cloudflare-crawl-mcp python server.py
```

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` in your project or `~/.cursor/mcp.json` globally:

```json
{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}
```

## Tool Reference

### `scrape_url`

Fetch a single page and return its content as Markdown.

```
url (string, required)    — The URL to fetch
render (boolean, optional) — Render JavaScript with headless browser (default: true, set false for static pages)
```

### `map_url`

Discover URLs on a website without fetching full content.

```
url (string, required)                  — Starting URL
limit (number, optional)                — Max URLs to discover (default: 50)
depth (number, optional)                — Link depth to follow (default: 2)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs (e.g. "https://example.com/docs/**")
exclude_patterns (string[], optional)   — Skip matching URLs
```

### `crawl_url`

Crawl multiple pages and return all content as Markdown.

```
url (string, required)                  — Starting URL
limit (number, optional)                — Max pages to crawl (default: 10)
depth (number, optional)                — Link depth (default: 1)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs
exclude_patterns (string[], optional)   — Skip matching URLs
render (boolean, optional)              — Render JavaScript (default: true)
```

## Cloudflare Plan Limits

| | Free | Paid |
|---|---|---|
| Browser time | 10 min/day | 10 hrs/month |
| API rate limit | 6 req/min | 600 req/min |
| Concurrent browsers | 3 | 10 |
| Max pages per job | 100,000 | 100,000 |
| Max job duration | 7 days | 7 days |
| Results available | 14 days | 14 days |

`render: false` crawls run on Workers instead of a headless browser and do not consume browser time.

## License

MIT