Skip to main content
Glama
cloudflare-crawl-mcp

cloudflare-crawl-mcp

by bidouilles

cloudflare-crawl-mcp

MCP server for Cloudflare Browser Rendering Crawl API. Fetches and crawls web pages, returning clean Markdown optimized for LLM consumption.

Built with FastMCP and uv.

Tools

Tool

Description

scrape_url

Fetch a single page as Markdown. The primary tool — use this when you know the URL.

map_url

Discover URLs on a site without fetching content. Use to find the right page first.

crawl_url

Crawl multiple pages and return all content as Markdown.

Typical workflow: map_url to find pages → scrape_url to read the right one.

Related MCP server: Scrapi MCP Server

Prerequisites

  1. A Cloudflare account with Browser Rendering enabled

  2. An API token with Account > Browser Rendering > Edit permission (create one here)

  3. uv installed

Configuration

Variable

Required

Description

CF_API_TOKEN

Yes

Cloudflare API token

CF_ACCOUNT_ID

Yes

Cloudflare Account ID

CF_RATE_LIMIT

No

API requests per minute (default: 6 for Free, set to 600 for Paid)

Setup

Claude Code

claude mcp add cloudflare-crawl \
  -e CF_API_TOKEN=your_api_token \
  -e CF_ACCOUNT_ID=your_account_id \
  -- uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Codex

codex mcp add cloudflare-crawl \
  -- env CF_API_TOKEN="your_api_token" CF_ACCOUNT_ID="your_account_id" \
  uv run --directory /path/to/cloudflare-crawl-mcp python server.py

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project or ~/.cursor/mcp.json globally:

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": ["run", "--directory", "/path/to/cloudflare-crawl-mcp", "python", "server.py"],
      "env": {
        "CF_API_TOKEN": "your_api_token",
        "CF_ACCOUNT_ID": "your_account_id"
      }
    }
  }
}

Tool Reference

scrape_url

Fetch a single page and return its content as Markdown.

url (string, required)    — The URL to fetch
render (boolean, optional) — Render JavaScript with headless browser (default: true, set false for static pages)

map_url

Discover URLs on a website without fetching full content.

url (string, required)                  — Starting URL
limit (number, optional)                — Max URLs to discover (default: 50)
depth (number, optional)                — Link depth to follow (default: 2)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs (e.g. "https://example.com/docs/**")
exclude_patterns (string[], optional)   — Skip matching URLs

crawl_url

Crawl multiple pages and return all content as Markdown.

url (string, required)                  — Starting URL
limit (number, optional)                — Max pages to crawl (default: 10)
depth (number, optional)                — Link depth (default: 1)
include_subdomains (boolean, optional)  — Follow subdomain links (default: false)
include_external_links (boolean, optional) — Follow external links (default: false)
include_patterns (string[], optional)   — Only visit matching URLs
exclude_patterns (string[], optional)   — Skip matching URLs
render (boolean, optional)              — Render JavaScript (default: true)

Cloudflare Plan Limits

Free

Paid

Browser time

10 min/day

10 hrs/month

API rate limit

6 req/min

600 req/min

Concurrent browsers

3

10

Max pages per job

100,000

100,000

Max job duration

7 days

7 days

Results available

14 days

14 days

render: false crawls run on Workers instead of a headless browser and do not consume browser time.

License

MIT

Available Tools

3 tools
crawl_urlA

Crawl multiple pages starting from a URL and return all content as Markdown.

Best for: Fetching entire documentation sections, blog archives, or multiple related pages at once.

Not recommended for: Single pages (use scrape_url — it's faster). Large sites without filters (responses can be very large and exceed token limits).

Tip: Use include_patterns to scope the crawl (e.g. "https://example.com/docs/**").

Args: url: The starting URL to crawl. limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth from the starting URL (default: 1). include_subdomains: If true, follows links to subdomains. include_external_links: If true, follows links to external domains. include_patterns: Only visit URLs matching these wildcard patterns. exclude_patterns: Skip URLs matching these wildcard patterns. render: If true (default), renders JavaScript. Set false for faster static fetch.

Inputs: url (required), depth, limit, render, +4 more. Returns: result

map_urlA

Discover URLs on a website without fetching full page content.

Returns a list of URLs found by crawling from the starting URL. Use this to find the right page before scraping it.

Best for: Finding documentation pages, locating specific content on a site, understanding site structure before scraping.

Typical workflow: map_url to find URLs -> scrape_url on the right page.

Args: url: The starting URL to discover links from. limit: Maximum number of URLs to discover (default: 50, max: 100000). depth: How many links deep to follow (default: 2). include_subdomains: If true, follows links to subdomains. include_external_links: If true, follows links to external domains. include_patterns: Only visit URLs matching these wildcard patterns (e.g. "https://example.com/docs/**"). exclude_patterns: Skip URLs matching these wildcard patterns.

Inputs: url (required), depth, limit, exclude_patterns, +3 more. Returns: result

scrape_urlA

Fetch a single web page and return its content as Markdown.

This is the primary tool for reading web content. Use this when you know which URL contains the information you need.

Best for: Reading documentation pages, articles, blog posts, API references, or any single page where you know the URL.

Not recommended for: When you don't know which page has the info (use map_url first), or when you need content from many pages (use crawl_url).

Args: url: The URL to fetch. render: If true (default), renders JavaScript with a headless browser. Set to false for faster fetch of static pages.

Inputs: url (required), render. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.9/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool has a clear, distinct purpose: scrape_url fetches one page, map_url discovers URLs without fetching content, and crawl_url fetches multiple pages. The descriptions explicitly state when to use each, so there is no ambiguity.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern with underscores: scrape_url, map_url, crawl_url. The style is uniform and predictable, making it easy to infer functionality.

    Tool Count5/5

    Three tools is well-scoped for a crawling server. Each tool covers a distinct core operation—single fetch, URL discovery, and bulk crawl—without unnecessary redundancy or bloat.

    Completeness4/5

    The tools cover the essential crawl lifecycle: discover URLs, scrape a single page, and crawl multiple pages. Minor gaps exist (e.g., no sitemap parsing or headless options for custom headers), but the core workflows are complete.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      MCP server to scrape web pages to clean Markdown via headless Chromium, with support for single or batch URLs.
      MIT
    • F
      license
      Not graded
      quality
      D
      maintenance
      MCP server that fetches and renders dynamic web pages (SPA) using Chromium, returning content as Markdown or text. It uses lightweight HTTP first with automatic fallback to a browser for JS-heavy sites, and includes SSRF protection, login session support, and batch fetching.
      -