Skip to main content
Glama
Crawlbase MCP

Crawlbase MCP

Official
by crawlbase

What is Crawlbase MCP?

Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.

It handles the complexity of scraping for you:

  • JavaScript rendering for modern web apps

  • Proxy rotation & anti-bot evasion

  • Structured outputs (HTML, Markdown, screenshots)

How It Works

  • Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.

  • Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.

  • Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.

Related MCP server: Raccoon AI MCP Server

Setup & Integration

Claude Desktop

  1. Open Claude Desktop → Settings → Developer → Edit Config

  2. Add to claude_desktop_config.json:

  3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Claude Code

Add to your claude.json configuration:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Cursor IDE

  1. Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP

  2. Add to mcp.json:

  3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Windsurf IDE

  1. Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config

  2. Add to mcp_config.json:

  3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

HTTP Transport Mode

For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:

# Clone and install
git clone https://github.com/crawlbase/crawlbase-mcp.git
cd crawlbase-mcp
npm install

# Start HTTP server with tokens (default port: 3000)
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token npm run start:http

# Or with custom port
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token MCP_PORT=8080 npm run start:http

The server exposes:

  • POST /mcp - MCP Streamable HTTP endpoint

  • GET /health - Health check endpoint

Per-Request Token Authentication

HTTP mode supports per-request tokens via headers, allowing multiple users to share a single server:

curl -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "X-Crawlbase-Token: your_token" \
  -H "X-Crawlbase-JS-Token: your_js_token" \
  -d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'

Headers:

  • X-Crawlbase-Token - Normal token for HTML requests

  • X-Crawlbase-JS-Token - JavaScript token for JS-rendered pages/screenshots

Headers override environment variables when provided, enabling multi-tenant deployments.

🔑 Get your free tokens at Crawlbase ↗️.

Usage

Once configured, use these commands inside Claude, Cursor, or Windsurf:

  • crawl → Fetch raw HTML

  • crawl_markdown → Extract clean Markdown

  • crawl_screenshot → Capture screenshots

Example prompts:

  • “Crawl Hacker News and return top stories in markdown.”

  • “Take a screenshot of TechCrunch homepage.”

  • “Fetch Tesla investor relations page as HTML.”

Async Crawling with Cloud Storage

For larger jobs, Crawlbase MCP can push crawl results to Crawlbase Cloud Storage instead of returning them immediately. Your AI agent can then come back later to read, list, or clean up those pages — useful when crawling many URLs at once, revisiting a dataset across sessions, or keeping heavy HTML out of the chat until you actually need it.

Example prompts:

  • “Crawl these 50 product pages and save them to my Crawlbase storage. Once they're saved, summarize each one.”

  • “Save the Hacker News front page to storage so I can analyze it later.”

  • “How many pages do I have stored in Crawlbase right now? Show me the most recent 20.”

  • “Pull back everything I saved yesterday from my Crawlbase storage and give me a report.”

  • “Delete all the pages I have in Crawlbase storage — I'm done with that project.”

Use Cases

  • Market research → Pull live data from competitors, news, and reports

  • E-commerce monitoring → Track products, reviews, and prices in real time

  • News & finance feeds → Keep AI agents up-to-date with live events

  • Autonomous AI agents → Give them vision to act on fresh web data

Resources & Next Steps

Looking to supercharge your AI agents with live web data? Get started here:


MSeeP.ai Security Assessment Badge

Copyright 2026 Crawlbase

Available Tools

9 tools
crawlB

Crawl a URL and return HTML content. Pass store=true to push the result to Crawlbase Cloud Storage instead of returning the body — the response will then contain only the RID and metadata, which can later be retrieved via storage_get.

Inputs: url (required), store, device, country, +4 more.

crawl_markdownA

Crawl a URL and extract clean markdown content. Pass store=true to persist the original page in Cloud Storage and return only the RID + metadata.

Inputs: url (required), store, device, user_agent.

crawl_screenshotA

Take a screenshot of a webpage. Pass store=true to persist the underlying HTML page to Cloud Storage (the screenshot itself is not stored and is still returned as an ephemeral screenshot_url).

Inputs: url (required), mode, store, width, +3 more.

storage_bulk_deleteB

Bulk-delete up to 100 items from Crawlbase Cloud Storage by RID. Irreversible.

Inputs: rids (required), use_js_token.

storage_bulk_getA

Bulk-fetch up to 100 stored items by RID. Bodies are decoded automatically. Default as=metadata_only returns just RID/URL/timestamps to keep context lean; use as=html or as=markdown to include bodies.

Inputs: rids (required), as, auto_delete, use_js_token.

storage_countA

Get the total number of documents in Crawlbase Cloud Storage.

Inputs: use_js_token.

storage_deleteA

Delete a single item from Crawlbase Cloud Storage by RID.

Inputs: rid (required), use_js_token.

storage_getA

Retrieve a single stored page from Crawlbase Cloud Storage by RID or URL. Only works for pages previously crawled with store=true. Returns the raw JSON record by default; pass as=html to return just the body, or as=markdown to convert it.

Inputs: as, rid, url, use_js_token.

storage_listA

List RIDs of pages previously crawled with store=true in Crawlbase Cloud Storage. Supports scroll-based pagination (scroll session expires after 15s of inactivity). Returns at most 1000 per call.

Inputs: limit, scroll, scroll_id, scroll_order, +1 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 9 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: three crawl tools for different output formats (HTML, markdown, screenshot) and six storage tools for managing stored pages (get, list, delete, count, bulk operations). No overlap or ambiguity.

    Naming Consistency5/5

    All tool names follow a consistent pattern: 'crawl_*' for crawling actions and 'storage_*' for storage operations. The verbs are clear and the prefix clearly indicates the domain.

    Tool Count5/5

    9 tools is well-scoped for a web crawling and storage server. It covers crawling with three output formats and full CRUD for storage, without unnecessary bloat or missing essential operations.

    Completeness5/5

    The tool surface provides comprehensive coverage: crawling with HTML, markdown, and screenshot output; storage with create (via store=true), read (single, bulk, list), delete (single, bulk), and count. No obvious gaps for the intended purpose.

    Maintenance

    ActivityInactive
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers