Crawlbase MCP
OfficialClick on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Crawlbase MCP Crawl Hacker News and return top stories as markdown."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
What is Crawlbase MCP?
Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.
It handles the complexity of scraping for you:
JavaScript rendering for modern web apps
Proxy rotation & anti-bot evasion
Structured outputs (HTML, Markdown, screenshots)
How It Works
Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.
Related MCP server: Raccoon AI MCP Server
Setup & Integration
Claude Desktop
Open Claude Desktop → Settings → Developer → Edit Config
Add to
claude_desktop_config.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Claude Code
Add to your claude.json configuration:
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Cursor IDE
Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
Add to
mcp.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}Windsurf IDE
Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
Add to
mcp_config.json:Replace
your_token_hereandyour_js_token_herewith the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}HTTP Transport Mode
For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:
# Clone and install
git clone https://github.com/crawlbase/crawlbase-mcp.git
cd crawlbase-mcp
npm install
# Start HTTP server with tokens (default port: 3000)
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token npm run start:http
# Or with custom port
CRAWLBASE_TOKEN=your_token CRAWLBASE_JS_TOKEN=your_js_token MCP_PORT=8080 npm run start:httpThe server exposes:
POST /mcp- MCP Streamable HTTP endpointGET /health- Health check endpoint
Per-Request Token Authentication
HTTP mode supports per-request tokens via headers, allowing multiple users to share a single server:
curl -X POST http://localhost:3000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "X-Crawlbase-Token: your_token" \
-H "X-Crawlbase-JS-Token: your_js_token" \
-d '{"jsonrpc": "2.0", "method": "tools/list", "id": 1}'Headers:
X-Crawlbase-Token- Normal token for HTML requestsX-Crawlbase-JS-Token- JavaScript token for JS-rendered pages/screenshots
Headers override environment variables when provided, enabling multi-tenant deployments.
🔑 Get your free tokens at Crawlbase ↗️.
Usage
Once configured, use these commands inside Claude, Cursor, or Windsurf:
crawl → Fetch raw HTML
crawl_markdown → Extract clean Markdown
crawl_screenshot → Capture screenshots
Example prompts:
“Crawl Hacker News and return top stories in markdown.”
“Take a screenshot of TechCrunch homepage.”
“Fetch Tesla investor relations page as HTML.”
Async Crawling with Cloud Storage
For larger jobs, Crawlbase MCP can push crawl results to Crawlbase Cloud Storage instead of returning them immediately. Your AI agent can then come back later to read, list, or clean up those pages — useful when crawling many URLs at once, revisiting a dataset across sessions, or keeping heavy HTML out of the chat until you actually need it.
Example prompts:
“Crawl these 50 product pages and save them to my Crawlbase storage. Once they're saved, summarize each one.”
“Save the Hacker News front page to storage so I can analyze it later.”
“How many pages do I have stored in Crawlbase right now? Show me the most recent 20.”
“Pull back everything I saved yesterday from my Crawlbase storage and give me a report.”
“Delete all the pages I have in Crawlbase storage — I'm done with that project.”
Use Cases
Market research → Pull live data from competitors, news, and reports
E-commerce monitoring → Track products, reviews, and prices in real time
News & finance feeds → Keep AI agents up-to-date with live events
Autonomous AI agents → Give them vision to act on fresh web data
Resources & Next Steps
Looking to supercharge your AI agents with live web data? Get started here:
✍️ Learn More – See how MCP powers AI agents with real-time web data ↗️
🌐 Crawlbase Website – Get free tokens & start crawling today ↗️

Copyright 2026 Crawlbase
Available Tools
9 toolscrawlB
Crawl a URL and return HTML content. Pass store=true to push the result to Crawlbase Cloud Storage instead of returning the body — the response will then contain only the RID and metadata, which can later be retrieved via storage_get.
Inputs: url (required), store, device, country, +4 more.
crawl_markdownA
Crawl a URL and extract clean markdown content. Pass store=true to persist the original page in Cloud Storage and return only the RID + metadata.
Inputs: url (required), store, device, user_agent.
crawl_screenshotA
Take a screenshot of a webpage. Pass store=true to persist the underlying HTML page to Cloud Storage (the screenshot itself is not stored and is still returned as an ephemeral screenshot_url).
Inputs: url (required), mode, store, width, +3 more.
storage_bulk_deleteB
Bulk-delete up to 100 items from Crawlbase Cloud Storage by RID. Irreversible.
Inputs: rids (required), use_js_token.
storage_bulk_getA
Bulk-fetch up to 100 stored items by RID. Bodies are decoded automatically. Default as=metadata_only returns just RID/URL/timestamps to keep context lean; use as=html or as=markdown to include bodies.
Inputs: rids (required), as, auto_delete, use_js_token.
storage_countA
Get the total number of documents in Crawlbase Cloud Storage.
Inputs: use_js_token.
storage_deleteA
Delete a single item from Crawlbase Cloud Storage by RID.
Inputs: rid (required), use_js_token.
storage_getA
Retrieve a single stored page from Crawlbase Cloud Storage by RID or URL. Only works for pages previously crawled with store=true. Returns the raw JSON record by default; pass as=html to return just the body, or as=markdown to convert it.
Inputs: as, rid, url, use_js_token.
storage_listA
List RIDs of pages previously crawled with store=true in Crawlbase Cloud Storage. Supports scroll-based pagination (scroll session expires after 15s of inactivity). Returns at most 1000 per call.
Inputs: limit, scroll, scroll_id, scroll_order, +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 9 tools
Each tool has a clearly distinct purpose: three crawl tools for different output formats (HTML, markdown, screenshot) and six storage tools for managing stored pages (get, list, delete, count, bulk operations). No overlap or ambiguity.
All tool names follow a consistent pattern: 'crawl_*' for crawling actions and 'storage_*' for storage operations. The verbs are clear and the prefix clearly indicates the domain.
9 tools is well-scoped for a web crawling and storage server. It covers crawling with three output formats and full CRUD for storage, without unnecessary bloat or missing essential operations.
The tool surface provides comprehensive coverage: crawling with HTML, markdown, and screenshot output; storage with create (via store=true), read (single, bulk, list), delete (single, bulk), and count. No obvious gaps for the intended purpose.
Maintenance
Related MCP Connectors
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
Official Browserless MCP server: browser automation, scraping, screenshots and PDFs for AI agents.
Proxy-rotated web scraping and cross-platform intelligence for AI agents.
Cloud browser, web extraction, screenshots, and remote CDP tools for AI agents.
Related MCP Servers
- FlicenseNot gradedqualityDmaintenanceA server that enables AI systems to browse, retrieve content from, and interact with web pages through the Model Context Protocol.1-

Raccoon AI MCP Serverofficial
FlicenseNot gradedqualityDmaintenanceA Model Context Protocol server that enables web browsing, data extraction, and complex web tasks automation through Raccoon AI's LAM API.-- AlicenseBqualityDmaintenanceA Model Context Protocol server that enables LLMs to interact with web pages, take screenshots, generate test code, scrape web pages, and execute JavaScript in a real browser environment.2922MIT
- AlicenseBqualityDmaintenanceA Model Context Protocol server that enables AI assistants to control a real web browser with stealth capabilities, avoiding bot detection while performing tasks like clicking, filling forms, taking screenshots, and extracting data.1156 npm26MIT