Spicrawl MCP Server
Spicrawl MCP server gives AI agents web-scraping tools to read, render, extract and batch-fetch websites, plus browser sessions, usage tracking and docs lookup.
Scrape one URL (
spicrawl_scrape) as markdown (default), text, HTML, JSON (content + heading sections) or PDF, with caching, tag include/exclude, link discovery and cost caps.Render JavaScript pages with
render: true,mode: "auto"escalation,wait_forselectors, or block resources for speed.Extract structured data via CSS/XPath selector maps (
extract), page-published metadata (autoparse: JSON-LD, OpenGraph, Twitter Card, microdata), or record network responses (network_capture).Drive pages before capture with typed actions: click, fill, wait_for, wait_for_navigation, scroll (incl. infinite scroll), select, evaluate, screenshot.
Get screenshots and PDFs back as image/resource content blocks the model can see.
Batch scrape up to 10,000 URLs per async job: submit, status, paged results, single-item content, retry, cancel, append to open jobs, close.
Keep logged-in browser state with persistent sessions (cookies, storage, fixed rendering tier): create, list, get, dump context, release, delete.
Drive a remote browser over CDP: mint a connect URL, hand control to a human via a live-view link (logins, OTPs, captchas), query/take/release control.
Track cost and debug with usage breakdowns, period summaries, daily reconciliation, and request list/detail logs (status, engine, latency, errors, credits).
Look up the Spicrawl docs in-agent: search, read a page as Markdown, or list the full
llms.txtindex.
Provides the (announced, not yet available) capability to drive a Spicrawl-hosted remote browser from Puppeteer by minting a single-use CDP WebSocket URL, so agents can attach Puppeteer to a persistent Spicrawl browser session.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Spicrawl MCP Server scrape example.com/pricing as markdown and list each plan with its price"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Spicrawl MCP Server: web scraping for Claude, Cursor and AI agents
To let Claude, Cursor or another AI agent scrape websites, connect the Spicrawl MCP server. Add the hosted endpoint https://mcp.spicrawl.com/mcp (or run npx -y @spicrawl/mcp locally) with a Spicrawl API key, and the agent can call spicrawl_scrape to read any URL as Markdown, text, HTML or JSON. It renders JavaScript pages, extracts structured data, scrapes up to 10,000 URLs in one batch job, takes screenshots and keeps logged-in browser sessions, all as native spicrawl_* tools in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex. The free tier includes 1,000 credits per month.
Docs · Get an API key · TypeScript SDK · CLI · Agent plugins · Changelog
Install from: npm · Smithery · Glama · MCP Registry · Dify plugin · Cursor Directory
Quickstart
Create an API key at app.spicrawl.com. Keys start with
spicrawl_live_orspicrawl_test_.Connect the hosted endpoint,
https://mcp.spicrawl.com/mcp(MCP Streamable HTTP, nothing to install):export SPICRAWL_API_KEY=spicrawl_live_... claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \ --header "Authorization: Bearer $SPICRAWL_API_KEY"Codex:
codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEYOr run it locally over stdio (Node.js 20+), for clients that only launch local servers:
claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcpAsk your agent: "Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price." It calls
spicrawl_scrapeand gets the page back as Markdown.
Both transports serve the same 27 spicrawl_* tools (see Tools), and every call runs under your key with the same scopes, limits and credits as a direct API call.
Related MCP server: snoopscan
What it does
Scrape any URL to Markdown, text, HTML, a JSON envelope or a printed PDF with one tool,
spicrawl_scrape. Markdown is the default and strips the page to its main content.format: "json"returns that main content as Markdown incontent, plussections({level, heading, text}split on headings); useformat: "html"for the raw document.Render JavaScript with
render: true(3 credits). A plain fetch costs 1 credit, screenshots, PDFs andactionsuse a full browser (8 credits), andmode: "auto"escalates from fetch to browser and bills only the rung that worked (price table). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.Extract structured data with a CSS/XPath selector map (
extract) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (autoparse). A trailing[]on a selector returns every match as an array, e.g.{"titles": ".titleline > a[]"}; without it only the first match is returned.Drive the page before capture with 8 typed browser actions:
click,fill,wait_for,wait_for_navigation,scroll(including infinite scroll),select,evaluateandscreenshot.Return screenshots and PDFs the model can see: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as
application/pdfresource blocks (up to 10 MB).Batch scrape thousands of URLs as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open so you can append URLs as you discover them.
Keep a logged-in browser with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.
Answer "what did that cost?" from usage and request-log tools, and read the Spicrawl docs from inside the agent.
Use cases
Scrape docs into a RAG knowledge base. Read documentation pages as clean Markdown with
spicrawl_scrape(main_content_onlyis on by default for Markdown), or queue up to 10,000 URLs in onespicrawl_batch_submitjob. Spicrawl returns the Markdown; chunking and embedding are yours. See the knowledge-base recipe.Monitor prices. Pull a product's title, price and availability with an
extractselector map orautoparse, and setcache: falsefor a fresh reading. A plain-fetch check costs 1 credit, a rendered one 3. Run it on your own schedule. See price monitoring.Extract job postings. Collect listings from job boards, including infinite-scroll pages (
render: truewith ascrollaction), as structured fields. See job postings.Read JavaScript-heavy pages. When a plain fetch returns an empty shell, call
spicrawl_scrapewithrender: true(3 credits) so the page builds itself first.
Check each site's terms and robots.txt before you scrape it.
Which tool to use
If you need one page's content or data now, use
spicrawl_scrape.If the page is blank or a skeleton without JavaScript, call
spicrawl_scrapeagain withrender: true.If you know the page's structure, use
extractwith CSS selectors; if you don't, tryautoparse.If you have more than about 20 URLs, use
spicrawl_batch_submit, thenspicrawl_batch_statusandspicrawl_batch_results.If you discover URLs as you go, submit with
open: true, append withspicrawl_batch_add_itemsand finish withspicrawl_batch_close.If you must log in first, use
spicrawl_session_createand pass itsidassession_idtospicrawl_scrape.If a call failed and you need to know why, use
spicrawl_request_getwith the request id, orspicrawl_requests_listwithonly_errors.If you are unsure of a parameter or an error code, use
spicrawl_docs_searchbefore guessing.
Tools
27 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a title and explicit readOnlyHint, destructiveHint and openWorldHint annotations, so a client knows which calls to confirm: spicrawl_scrape is not read-only (its method and actions can submit forms), and spicrawl_batch_cancel, spicrawl_session_release and spicrawl_session_delete are destructive. Every tool also declares an outputSchema and returns matching structuredContent beside the same text it always returned, so a client can read fields (credits, status, next_cursor, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (spicrawl_docs_read, spicrawl_batch_task_content) puts it under content in structuredContent. Full argument reference: docs.spicrawl.com/agents/mcp.
Scrape a website to Markdown
Tool | What it does | Key parameters |
| Retrieve one URL as |
|
Batch scrape many URLs
Tool | What it does | Key parameters |
| Queue many URLs as one async job with shared settings; returns the job with |
|
| List the project's batch jobs, newest first. |
|
| One job's status and progress. |
|
| A page of finished items; works while the job runs. |
|
| The full payload of one item. |
|
| Cancel a job; finished items keep their results. |
|
| Re-queue every failed item. |
|
| Append URLs to a job submitted with |
|
| Stop an open job accepting items so it can complete. |
|
Browser sessions
Tool | What it does | Key parameters |
| Create a persistent browser identity (cookies, storage, a fixed rendering tier). |
|
| List sessions, newest first. |
|
| One session's status, rendering tier, exit, usage and expiry. |
|
| Dump a live session's cookies and storage (contains secrets). |
|
| End a session; its record stays as |
|
| Permanently delete a session and its state. |
|
Remote browser
Tool | What it does | Key parameters |
| Mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from your own CDP client. Returns its |
|
| A link a person opens to watch that browser and take control: for logins, one-time codes and captchas. Single use, open within 60 s. |
|
| Who drives the browser, |
|
While a person has control, the agent's input and navigation commands fail with CDP error -32001 and reading the page still works. When they give control back, the agent carries on in the same tab with the same cookies.
Usage and request history
Tool | What it does | Key parameters |
| Billing-grade usage over a date window ( |
|
| Current-period totals ( | none |
| One day's drift between the request log and the billed rollup ( |
|
| Recent API calls with status, error code, rendering tier, latency and credits. |
|
| The full log record of one call. |
|
Docs
Tool | What it does | Key parameters |
| Search the Spicrawl docs; an error code also links to its entry. |
|
| Read one docs page as Markdown (truncated over 60,000 characters). |
|
| The docs' | none |
Coming soon
The schemas accept ai_extract, stealth, extract_preset, premium_proxy, proxy_country and sticky_key ahead of launch. Agents should not send them yet. To list only what works today, set SPICRAWL_MCP_HIDE_UNAVAILABLE=1 (see Environment variables).
Example prompts
"Scrape https://example.com/pricing as markdown and summarise the plans."
"Get the product name, price and stock status from this URL." (
spicrawl_scrapewithextractorautoparse)"Take a full-page screenshot of https://example.com after the cookie banner is gone." (
render,actions,screenshot_fullpage)"Scrape these 500 URLs and give me each page title." (
spicrawl_batch_submit, thenspicrawl_batch_results)"Why did my last scrape fail?" (
spicrawl_requests_listwithonly_errors, thenspicrawl_request_get)"How many credits have I used this month?" (
spicrawl_usage_summary)
Set up your MCP client
Put your key in SPICRAWL_API_KEY and reference it from the config where the client allows, so the key never lands in a committed file.
Hosted endpoint, stored for you only (add --scope user for every project):
claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
--header "Authorization: Bearer $SPICRAWL_API_KEY"To share it with a team, commit .mcp.json at the project root. Claude Code expands ${VAR}, so each teammate's own key is used:
{
"mcpServers": {
"spicrawl": {
"type": "http",
"url": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
}
}
}Or install the plugin, which bundles the MCP server and the Spicrawl agent skill:
claude plugin marketplace add OfficialSpicrawl/agent-plugins
claude plugin install spicrawl@spicrawl-pluginsCheck with claude mcp list, or /mcp in a session. More: docs.spicrawl.com/agents/claude-code.
Claude Desktop's claude_desktop_config.json starts local (stdio) servers only, and custom connectors cannot send an Authorization header, so run the package locally. The file is at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows:
{
"mcpServers": {
"spicrawl": {
"command": "npx",
"args": ["-y", "@spicrawl/mcp"],
"env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
}
}
}Restart Claude Desktop after saving.
Use the Add to Cursor button above, or add this to .cursor/mcp.json in the project (or ~/.cursor/mcp.json for every project). Cursor resolves ${env:NAME} in url and headers, so start Cursor from an environment where SPICRAWL_API_KEY is set:
{
"mcpServers": {
"spicrawl": {
"url": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
}
}
}More: docs.spicrawl.com/agents/cursor.
Use the Install in VS Code button above, or add this to .vscode/mcp.json. VS Code uses the servers key, and the input variable makes it ask for the key once and store it securely, so the file is safe to commit:
{
"inputs": [
{ "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
],
"servers": {
"spicrawl": {
"type": "http",
"url": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
}
}
}VS Code does not expand ${env:...} inside headers, so use the input variable. More: docs.spicrawl.com/agents/github-copilot.
Windsurf is now named Devin Desktop. For the default Devin Local agent, add the server in local scope (saved to the gitignored .devin/mcp_config.local.json):
devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"For the legacy Cascade agent, edit ~/.config/devin/mcp_config.json (%APPDATA%\devin\mcp_config.json on Windows). Cascade reads the URL from serverUrl and replaces ${env:...}:
{
"mcpServers": {
"spicrawl": {
"serverUrl": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
}
}
}More: docs.spicrawl.com/agents/windsurf.
Gemini CLI strips variables whose names contain KEY, TOKEN or AUTH before expanding headers, so export the key under another name:
export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"Then add to ~/.gemini/settings.json (or .gemini/settings.json for one project). Streamable HTTP uses the httpUrl key:
{
"mcpServers": {
"spicrawl": {
"httpUrl": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
}
}
}Or: gemini mcp add --scope user --transport http --header 'Authorization: Bearer $SPICRAWL_MCP_BEARER' spicrawl https://mcp.spicrawl.com/mcp. Run /mcp to check. More: docs.spicrawl.com/agents/gemini-cli.
Codex reads the key from the environment each time it connects, so the key never lands in a file:
codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEYThe equivalent ~/.codex/config.toml (or .codex/config.toml in a trusted project):
[mcp_servers.spicrawl]
url = "https://mcp.spicrawl.com/mcp"
bearer_token_env_var = "SPICRAWL_API_KEY"Local stdio instead:
[mcp_servers.spicrawl]
command = "npx"
args = ["-y", "@spicrawl/mcp"]
env = { SPICRAWL_API_KEY = "spicrawl_live_..." }More: docs.spicrawl.com/agents/codex.
A gateway that passes a connection parameter through as a header cannot always send Authorization: Bearer <key>: Smithery maps a parameter to a header but sends the raw value, with no Bearer prefix, and a client often cannot name its source authorization. /mcp takes the key in any of these headers:
Header | Value |
|
|
|
|
|
|
In Smithery, make the API key a connection parameter, map it to the x-api-key header, and mark it secret. Or map it to Authorization and send the raw key; the server accepts that when the whole value is a Spicrawl key.
The key can be left optional in the connection schema (no required): initialize, ping and tools/list on /mcp answer without a key, with the same tool list as the server card, so a directory can list the tools. Every tool call still needs a valid key and is refused with 401 otherwise; no session is opened and nothing is billed or read without one. Say in the parameter's description that it is needed to run any tool.
If more than one is sent, the first that holds a well-formed key wins, in the order Authorization: Bearer, raw Authorization, x-api-key, api-key. A key in the query string is not read, because URLs end up in access logs. /chatgpt/mcp takes OAuth access tokens only and accepts none of these.
Setting | Value |
Transport | Streamable HTTP |
URL |
|
Header |
|
A stdio-only client runs npx -y @spicrawl/mcp with SPICRAWL_API_KEY in its environment. The server accepts API keys only, not OAuth. Setup pages for more clients (OpenCode, Zed, JetBrains, Cline, Goose and others): docs.spicrawl.com/agents/mcp.
Environment variables
Variable | Default | Used by | Purpose |
| none (required) | stdio | Your API key. The HTTP server ignores it and uses each caller's bearer token. |
|
| both | The Spicrawl API the tools call. |
|
| both | The API as the end user reaches it, used only in URLs handed back. |
|
| both | Docs base URL for the docs tools. Your key is never sent there. |
| none | both | Legacy: an origin whose docs are at |
|
| HTTP | Listen address, |
|
| HTTP | Idle MCP sessions are dropped after this long. |
| off | both |
|
| off | HTTP |
|
|
| HTTP | The OAuth resource identifier. Tokens must be issued for exactly this |
|
| HTTP | The authorization server, listed in the metadata. Tokens are introspected at |
| none | HTTP | This server's bearer secret at the introspection endpoint. Unset, every token is answered |
| off | HTTP | Also serve the metadata at the root |
| none | HTTP | The OpenAI domain-verification token served at |
| none | HTTP | A file holding that token (surrounding whitespace and the trailing newline are dropped). Read when |
With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at <SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs.
Self-host the MCP server over HTTP
The hosted endpoint runs this package's Streamable HTTP server. To run your own:
SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
SPICRAWL_BASE_URL=https://api.spicrawl.com \
npx -y @spicrawl/mcp --httpSPICRAWL_BASE_URLdefaults tohttp://127.0.0.1:8080in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it tohttps://api.spicrawl.comto front the public API.MCP is served at
POST/GET/DELETE /mcp; liveness atGET /healthz(no auth).GET /.well-known/mcp/server-card.json(no auth,Cache-Control: public, max-age=3600) is a static server card for directories such as Smithery that cannot scan/mcpwithout a key:serverInfo, the transport, bearer auth and the tool list. The tools are read from the same definitions astools/list, as ifSPICRAWL_MCP_HIDE_UNAVAILABLEwere on: every tool, without the arguments marked "Coming soon".Without any key,
/mcpanswers only discovery:initialize,ping, theinitializednotification andtools/list(the same tool list as the server card; noMcp-Session-Idis issued and nothing is sent upstream). Any other request, atools/callincluded, is401, as is a request that sends a malformed key or names a session. A session-lessGETis405.Each caller sends their own key as
Authorization: Bearer spicrawl_live_..., or, for a gateway that cannot send theBearerprefix, as a rawAuthorization: spicrawl_live_...,x-api-keyorapi-keyheader (see "Smithery and other gateways" above; never a query parameter). The key is checked against the API before a session opens (401if refused,503withRetry-Afterif the API cannot be reached), and a session is pinned to the key that opened it.The server speaks plain HTTP. Put it behind TLS before sending keys across a network.
ChatGPT endpoint (OAuth)
The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless SPICRAWL_CHATGPT_ENABLED is on, and /mcp is unchanged either way.
Route | What it does |
| MCP Streamable HTTP, stateless: no |
| Protected-resource metadata (RFC 9728): |
| The same document, only with |
| The verification token as |
Authentication takes OAuth access tokens (spicrawl_oat_…) only:
No token:
401withWWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch".A bad, inactive, expired or wrong-audience token, or an API key (
spicrawl_live_…/spicrawl_test_…, never accepted here): the same401pluserror="invalid_token", error_description="…".A tool call the token's scopes do not cover:
403witherror="insufficient_scope".spicrawl_scrapeneedsscrape, the batch tools needbatch, the docs tools need only a valid token.The authorization server unreachable or refusing this server's secret:
503withRetry-After.
Each token is checked with POST <issuer>/api/oauth/introspect (Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>, form body token=…). Only active: true with aud equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its exp. Tools run upstream under the api_key the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose _meta["mcp/www_authenticate"] carries a fresh invalid_token challenge, and the token is dropped from the cache.
The endpoint lists six tools, each with explicit readOnlyHint/destructiveHint/openWorldHint, a title, and securitySchemes: [{type: "oauth2", scopes: [...]}] (also in _meta.securitySchemes):
Tool | Scopes | Arguments |
|
|
|
|
|
|
|
|
|
|
|
|
| none | As on |
spicrawl_scrape is read-only here, unlike on /mcp: this endpoint only sends a GET, with no method, body, headers or actions. spicrawl_batch_submit is not read-only, because it creates a job.
spicrawl_batch_results fills in the content of a succeeded item that the results page lists without it, which happens while the job is still finishing, from GET /v1/batch/{id}/tasks/{seq}/content. It fetches at most 25 items and 2 MiB per page. An item that can't be read yet comes back unavailable, and one past the size cap omitted.
Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.
Installed globally (npm install -g @spicrawl/mcp), the command is spicrawl-mcp (alias spicrawl-mcp-server), with --http, --version and --help.
FAQ
Which MCP server scrapes JavaScript sites?
The Spicrawl MCP server does. Call spicrawl_scrape with render: true (3 credits) and the page loads in a browser, runs its scripts and comes back as finished Markdown, text, HTML or JSON. Add wait_for with a CSS selector for content that appears late, or set mode: "auto" to start with a plain fetch and move up to a stronger rendering only when needed. Spicrawl picks the rendering for each request and bills the tier that served it (fetch, js or browser, reported as engine in the JSON result); there is no argument to choose it.
How do I give Claude web access?
Connect the Spicrawl MCP server to Claude with an API key. In Claude Code:
claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
--header "Authorization: Bearer $SPICRAWL_API_KEY"Claude can then read any URL you give it by calling spicrawl_scrape. Spicrawl reads pages by URL; it is not a search engine. For Claude Desktop, see the install answer below.
Is there a free tier?
Yes. During the beta every organization is on one plan of 1,000 credits per month. A plain fetch costs 1 credit, JavaScript rendering 3, and full-browser rendering (screenshots, PDFs, actions) 8. Failed requests cost 0, and a cache hit is billed at the same price as the fetch that stored it. When the month's credits run out, calls fail with ERR::LIMIT::QUOTA_EXCEEDED until it resets. See Credits.
Does it handle bot-protected pages?
Only with your own proxy. Pass it in proxy (an http:// URL; residential proxies work best). When a site answers with a Cloudflare managed challenge, Spicrawl detects it, moves the request to full-browser rendering on the same proxy exit and returns the real page, for 8 credits. If every step is challenged you get ERR::UPSTREAM::CHALLENGE and pay 0. Success is likely, not guaranteed, and Spicrawl does not solve CAPTCHAs. On the default shared pool a challenged page returns that error. See Scrape Cloudflare-protected pages.
How do I install it in Cursor or Claude Desktop?
Cursor: click Add to Cursor above, or add this to .cursor/mcp.json and start Cursor with SPICRAWL_API_KEY set:
{
"mcpServers": {
"spicrawl": {
"url": "https://mcp.spicrawl.com/mcp",
"headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
}
}
}Claude Desktop starts local (stdio) servers only, so run the package. Add this to claude_desktop_config.json and restart the app:
{
"mcpServers": {
"spicrawl": {
"command": "npx",
"args": ["-y", "@spicrawl/mcp"],
"env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
}
}
}Get a key at app.spicrawl.com. Steps for Claude Code, VS Code, Windsurf, Gemini CLI and Codex are under Set up your MCP client.
Can it crawl a whole website?
There is no single crawl tool. Scrape a page with links: true to get its links, then feed them to an open batch job (open: true and spicrawl_batch_add_items).
Other questions
Is there a hosted Spicrawl MCP server?
Yes: https://mcp.spicrawl.com/mcp, MCP Streamable HTTP, authenticated with Authorization: Bearer <API key>, so there is nothing to install.
Does it support OAuth?
Not on /mcp: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a 401. The separate ChatGPT endpoint at /chatgpt/mcp takes OAuth access tokens only, when the operator turns it on.
Should I use the MCP server, the SDK or the CLI?
Use the MCP server when an AI agent should call Spicrawl as tools. Use @spicrawl/sdk in your own TypeScript or JavaScript code, @spicrawl/cli in a terminal or shell script, and the REST API from any other language.
Troubleshooting
Symptom | Fix |
HTTP | The header is missing, or the key is malformed, unknown, revoked or expired. Check that the variable is set in the environment the client started from. A gateway that cannot send |
The header contains a literal | The client did not expand it. Use |
HTTP | The server could not reach the API to check your key. Retry; your key is fine. |
| Grant the key the |
| The monthly credit allowance is used up. Not retryable until it resets. |
| A plain fetch got a near-empty, script-driven page (a bot challenge stub), so nothing was billed. Retry with |
"Unknown argument" on | Use the tool's names: |
Tools do not appear | Restart the client, and check the config key ( |
| Install Node.js 20 or later. |
curl -i https://mcp.spicrawl.com/mcp without a key answers 401: the endpoint is reachable.
Related packages
@spicrawl/sdk: the official TypeScript SDK for the Spicrawl API.@spicrawl/cli: the Spicrawl command-line interface.OfficialSpicrawl/agent-plugins: plugins that bundle this server and the Spicrawl skill for Claude Code, Codex, Cursor, Gemini CLI and more.
Links
Documentation: docs.spicrawl.com/agents/mcp
Docs for LLMs: docs.spicrawl.com/llms.txt
Dashboard and API keys: app.spicrawl.com
Security: SECURITY.md. Never commit an API key; revoke a leaked one at app.spicrawl.com.
Publishing to the MCP Registry
server.json lists both the npm/stdio package and the hosted
Streamable HTTP endpoint under io.github.OfficialSpicrawl/mcp. API keys are
requested from the person installing the server; no key belongs in the manifest.
The existing release workflow publishes to the MCP Registry after its npm
job succeeds, on pushes to main or manual runs from main. Registry authentication
uses GitHub OIDC (id-token: write), so it needs no new secret, device login or DNS
record. The npm release keeps using the existing NPM_TOKEN secret.
For a release, run npm version --no-git-tag-version X.Y.Z, update the changelog,
and submit the changes to main. The npm version hook synchronizes both versions
in server.json. If you edit package.json by hand, run npm run registry:sync.
CI checks the manifest against its official $schema with Ajv and rejects name
or version drift before npm publication. To run that check locally:
npm ci
npm run registry:validate
# Offline: pass a downloaded copy of the exact server.json $schema URL.
npm run registry:validate -- /path/to/server.schema.jsonThe registry job checks that the published npm package has the matching mcpName
before publishing. An existing registry version is skipped; lookup failures stop
the job. To retry a registry failure or list the already-published current npm
version, run the release workflow on main again. Use a new version for metadata
changes to an existing listing. The publisher binary is pinned and its download
is checked against the release checksums; update its version in release.yml
when upgrading the CLI.
Verify publication at
https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.OfficialSpicrawl/mcp.
The workflow must be merged into OfficialSpicrawl/mcp before OIDC can publish
that namespace. Fork pull requests validate and test, but never run the registry job.
Official references: publishing quickstart, GitHub Actions, and remote servers.
License
Available Tools
27 toolsspicrawl_batch_add_itemsAdd URLs to an open batch jobAInspect
Append URLs to a batch job that was submitted with open: true. Same item vocabulary as spicrawl_batch_submit (EITHER urls or items, not both, plus shared settings for these new items). Fails with CONFLICT if the job is closed or finished. Not idempotent: calling twice adds the URLs twice. Returns { job, items_added, items_dispatched }.
Inputs: job_id (required), urls, wait, items, +9 more. Returns: job, items_added, items_dispatched
spicrawl_batch_cancelCancel a batch jobADestructiveIdempotentInspect
Cancel a batch job: items not yet started are cancelled and will never run; items already finished keep their results; items already running finish first, so the job comes back cancelling and moves to cancelled when they drain. Cannot be undone — resubmit to run the URLs again. Returns the job object.
Inputs: job_id (required). Returns: id, name, open, params, +20 more
spicrawl_batch_closeClose an open batch jobAIdempotentInspect
Stop an open batch job accepting items, so it completes once its queued work drains. Idempotent (closing a closed job succeeds). Returns the job object.
Inputs: job_id (required). Returns: id, name, open, params, +20 more
spicrawl_batch_listList batch jobsARead-onlyIdempotentInspect
List this API key's project's batch jobs, newest first, with status and progress. Returns { batches: [...], next_cursor? }; pass next_cursor back as cursor for the next page.
Inputs: limit, cursor, status. Returns: batches, next_cursor
spicrawl_batch_resultsRead a batch job's resultsARead-onlyIdempotentInspect
Return a page of FINISHED items of a batch job (one record per item: seq, url, external_id, status, error, and content, or a result_url API path (GET /v1/batch/{id}/tasks/{seq}/content, same key) for large bodies). Unfinished items are not listed yet. Returns { results: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Results expire after the job's retention window (410 Gone).
Inputs: job_id (required), limit, cursor, status. Returns: results, next_cursor
spicrawl_batch_retryRetry a batch job's failed itemsAInspect
Re-queue every FAILED item of a batch job so it runs again (succeeded items are untouched; a job with no failures is returned unchanged). Returns { job, items_reset, items_dispatched }.
Inputs: job_id (required). Returns: job, items_reset, items_dispatched
spicrawl_batch_statusCheck a batch job's statusARead-onlyIdempotentInspect
Return one batch job with its status and progress (total/completed/succeeded/failed/remaining, credits charged). Poll this after spicrawl_batch_submit; once status is completed, failed or cancelled, read spicrawl_batch_results.
Inputs: job_id (required). Returns: id, name, open, params, +20 more
spicrawl_batch_submitSubmit a batch scrape jobAInspect
Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER urls or items, not both (at least one URL, unless open is true). Returns the job object (id, status, estimated_credits, progress). Poll with spicrawl_batch_status, read with spicrawl_batch_results or spicrawl_batch_task_content. Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs. Set open: true to keep the job accepting more URLs via spicrawl_batch_add_items; an open job never finishes until you call spicrawl_batch_close.
Inputs: name, open, urls, wait, +16 more. Returns: id, name, open, params, +20 more
spicrawl_batch_task_contentRead one batch item's contentARead-onlyIdempotentInspect
Return the full payload one item of a batch job produced, by its seq (items are numbered from 0 in submission order). Cheaper than paging results when you need one URL's output. A CONFLICT error means either the item has not finished yet (poll spicrawl_batch_status and ask again) or it finished as failed (it will never produce content; the message says which). NOT_FOUND means no such seq.
Inputs: seq (required), job_id (required). Returns: content
spicrawl_browser_connect_urlBrowser (CDP) connect URLAInspect
Mints a single-use connect URL for Spicrawl's remote browser (GET /v1/browser, a DevTools Protocol (CDP) WebSocket) so YOUR code can drive it with any CDP client, using the URL as the browser WebSocket endpoint. It calls POST /v1/browser/token and returns a ws:// or wss:// URL carrying a short-lived token, never the API key: the URL works ONCE, within 60 seconds, and only with the options given here (editing them in the URL is refused), so call this right before connecting and again for every new session. Prefer sending the key as an Authorization header when the client supports headers on the WebSocket upgrade: then connect to the plain /v1/browser URL with 'Authorization: Bearer ' and no token is needed. The key must carry the browser scope. Parameter errors are reported when minting. The result's session_id names the session once connected: when a page needs a person (a login, a one-time code, a captcha), pass it to spicrawl_browser_live_view for a link they can open, and to spicrawl_browser_control to see when they hand control back.
Inputs: proxy, engine, sticky_key, session_ttl, +2 more. Returns: url, notes, expires_at, session_id, +1 more
spicrawl_browser_controlWho controls a remote browserAIdempotentInspect
Reads or changes who drives a connected remote browser: agent (your CDP connection) or human (a person in its live view, see spicrawl_browser_live_view). status reads it: poll it after sending a live-view link, and continue once controller is agent again. takeover hands control to the person before they ask (your input and navigation then fail with error -32001); release takes it back. The person can also take and give back control from the live view themselves.
Inputs: session_id (required), action. Returns: status, controller, expires_at, session_id
spicrawl_browser_live_viewLive view link for a remote browserAInspect
Gets a link a person can open to watch the remote browser your CDP connection is driving, and take control of it: the same tabs, cookies and page. Use it when a page needs a human: a login, a one-time code, a captcha. Send them live_view_url; they press "Take control", do the step, and give control back. While they have it, your connection's input and navigation commands fail with error -32001 (reading the page still works); poll spicrawl_browser_control (action status) until controller is agent, then carry on. The link works once and must be opened within 60 seconds; ask again for a new one. The session must still be connected. Watching costs nothing beyond the session itself.
Inputs: session_id (required). Returns: live_url, expires_in, session_id, single_use, +1 more
spicrawl_docs_indexList the Spicrawl docsARead-onlyIdempotentInspect
The docs' llms.txt: every Spicrawl documentation page with its title, a one-line description and its Markdown URL. Use it to find the right page when a search does not, or to get an overview of what is documented; read a page with spicrawl_docs_read. No parameters.
Inputs: no named parameters. Returns: url, content, truncated
spicrawl_docs_readRead a Spicrawl docs pageARead-onlyIdempotentInspect
Fetches one Spicrawl documentation page as Markdown. path may be a page path (guides/anti-bot), a docs URL from spicrawl_docs_search (url or md_url), or either with a #anchor (the whole page is returned; the anchor names the section to look at). Only pages on the Spicrawl docs (https://docs.spicrawl.com) can be read. Pages over 60000 characters are truncated, and the text says so. Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page.
Inputs: path (required). Returns: url, title, md_url, content, +2 more
spicrawl_docs_searchSearch the Spicrawl docsARead-onlyIdempotentInspect
Full-text search over the Spicrawl documentation (guides, API reference, errors, limits). Use it to look up a parameter name, value or combination for a spicrawl_* tool, or to explain an error a spicrawl_* tool returned: search the error code itself (e.g. ERR::UPSTREAM::CHALLENGE) and the result also carries a direct link to that code's entry on the errors page. Returns the best-matching pages, each with its title, the matching sections (heading, a short snippet, URL) and md_url; then call spicrawl_docs_read with a page's md_url or url to read it in full.
Inputs: query (required), limit. Returns: hint, query, results, error_code
spicrawl_request_getGet one requestARead-onlyIdempotentInspect
The full log record of one past API call by its request id. Every response carries its id in the X-Request-Id header (and error bodies carry it too), so this is the tool for debugging a failed or slow scrape: it shows status, error_code/error_detail, attempts, the engine and proxy (tier, country, provider, sticky) actually used, whether and how the target blocked it (vendor/signal/rule), per-stage latency spans and credits charged. Returns not_found both for unknown ids and for ones older than your retention window, and for another project's request unless your key has the read scope.
Inputs: id (required). Returns: id, url, proxy, spans, +15 more
spicrawl_requests_listList recent requestsARead-onlyIdempotentInspect
Recent API calls in your key's project, newest first, each with project_id, status, engine, http_status, error_code/error_detail, attempts, URL, proxy used, latency spans (queue/engine/upstream/total ms), block detection and credits. Use it to find failing calls (only_errors=true) or to spot patterns across a target. all_projects=true lists every project in your organization instead (needs a key with the read scope; 403 ERR::AUTH::INSUFFICIENT_SCOPE otherwise). Records only exist within the retention window reported in page.retention_hours / page.retained_since; an empty page means nothing in that window. Paginate by passing page.next_before and page.next_before_id back as before and before_id while page.has_more is true, keeping all_projects the same. Not billing-grade: use spicrawl_usage for totals.
Inputs: limit, before, status, before_id, +2 more. Returns: data, page
spicrawl_scrapeScrape a URLAInspect
Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF. Handles the fetch vs. browser decision, charset decoding, and PDF-to-text for you. Optionally extract structured data with CSS selectors (extract) or autoparse (a natural-language ai_extract is coming soon), return discovered links, take a screenshot, or scope the DOM with include_tags/exclude_tags. Each successful call is billed in credits (a failure costs 0). Results are cached by default, and a cache hit is billed like the fetch that stored it (the cache saves time, not credits); set cache=false for time-sensitive pages. With the defaults it only reads the page. A method other than GET, HEAD or OPTIONS, and the actions that click, fill or run scripts on the page, can change state on the target site: set them only when the user asks for that. For markdown, text and HTML it returns { content }. With format: "json", or when extract, ai_extract, autoparse, links, screenshot or network_capture is set, it returns the JSON envelope: content, the site's status, credits, engine (the rendering tier: fetch, js or browser), warnings and data for extraction. Screenshots come back as image content blocks; each screenshots entry in the JSON keeps its metadata and says which block holds it. A PDF comes back as a resource content block, with its size, the rendering tier and the credits in the JSON.
Inputs: url (required), mode, wait, cache, +35 more. Returns: pdf, url, data, links, +13 more
spicrawl_session_contextDump a session's stored contextARead-onlyIdempotentInspect
Return the full stored state of a LIVE session — cookies, localStorage and other credentials — plus its fingerprint and domain scores. The session_context object can be passed to spicrawl_session_create to clone the session. The result contains secrets (login cookies): do not echo it to the user or logs unless asked.
Inputs: session_id (required). Returns: retired, warning, created_at, expires_at, +7 more
spicrawl_session_createCreate a browsing sessionAInspect
Create a session: a persistent browser identity (cookie jar, localStorage, a fixed rendering tier and, by default, a sticky exit IP) that survives across scrapes. Use it for multi-step flows — log in, then fetch pages behind the login — by passing the returned id as session_id to every spicrawl_scrape call that should share the state. The rendering tier is fixed for the session's lifetime and every scrape on the session runs on it. Leave engine out and one is chosen for you. Release the session with spicrawl_session_release when done. Returns the session object (id, engine, status, sticky_key, proxy, expires_at).
Inputs: engine, rotate_ip, sticky_key, fingerprint, +6 more. Returns: id, lease, proxy, engine, +14 more
spicrawl_session_deleteDelete a sessionADestructiveIdempotentInspect
Permanently delete a session and its stored state, including the record itself (prefer spicrawl_session_release if you only want to stop using it). Idempotent: deleting an already-deleted session succeeds. Refused while a scrape holds the session's lease unless force is true.
Inputs: session_id (required), force. Returns: deleted, session_id
spicrawl_session_getGet a sessionARead-onlyIdempotentInspect
Return one session's metadata: status, engine (its rendering tier), proxy/exit, usage count, expiry, lease holder, and a summary of its stored context. Never returns the credentials themselves (use spicrawl_session_context for that).
Inputs: session_id (required). Returns: id, lease, proxy, engine, +14 more
spicrawl_session_listList sessionsARead-onlyIdempotentInspect
List this API key's project's sessions, newest first. Returns { sessions: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Never includes the stored credentials.
Inputs: limit, cursor, engine, status. Returns: sessions, next_cursor
spicrawl_session_releaseRelease a sessionADestructiveIdempotentInspect
End a session when you are done with it: frees its exit IP and browser state; it can no longer be used with spicrawl_scrape. Its stored cookies and storage are purged and cannot be recovered; start a new session and log in again to continue. The record stays visible (status released) so its history can still be inspected. Refused while a scrape holds the session's lease unless force is true. Returns the session object.
Inputs: session_id (required), force. Returns: id, lease, proxy, engine, +14 more
spicrawl_usageUsage breakdownARead-onlyIdempotentInspect
Billing-grade usage for your org over a date window, from the daily rollup (usage_daily), grouped by day, project, engine, feature or key. group_by=key returns per API key (key_id, name, prefix, project_id) counters per metric per day, from usage_key_daily; keyless (dashboard) usage is not in it, and a project-scoped key sees only its own project's keys. Use it to answer 'how many requests/credits did I use, and where'. The window defaults to the last 30 days including today; to is EXCLUSIVE (a single day 2026-08-01 is from=2026-08-01, to=2026-08-02) and the window may not exceed 400 days. The response carries stale_seconds (age of the newest rollup row; null when the window is empty). Requires the key to have the read scope.
Inputs: to, from, metrics, group_by, +1 more. Returns: keys, groups, period, totals, +5 more
spicrawl_usage_reconciliationUsage reconciliationARead-onlyIdempotentInspect
Drift report for ONE day: re-derives usage from that day's request log and compares it with the billed daily rollup, so you can see whether the rollup missed or double-counted anything. Defaults to yesterday (UTC); today is still accumulating and would always show drift. Requires the read scope.
Inputs: day. Returns: day, drift, warnings, repairable, +2 more
spicrawl_usage_summaryUsage summaryARead-onlyIdempotentInspect
A compact current-period summary of your org's usage (totals as of now, from the daily rollup). No parameters. Use for a quick 'how much have I used so far'; use spicrawl_usage for a windowed breakdown. Requires the read scope.
Inputs: no named parameters. Returns: plan, period, credits, metrics, +6 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 27 tools
Most tools map to clearly distinct resources and actions, and descriptions explicitly clarify overlaps such as batch_results vs batch_task_content and session_get vs session_context. A few pairs remain close enough that an agent could hesitate, especially around batch status/results/content and usage variants, but the boundaries are documented.
All names use the spicrawl_ prefix, snake_case, and domain grouping (batch, session, browser, usage, docs, request), making the set easy to scan. The action grammar is slightly mixed—some are verb_noun (session_create, docs_read) while others are noun_action (batch_list, batch_status)—but it is predictable overall.
27 tools is heavy for an agent, exceeding the typical 25-tool ceiling even though the server spans several subdomains. Batch lifecycle alone has nine tools, and some surface area could likely be consolidated or conditionally exposed.
Coverage is broad: single scrape, full batch lifecycle, sessions with create/list/get/context/release/delete, browser control, usage, request logs, and docs. Minor gaps remain, such as project/key administration and explicit batch deletion, but core workflows are covered end to end.
Maintenance
Related MCP Connectors
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Scrape, crawl and search the web for AI agents via MCP.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
Scraper API for HTML, Markdown, and screenshots, web search, and account usage for AI agents.
Related MCP Servers
- AlicenseAqualityBmaintenanceProvides 27 MCP-native tools for web scraping, crawling, deep research, and autonomous extraction, delivering clean Markdown and structured JSON from any website.31696 npm6MIT
- AlicenseNot gradedqualityCmaintenanceEnables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.AGPL 3.0
- AlicenseAqualityBmaintenanceEnables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.261MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-