Skip to main content
Glama
Spicrawl MCP Server

Spicrawl MCP Server

Spicrawl MCP Server: web scraping for Claude, Cursor and AI agents

To let Claude, Cursor or another AI agent scrape websites, connect the Spicrawl MCP server. Add the hosted endpoint https://mcp.spicrawl.com/mcp (or run npx -y @spicrawl/mcp locally) with a Spicrawl API key, and the agent can call spicrawl_scrape to read any URL as Markdown, text, HTML or JSON. It renders JavaScript pages, extracts structured data, scrapes up to 10,000 URLs in one batch job, takes screenshots and keeps logged-in browser sessions, all as native spicrawl_* tools in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex. The free tier includes 1,000 credits per month.

npm version npm downloads license LightNow capabilities CI MCP Registry Glama Dify Marketplace Cursor Directory

Docs · Get an API key · TypeScript SDK · CLI · Agent plugins · Changelog

Install from: npm · Smithery · Glama · MCP Registry · Dify plugin · Cursor Directory

Quickstart

  1. Create an API key at app.spicrawl.com. Keys start with spicrawl_live_ or spicrawl_test_.

  2. Connect the hosted endpoint, https://mcp.spicrawl.com/mcp (MCP Streamable HTTP, nothing to install):

    export SPICRAWL_API_KEY=spicrawl_live_...
    claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
      --header "Authorization: Bearer $SPICRAWL_API_KEY"

    Codex: codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

  3. Or run it locally over stdio (Node.js 20+), for clients that only launch local servers:

    claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
  4. Ask your agent: "Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price." It calls spicrawl_scrape and gets the page back as Markdown.

Both transports serve the same 27 spicrawl_* tools (see Tools), and every call runs under your key with the same scopes, limits and credits as a direct API call.

Related MCP server: snoopscan

What it does

  • Scrape any URL to Markdown, text, HTML, a JSON envelope or a printed PDF with one tool, spicrawl_scrape. Markdown is the default and strips the page to its main content. format: "json" returns that main content as Markdown in content, plus sections ({level, heading, text} split on headings); use format: "html" for the raw document.

  • Render JavaScript with render: true (3 credits). A plain fetch costs 1 credit, screenshots, PDFs and actions use a full browser (8 credits), and mode: "auto" escalates from fetch to browser and bills only the rung that worked (price table). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.

  • Extract structured data with a CSS/XPath selector map (extract) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (autoparse). A trailing [] on a selector returns every match as an array, e.g. {"titles": ".titleline > a[]"}; without it only the first match is returned.

  • Drive the page before capture with 8 typed browser actions: click, fill, wait_for, wait_for_navigation, scroll (including infinite scroll), select, evaluate and screenshot.

  • Return screenshots and PDFs the model can see: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as application/pdf resource blocks (up to 10 MB).

  • Batch scrape thousands of URLs as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open so you can append URLs as you discover them.

  • Keep a logged-in browser with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.

  • Answer "what did that cost?" from usage and request-log tools, and read the Spicrawl docs from inside the agent.

Use cases

  • Scrape docs into a RAG knowledge base. Read documentation pages as clean Markdown with spicrawl_scrape (main_content_only is on by default for Markdown), or queue up to 10,000 URLs in one spicrawl_batch_submit job. Spicrawl returns the Markdown; chunking and embedding are yours. See the knowledge-base recipe.

  • Monitor prices. Pull a product's title, price and availability with an extract selector map or autoparse, and set cache: false for a fresh reading. A plain-fetch check costs 1 credit, a rendered one 3. Run it on your own schedule. See price monitoring.

  • Extract job postings. Collect listings from job boards, including infinite-scroll pages (render: true with a scroll action), as structured fields. See job postings.

  • Read JavaScript-heavy pages. When a plain fetch returns an empty shell, call spicrawl_scrape with render: true (3 credits) so the page builds itself first.

Check each site's terms and robots.txt before you scrape it.

Which tool to use

  • If you need one page's content or data now, use spicrawl_scrape.

  • If the page is blank or a skeleton without JavaScript, call spicrawl_scrape again with render: true.

  • If you know the page's structure, use extract with CSS selectors; if you don't, try autoparse.

  • If you have more than about 20 URLs, use spicrawl_batch_submit, then spicrawl_batch_status and spicrawl_batch_results.

  • If you discover URLs as you go, submit with open: true, append with spicrawl_batch_add_items and finish with spicrawl_batch_close.

  • If you must log in first, use spicrawl_session_create and pass its id as session_id to spicrawl_scrape.

  • If a call failed and you need to know why, use spicrawl_request_get with the request id, or spicrawl_requests_list with only_errors.

  • If you are unsure of a parameter or an error code, use spicrawl_docs_search before guessing.

Tools

27 tools, grouped by task. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a title and explicit readOnlyHint, destructiveHint and openWorldHint annotations, so a client knows which calls to confirm: spicrawl_scrape is not read-only (its method and actions can submit forms), and spicrawl_batch_cancel, spicrawl_session_release and spicrawl_session_delete are destructive. Every tool also declares an outputSchema and returns matching structuredContent beside the same text it always returned, so a client can read fields (credits, status, next_cursor, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (spicrawl_docs_read, spicrawl_batch_task_content) puts it under content in structuredContent. Full argument reference: docs.spicrawl.com/agents/mcp.

Scrape a website to Markdown

Tool

What it does

Key parameters

spicrawl_scrape

Retrieve one URL as markdown (default), text, html, json or pdf, with optional rendering, extraction, links, screenshots and browser actions.

url, format, render, mode, main_content_only, include_tags, exclude_tags, extract, autoparse, links, screenshot, actions, wait_for, session_id, proxy, cache, max_cost

Batch scrape many URLs

Tool

What it does

Key parameters

spicrawl_batch_submit

Queue many URLs as one async job with shared settings; returns the job with estimated_credits.

urls or items, render, format, open, max_attempts, credit_budget, max_cost

spicrawl_batch_list

List the project's batch jobs, newest first.

status, limit (max 200), cursor

spicrawl_batch_status

One job's status and progress.

job_id

spicrawl_batch_results

A page of finished items; works while the job runs.

job_id, status, limit (max 5000), cursor

spicrawl_batch_task_content

The full payload of one item.

job_id, seq

spicrawl_batch_cancel

Cancel a job; finished items keep their results.

job_id

spicrawl_batch_retry

Re-queue every failed item.

job_id

spicrawl_batch_add_items

Append URLs to a job submitted with open: true.

job_id, urls or items

spicrawl_batch_close

Stop an open job accepting items so it can complete.

job_id

Browser sessions

Tool

What it does

Key parameters

spicrawl_session_create

Create a persistent browser identity (cookies, storage, a fixed rendering tier).

engine, ttl_seconds, session_context

spicrawl_session_list

List sessions, newest first.

status, engine, limit, cursor

spicrawl_session_get

One session's status, rendering tier, exit, usage and expiry.

session_id

spicrawl_session_context

Dump a live session's cookies and storage (contains secrets).

session_id

spicrawl_session_release

End a session; its record stays as released.

session_id

spicrawl_session_delete

Permanently delete a session and its state.

session_id

Remote browser

Tool

What it does

Key parameters

spicrawl_browser_connect_url

Mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from your own CDP client. Returns its session_id.

engine (browser, js), proxy_country, session_ttl

spicrawl_browser_live_view

A link a person opens to watch that browser and take control: for logins, one-time codes and captchas. Single use, open within 60 s.

session_id

spicrawl_browser_control

Who drives the browser, agent or human; poll it after sending a live-view link. Also takeover and release.

session_id, action

While a person has control, the agent's input and navigation commands fail with CDP error -32001 and reading the page still works. When they give control back, the agent carries on in the same tab with the same cookies.

Usage and request history

Tool

What it does

Key parameters

spicrawl_usage

Billing-grade usage over a date window (read scope).

from, to, group_by, metrics

spicrawl_usage_summary

Current-period totals (read scope).

none

spicrawl_usage_reconciliation

One day's drift between the request log and the billed rollup (read scope).

day

spicrawl_requests_list

Recent API calls with status, error code, rendering tier, latency and credits.

only_errors, status, limit, all_projects

spicrawl_request_get

The full log record of one call.

id

Docs

Tool

What it does

Key parameters

spicrawl_docs_search

Search the Spicrawl docs; an error code also links to its entry.

query, limit (1-20, default 8)

spicrawl_docs_read

Read one docs page as Markdown (truncated over 60,000 characters).

path

spicrawl_docs_index

The docs' llms.txt: every page with its title and .md link.

none

Coming soon

The schemas accept ai_extract, stealth, extract_preset, premium_proxy, proxy_country and sticky_key ahead of launch. Agents should not send them yet. To list only what works today, set SPICRAWL_MCP_HIDE_UNAVAILABLE=1 (see Environment variables).

Example prompts

  • "Scrape https://example.com/pricing as markdown and summarise the plans."

  • "Get the product name, price and stock status from this URL." (spicrawl_scrape with extract or autoparse)

  • "Take a full-page screenshot of https://example.com after the cookie banner is gone." (render, actions, screenshot_fullpage)

  • "Scrape these 500 URLs and give me each page title." (spicrawl_batch_submit, then spicrawl_batch_results)

  • "Why did my last scrape fail?" (spicrawl_requests_list with only_errors, then spicrawl_request_get)

  • "How many credits have I used this month?" (spicrawl_usage_summary)

Set up your MCP client

Put your key in SPICRAWL_API_KEY and reference it from the config where the client allows, so the key never lands in a committed file.

Hosted endpoint, stored for you only (add --scope user for every project):

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

To share it with a team, commit .mcp.json at the project root. Claude Code expands ${VAR}, so each teammate's own key is used:

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

Or install the plugin, which bundles the MCP server and the Spicrawl agent skill:

claude plugin marketplace add OfficialSpicrawl/agent-plugins
claude plugin install spicrawl@spicrawl-plugins

Check with claude mcp list, or /mcp in a session. More: docs.spicrawl.com/agents/claude-code.

Claude Desktop's claude_desktop_config.json starts local (stdio) servers only, and custom connectors cannot send an Authorization header, so run the package locally. The file is at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows:

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

Restart Claude Desktop after saving.

Use the Add to Cursor button above, or add this to .cursor/mcp.json in the project (or ~/.cursor/mcp.json for every project). Cursor resolves ${env:NAME} in url and headers, so start Cursor from an environment where SPICRAWL_API_KEY is set:

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

More: docs.spicrawl.com/agents/cursor.

Use the Install in VS Code button above, or add this to .vscode/mcp.json. VS Code uses the servers key, and the input variable makes it ask for the key once and store it securely, so the file is safe to commit:

{
  "inputs": [
    { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
  ],
  "servers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
    }
  }
}

VS Code does not expand ${env:...} inside headers, so use the input variable. More: docs.spicrawl.com/agents/github-copilot.

Windsurf is now named Devin Desktop. For the default Devin Local agent, add the server in local scope (saved to the gitignored .devin/mcp_config.local.json):

devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"

For the legacy Cascade agent, edit ~/.config/devin/mcp_config.json (%APPDATA%\devin\mcp_config.json on Windows). Cascade reads the URL from serverUrl and replaces ${env:...}:

{
  "mcpServers": {
    "spicrawl": {
      "serverUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

More: docs.spicrawl.com/agents/windsurf.

Gemini CLI strips variables whose names contain KEY, TOKEN or AUTH before expanding headers, so export the key under another name:

export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"

Then add to ~/.gemini/settings.json (or .gemini/settings.json for one project). Streamable HTTP uses the httpUrl key:

{
  "mcpServers": {
    "spicrawl": {
      "httpUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
    }
  }
}

Or: gemini mcp add --scope user --transport http --header 'Authorization: Bearer $SPICRAWL_MCP_BEARER' spicrawl https://mcp.spicrawl.com/mcp. Run /mcp to check. More: docs.spicrawl.com/agents/gemini-cli.

Codex reads the key from the environment each time it connects, so the key never lands in a file:

codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

The equivalent ~/.codex/config.toml (or .codex/config.toml in a trusted project):

[mcp_servers.spicrawl]
url = "https://mcp.spicrawl.com/mcp"
bearer_token_env_var = "SPICRAWL_API_KEY"

Local stdio instead:

[mcp_servers.spicrawl]
command = "npx"
args = ["-y", "@spicrawl/mcp"]
env = { SPICRAWL_API_KEY = "spicrawl_live_..." }

More: docs.spicrawl.com/agents/codex.

A gateway that passes a connection parameter through as a header cannot always send Authorization: Bearer <key>: Smithery maps a parameter to a header but sends the raw value, with no Bearer prefix, and a client often cannot name its source authorization. /mcp takes the key in any of these headers:

Header

Value

x-api-key

spicrawl_live_... (recommended for a gateway)

api-key

spicrawl_live_...

Authorization

spicrawl_live_... (the raw key, no Bearer ), or Bearer spicrawl_live_...

In Smithery, make the API key a connection parameter, map it to the x-api-key header, and mark it secret. Or map it to Authorization and send the raw key; the server accepts that when the whole value is a Spicrawl key.

The key can be left optional in the connection schema (no required): initialize, ping and tools/list on /mcp answer without a key, with the same tool list as the server card, so a directory can list the tools. Every tool call still needs a valid key and is refused with 401 otherwise; no session is opened and nothing is billed or read without one. Say in the parameter's description that it is needed to run any tool.

If more than one is sent, the first that holds a well-formed key wins, in the order Authorization: Bearer, raw Authorization, x-api-key, api-key. A key in the query string is not read, because URLs end up in access logs. /chatgpt/mcp takes OAuth access tokens only and accepts none of these.

Setting

Value

Transport

Streamable HTTP

URL

https://mcp.spicrawl.com/mcp

Header

Authorization: Bearer <your key> (or x-api-key: <your key>)

A stdio-only client runs npx -y @spicrawl/mcp with SPICRAWL_API_KEY in its environment. The server accepts API keys only, not OAuth. Setup pages for more clients (OpenCode, Zed, JetBrains, Cline, Goose and others): docs.spicrawl.com/agents/mcp.

Environment variables

Variable

Default

Used by

Purpose

SPICRAWL_API_KEY

none (required)

stdio

Your API key. The HTTP server ignores it and uses each caller's bearer token.

SPICRAWL_BASE_URL

https://api.spicrawl.com (stdio), http://127.0.0.1:8080 (HTTP)

both

The Spicrawl API the tools call.

SPICRAWL_PUBLIC_BASE_URL

SPICRAWL_BASE_URL

both

The API as the end user reaches it, used only in URLs handed back.

SPICRAWL_DOCS_URL

https://docs.spicrawl.com

both

Docs base URL for the docs tools. Your key is never sent there.

SPICRAWL_DOCS_HOST

none

both

Legacy: an origin whose docs are at /docs. Read only when SPICRAWL_DOCS_URL is unset.

SPICRAWL_MCP_ADDR

127.0.0.1:8090

HTTP

Listen address, host:port.

SPICRAWL_MCP_SESSION_IDLE_MS

1800000 (30 min)

HTTP

Idle MCP sessions are dropped after this long.

SPICRAWL_MCP_HIDE_UNAVAILABLE

off

both

1, true, yes or on leaves out what is announced but not available yet: every argument marked "Coming soon" is dropped from the schemas (ai_extract, stealth, extract_preset, premium_proxy, proxy_country, sticky_key, and the session rotate_ip, region_pool and fingerprint). Read when the server starts a session, so restart the process after changing it. The keyless tools/list and the server card behave as if it were always on.

SPICRAWL_CHATGPT_ENABLED

off

HTTP

1, true, yes or on turns on the ChatGPT endpoint. Off, /chatgpt/mcp, its metadata and the OpenAI challenge all answer 404.

SPICRAWL_CHATGPT_RESOURCE

https://mcp.spicrawl.com/chatgpt/mcp

HTTP

The OAuth resource identifier. Tokens must be issued for exactly this aud; the metadata is published at /.well-known/oauth-protected-resource + its path.

SPICRAWL_OAUTH_ISSUER

https://app.spicrawl.com

HTTP

The authorization server, listed in the metadata. Tokens are introspected at <issuer>/api/oauth/introspect.

SPICRAWL_OAUTH_INTROSPECT_SECRET

none

HTTP

This server's bearer secret at the introspection endpoint. Unset, every token is answered 503.

SPICRAWL_CHATGPT_ROOT_PRM

off

HTTP

Also serve the metadata at the root /.well-known/oauth-protected-resource. Off by default because /mcp takes API keys and must not advertise OAuth.

SPICRAWL_OPENAI_APPS_CHALLENGE

none

HTTP

The OpenAI domain-verification token served at /.well-known/openai-apps-challenge.

SPICRAWL_OPENAI_APPS_CHALLENGE_FILE

none

HTTP

A file holding that token (surrounding whitespace and the trailing newline are dropped). Read when SPICRAWL_OPENAI_APPS_CHALLENGE is unset.

With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at <SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs.

Self-host the MCP server over HTTP

The hosted endpoint runs this package's Streamable HTTP server. To run your own:

SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
SPICRAWL_BASE_URL=https://api.spicrawl.com \
npx -y @spicrawl/mcp --http
  • SPICRAWL_BASE_URL defaults to http://127.0.0.1:8080 in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it to https://api.spicrawl.com to front the public API.

  • MCP is served at POST/GET/DELETE /mcp; liveness at GET /healthz (no auth).

  • GET /.well-known/mcp/server-card.json (no auth, Cache-Control: public, max-age=3600) is a static server card for directories such as Smithery that cannot scan /mcp without a key: serverInfo, the transport, bearer auth and the tool list. The tools are read from the same definitions as tools/list, as if SPICRAWL_MCP_HIDE_UNAVAILABLE were on: every tool, without the arguments marked "Coming soon".

  • Without any key, /mcp answers only discovery: initialize, ping, the initialized notification and tools/list (the same tool list as the server card; no Mcp-Session-Id is issued and nothing is sent upstream). Any other request, a tools/call included, is 401, as is a request that sends a malformed key or names a session. A session-less GET is 405.

  • Each caller sends their own key as Authorization: Bearer spicrawl_live_..., or, for a gateway that cannot send the Bearer prefix, as a raw Authorization: spicrawl_live_..., x-api-key or api-key header (see "Smithery and other gateways" above; never a query parameter). The key is checked against the API before a session opens (401 if refused, 503 with Retry-After if the API cannot be reached), and a session is pinned to the key that opened it.

  • The server speaks plain HTTP. Put it behind TLS before sending keys across a network.

ChatGPT endpoint (OAuth)

The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless SPICRAWL_CHATGPT_ENABLED is on, and /mcp is unchanged either way.

Route

What it does

POST /chatgpt/mcp

MCP Streamable HTTP, stateless: no Mcp-Session-Id, a fresh server per request, JSON responses. Other methods answer 405.

GET /.well-known/oauth-protected-resource/chatgpt/mcp

Protected-resource metadata (RFC 9728): resource, authorization_servers (the issuer), scopes_supported: ["scrape","batch"], resource_documentation.

GET /.well-known/oauth-protected-resource

The same document, only with SPICRAWL_CHATGPT_ROOT_PRM.

GET /.well-known/openai-apps-challenge

The verification token as text/plain, exact bytes, no trailing newline; 404 when none is configured.

Authentication takes OAuth access tokens (spicrawl_oat_…) only:

  • No token: 401 with WWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch".

  • A bad, inactive, expired or wrong-audience token, or an API key (spicrawl_live_…/spicrawl_test_…, never accepted here): the same 401 plus error="invalid_token", error_description="…".

  • A tool call the token's scopes do not cover: 403 with error="insufficient_scope". spicrawl_scrape needs scrape, the batch tools need batch, the docs tools need only a valid token.

  • The authorization server unreachable or refusing this server's secret: 503 with Retry-After.

Each token is checked with POST <issuer>/api/oauth/introspect (Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>, form body token=…). Only active: true with aud equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its exp. Tools run upstream under the api_key the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose _meta["mcp/www_authenticate"] carries a fresh invalid_token challenge, and the token is dropped from the cache.

The endpoint lists six tools, each with explicit readOnlyHint/destructiveHint/openWorldHint, a title, and securitySchemes: [{type: "oauth2", scopes: [...]}] (also in _meta.securitySchemes):

Tool

Scopes

Arguments

spicrawl_scrape

scrape

url (http/https only), format (markdown/text/html/json), render, main_content_only, include_tags, exclude_tags, links, extract (CSS selectors), autoparse, wait_for, cache, cache_ttl, max_cost (default 5). Always an HTTP GET.

spicrawl_batch_submit

batch

urls (1 to 25), format, render, main_content_only, max_cost, credit_budget (default 50).

spicrawl_batch_status

batch

job_id.

spicrawl_batch_results

batch

job_id, status, limit (default 25, max 100), cursor.

spicrawl_docs_search, spicrawl_docs_read

none

As on /mcp.

spicrawl_scrape is read-only here, unlike on /mcp: this endpoint only sends a GET, with no method, body, headers or actions. spicrawl_batch_submit is not read-only, because it creates a job.

spicrawl_batch_results fills in the content of a succeeded item that the results page lists without it, which happens while the job is still finishing, from GET /v1/batch/{id}/tasks/{seq}/content. It fetches at most 25 items and 2 MiB per page. An item that can't be read yet comes back unavailable, and one past the size cap omitted.

Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.

Installed globally (npm install -g @spicrawl/mcp), the command is spicrawl-mcp (alias spicrawl-mcp-server), with --http, --version and --help.

FAQ

Which MCP server scrapes JavaScript sites?

The Spicrawl MCP server does. Call spicrawl_scrape with render: true (3 credits) and the page loads in a browser, runs its scripts and comes back as finished Markdown, text, HTML or JSON. Add wait_for with a CSS selector for content that appears late, or set mode: "auto" to start with a plain fetch and move up to a stronger rendering only when needed. Spicrawl picks the rendering for each request and bills the tier that served it (fetch, js or browser, reported as engine in the JSON result); there is no argument to choose it.

How do I give Claude web access?

Connect the Spicrawl MCP server to Claude with an API key. In Claude Code:

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Claude can then read any URL you give it by calling spicrawl_scrape. Spicrawl reads pages by URL; it is not a search engine. For Claude Desktop, see the install answer below.

Is there a free tier?

Yes. During the beta every organization is on one plan of 1,000 credits per month. A plain fetch costs 1 credit, JavaScript rendering 3, and full-browser rendering (screenshots, PDFs, actions) 8. Failed requests cost 0, and a cache hit is billed at the same price as the fetch that stored it. When the month's credits run out, calls fail with ERR::LIMIT::QUOTA_EXCEEDED until it resets. See Credits.

Does it handle bot-protected pages?

Only with your own proxy. Pass it in proxy (an http:// URL; residential proxies work best). When a site answers with a Cloudflare managed challenge, Spicrawl detects it, moves the request to full-browser rendering on the same proxy exit and returns the real page, for 8 credits. If every step is challenged you get ERR::UPSTREAM::CHALLENGE and pay 0. Success is likely, not guaranteed, and Spicrawl does not solve CAPTCHAs. On the default shared pool a challenged page returns that error. See Scrape Cloudflare-protected pages.

How do I install it in Cursor or Claude Desktop?

Cursor: click Add to Cursor above, or add this to .cursor/mcp.json and start Cursor with SPICRAWL_API_KEY set:

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

Claude Desktop starts local (stdio) servers only, so run the package. Add this to claude_desktop_config.json and restart the app:

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

Get a key at app.spicrawl.com. Steps for Claude Code, VS Code, Windsurf, Gemini CLI and Codex are under Set up your MCP client.

Can it crawl a whole website?

There is no single crawl tool. Scrape a page with links: true to get its links, then feed them to an open batch job (open: true and spicrawl_batch_add_items).

Other questions

Is there a hosted Spicrawl MCP server?

Yes: https://mcp.spicrawl.com/mcp, MCP Streamable HTTP, authenticated with Authorization: Bearer <API key>, so there is nothing to install.

Does it support OAuth?

Not on /mcp: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a 401. The separate ChatGPT endpoint at /chatgpt/mcp takes OAuth access tokens only, when the operator turns it on.

Should I use the MCP server, the SDK or the CLI?

Use the MCP server when an AI agent should call Spicrawl as tools. Use @spicrawl/sdk in your own TypeScript or JavaScript code, @spicrawl/cli in a terminal or shell script, and the REST API from any other language.

Troubleshooting

Symptom

Fix

HTTP 401 or "Unauthorized: send your Spicrawl API key"

The header is missing, or the key is malformed, unknown, revoked or expired. Check that the variable is set in the environment the client started from. A gateway that cannot send Bearer can use an x-api-key header; a key in the URL is never read.

The header contains a literal ${SPICRAWL_API_KEY}

The client did not expand it. Use ${VAR} in Claude Code, ${env:VAR} in Cursor, ${input:id} in VS Code, bearer_token_env_var in Codex.

HTTP 503 with Retry-After on connect

The server could not reach the API to check your key. Retry; your key is fine.

ERR::AUTH::INSUFFICIENT_SCOPE from a spicrawl_usage* tool

Grant the key the read scope in the dashboard.

ERR::LIMIT::QUOTA_EXCEEDED (HTTP 402)

The monthly credit allowance is used up. Not retryable until it resets.

ERR::UPSTREAM::CHALLENGE at 0 credits

A plain fetch got a near-empty, script-driven page (a bot challenge stub), so nothing was billed. Retry with render: true, or pass your own proxy. mode: "auto" escalates past it on its own.

"Unknown argument" on spicrawl_scrape

Use the tool's names: render, not js_render; format, not response_format.

Tools do not appear

Restart the client, and check the config key (servers in VS Code, mcpServers elsewhere).

npx not found (stdio)

Install Node.js 20 or later.

curl -i https://mcp.spicrawl.com/mcp without a key answers 401: the endpoint is reachable.

Publishing to the MCP Registry

server.json lists both the npm/stdio package and the hosted Streamable HTTP endpoint under io.github.OfficialSpicrawl/mcp. API keys are requested from the person installing the server; no key belongs in the manifest.

The existing release workflow publishes to the MCP Registry after its npm job succeeds, on pushes to main or manual runs from main. Registry authentication uses GitHub OIDC (id-token: write), so it needs no new secret, device login or DNS record. The npm release keeps using the existing NPM_TOKEN secret.

For a release, run npm version --no-git-tag-version X.Y.Z, update the changelog, and submit the changes to main. The npm version hook synchronizes both versions in server.json. If you edit package.json by hand, run npm run registry:sync. CI checks the manifest against its official $schema with Ajv and rejects name or version drift before npm publication. To run that check locally:

npm ci
npm run registry:validate
# Offline: pass a downloaded copy of the exact server.json $schema URL.
npm run registry:validate -- /path/to/server.schema.json

The registry job checks that the published npm package has the matching mcpName before publishing. An existing registry version is skipped; lookup failures stop the job. To retry a registry failure or list the already-published current npm version, run the release workflow on main again. Use a new version for metadata changes to an existing listing. The publisher binary is pinned and its download is checked against the release checksums; update its version in release.yml when upgrading the CLI.

Verify publication at https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.OfficialSpicrawl/mcp. The workflow must be merged into OfficialSpicrawl/mcp before OIDC can publish that namespace. Fork pull requests validate and test, but never run the registry job.

Official references: publishing quickstart, GitHub Actions, and remote servers.

License

Apache-2.0

Available Tools

27 tools
spicrawl_batch_add_itemsAdd URLs to an open batch jobAInspect

Append URLs to a batch job that was submitted with open: true. Same item vocabulary as spicrawl_batch_submit (EITHER urls or items, not both, plus shared settings for these new items). Fails with CONFLICT if the job is closed or finished. Not idempotent: calling twice adds the URLs twice. Returns { job, items_added, items_dispatched }.

Inputs: job_id (required), urls, wait, items, +9 more. Returns: job, items_added, items_dispatched

spicrawl_batch_cancelCancel a batch jobA
DestructiveIdempotent
Inspect

Cancel a batch job: items not yet started are cancelled and will never run; items already finished keep their results; items already running finish first, so the job comes back cancelling and moves to cancelled when they drain. Cannot be undone — resubmit to run the URLs again. Returns the job object.

Inputs: job_id (required). Returns: id, name, open, params, +20 more

spicrawl_batch_closeClose an open batch jobA
Idempotent
Inspect

Stop an open batch job accepting items, so it completes once its queued work drains. Idempotent (closing a closed job succeeds). Returns the job object.

Inputs: job_id (required). Returns: id, name, open, params, +20 more

spicrawl_batch_listList batch jobsA
Read-onlyIdempotent
Inspect

List this API key's project's batch jobs, newest first, with status and progress. Returns { batches: [...], next_cursor? }; pass next_cursor back as cursor for the next page.

Inputs: limit, cursor, status. Returns: batches, next_cursor

spicrawl_batch_resultsRead a batch job's resultsA
Read-onlyIdempotent
Inspect

Return a page of FINISHED items of a batch job (one record per item: seq, url, external_id, status, error, and content, or a result_url API path (GET /v1/batch/{id}/tasks/{seq}/content, same key) for large bodies). Unfinished items are not listed yet. Returns { results: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Results expire after the job's retention window (410 Gone).

Inputs: job_id (required), limit, cursor, status. Returns: results, next_cursor

spicrawl_batch_retryRetry a batch job's failed itemsAInspect

Re-queue every FAILED item of a batch job so it runs again (succeeded items are untouched; a job with no failures is returned unchanged). Returns { job, items_reset, items_dispatched }.

Inputs: job_id (required). Returns: job, items_reset, items_dispatched

spicrawl_batch_statusCheck a batch job's statusA
Read-onlyIdempotent
Inspect

Return one batch job with its status and progress (total/completed/succeeded/failed/remaining, credits charged). Poll this after spicrawl_batch_submit; once status is completed, failed or cancelled, read spicrawl_batch_results.

Inputs: job_id (required). Returns: id, name, open, params, +20 more

spicrawl_batch_submitSubmit a batch scrape jobAInspect

Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER urls or items, not both (at least one URL, unless open is true). Returns the job object (id, status, estimated_credits, progress). Poll with spicrawl_batch_status, read with spicrawl_batch_results or spicrawl_batch_task_content. Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs. Set open: true to keep the job accepting more URLs via spicrawl_batch_add_items; an open job never finishes until you call spicrawl_batch_close.

Inputs: name, open, urls, wait, +16 more. Returns: id, name, open, params, +20 more

spicrawl_batch_task_contentRead one batch item's contentA
Read-onlyIdempotent
Inspect

Return the full payload one item of a batch job produced, by its seq (items are numbered from 0 in submission order). Cheaper than paging results when you need one URL's output. A CONFLICT error means either the item has not finished yet (poll spicrawl_batch_status and ask again) or it finished as failed (it will never produce content; the message says which). NOT_FOUND means no such seq.

Inputs: seq (required), job_id (required). Returns: content

spicrawl_browser_connect_urlBrowser (CDP) connect URLAInspect

Mints a single-use connect URL for Spicrawl's remote browser (GET /v1/browser, a DevTools Protocol (CDP) WebSocket) so YOUR code can drive it with any CDP client, using the URL as the browser WebSocket endpoint. It calls POST /v1/browser/token and returns a ws:// or wss:// URL carrying a short-lived token, never the API key: the URL works ONCE, within 60 seconds, and only with the options given here (editing them in the URL is refused), so call this right before connecting and again for every new session. Prefer sending the key as an Authorization header when the client supports headers on the WebSocket upgrade: then connect to the plain /v1/browser URL with 'Authorization: Bearer ' and no token is needed. The key must carry the browser scope. Parameter errors are reported when minting. The result's session_id names the session once connected: when a page needs a person (a login, a one-time code, a captcha), pass it to spicrawl_browser_live_view for a link they can open, and to spicrawl_browser_control to see when they hand control back.

Inputs: proxy, engine, sticky_key, session_ttl, +2 more. Returns: url, notes, expires_at, session_id, +1 more

spicrawl_browser_controlWho controls a remote browserA
Idempotent
Inspect

Reads or changes who drives a connected remote browser: agent (your CDP connection) or human (a person in its live view, see spicrawl_browser_live_view). status reads it: poll it after sending a live-view link, and continue once controller is agent again. takeover hands control to the person before they ask (your input and navigation then fail with error -32001); release takes it back. The person can also take and give back control from the live view themselves.

Inputs: session_id (required), action. Returns: status, controller, expires_at, session_id

spicrawl_browser_live_viewLive view link for a remote browserAInspect

Gets a link a person can open to watch the remote browser your CDP connection is driving, and take control of it: the same tabs, cookies and page. Use it when a page needs a human: a login, a one-time code, a captcha. Send them live_view_url; they press "Take control", do the step, and give control back. While they have it, your connection's input and navigation commands fail with error -32001 (reading the page still works); poll spicrawl_browser_control (action status) until controller is agent, then carry on. The link works once and must be opened within 60 seconds; ask again for a new one. The session must still be connected. Watching costs nothing beyond the session itself.

Inputs: session_id (required). Returns: live_url, expires_in, session_id, single_use, +1 more

spicrawl_docs_indexList the Spicrawl docsA
Read-onlyIdempotent
Inspect

The docs' llms.txt: every Spicrawl documentation page with its title, a one-line description and its Markdown URL. Use it to find the right page when a search does not, or to get an overview of what is documented; read a page with spicrawl_docs_read. No parameters.

Inputs: no named parameters. Returns: url, content, truncated

spicrawl_docs_readRead a Spicrawl docs pageA
Read-onlyIdempotent
Inspect

Fetches one Spicrawl documentation page as Markdown. path may be a page path (guides/anti-bot), a docs URL from spicrawl_docs_search (url or md_url), or either with a #anchor (the whole page is returned; the anchor names the section to look at). Only pages on the Spicrawl docs (https://docs.spicrawl.com) can be read. Pages over 60000 characters are truncated, and the text says so. Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page.

Inputs: path (required). Returns: url, title, md_url, content, +2 more

spicrawl_request_getGet one requestA
Read-onlyIdempotent
Inspect

The full log record of one past API call by its request id. Every response carries its id in the X-Request-Id header (and error bodies carry it too), so this is the tool for debugging a failed or slow scrape: it shows status, error_code/error_detail, attempts, the engine and proxy (tier, country, provider, sticky) actually used, whether and how the target blocked it (vendor/signal/rule), per-stage latency spans and credits charged. Returns not_found both for unknown ids and for ones older than your retention window, and for another project's request unless your key has the read scope.

Inputs: id (required). Returns: id, url, proxy, spans, +15 more

spicrawl_requests_listList recent requestsA
Read-onlyIdempotent
Inspect

Recent API calls in your key's project, newest first, each with project_id, status, engine, http_status, error_code/error_detail, attempts, URL, proxy used, latency spans (queue/engine/upstream/total ms), block detection and credits. Use it to find failing calls (only_errors=true) or to spot patterns across a target. all_projects=true lists every project in your organization instead (needs a key with the read scope; 403 ERR::AUTH::INSUFFICIENT_SCOPE otherwise). Records only exist within the retention window reported in page.retention_hours / page.retained_since; an empty page means nothing in that window. Paginate by passing page.next_before and page.next_before_id back as before and before_id while page.has_more is true, keeping all_projects the same. Not billing-grade: use spicrawl_usage for totals.

Inputs: limit, before, status, before_id, +2 more. Returns: data, page

spicrawl_scrapeScrape a URLAInspect

Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF. Handles the fetch vs. browser decision, charset decoding, and PDF-to-text for you. Optionally extract structured data with CSS selectors (extract) or autoparse (a natural-language ai_extract is coming soon), return discovered links, take a screenshot, or scope the DOM with include_tags/exclude_tags. Each successful call is billed in credits (a failure costs 0). Results are cached by default, and a cache hit is billed like the fetch that stored it (the cache saves time, not credits); set cache=false for time-sensitive pages. With the defaults it only reads the page. A method other than GET, HEAD or OPTIONS, and the actions that click, fill or run scripts on the page, can change state on the target site: set them only when the user asks for that. For markdown, text and HTML it returns { content }. With format: "json", or when extract, ai_extract, autoparse, links, screenshot or network_capture is set, it returns the JSON envelope: content, the site's status, credits, engine (the rendering tier: fetch, js or browser), warnings and data for extraction. Screenshots come back as image content blocks; each screenshots entry in the JSON keeps its metadata and says which block holds it. A PDF comes back as a resource content block, with its size, the rendering tier and the credits in the JSON.

Inputs: url (required), mode, wait, cache, +35 more. Returns: pdf, url, data, links, +13 more

spicrawl_session_contextDump a session's stored contextA
Read-onlyIdempotent
Inspect

Return the full stored state of a LIVE session — cookies, localStorage and other credentials — plus its fingerprint and domain scores. The session_context object can be passed to spicrawl_session_create to clone the session. The result contains secrets (login cookies): do not echo it to the user or logs unless asked.

Inputs: session_id (required). Returns: retired, warning, created_at, expires_at, +7 more

spicrawl_session_createCreate a browsing sessionAInspect

Create a session: a persistent browser identity (cookie jar, localStorage, a fixed rendering tier and, by default, a sticky exit IP) that survives across scrapes. Use it for multi-step flows — log in, then fetch pages behind the login — by passing the returned id as session_id to every spicrawl_scrape call that should share the state. The rendering tier is fixed for the session's lifetime and every scrape on the session runs on it. Leave engine out and one is chosen for you. Release the session with spicrawl_session_release when done. Returns the session object (id, engine, status, sticky_key, proxy, expires_at).

Inputs: engine, rotate_ip, sticky_key, fingerprint, +6 more. Returns: id, lease, proxy, engine, +14 more

spicrawl_session_deleteDelete a sessionA
DestructiveIdempotent
Inspect

Permanently delete a session and its stored state, including the record itself (prefer spicrawl_session_release if you only want to stop using it). Idempotent: deleting an already-deleted session succeeds. Refused while a scrape holds the session's lease unless force is true.

Inputs: session_id (required), force. Returns: deleted, session_id

spicrawl_session_getGet a sessionA
Read-onlyIdempotent
Inspect

Return one session's metadata: status, engine (its rendering tier), proxy/exit, usage count, expiry, lease holder, and a summary of its stored context. Never returns the credentials themselves (use spicrawl_session_context for that).

Inputs: session_id (required). Returns: id, lease, proxy, engine, +14 more

spicrawl_session_listList sessionsA
Read-onlyIdempotent
Inspect

List this API key's project's sessions, newest first. Returns { sessions: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Never includes the stored credentials.

Inputs: limit, cursor, engine, status. Returns: sessions, next_cursor

spicrawl_session_releaseRelease a sessionA
DestructiveIdempotent
Inspect

End a session when you are done with it: frees its exit IP and browser state; it can no longer be used with spicrawl_scrape. Its stored cookies and storage are purged and cannot be recovered; start a new session and log in again to continue. The record stays visible (status released) so its history can still be inspected. Refused while a scrape holds the session's lease unless force is true. Returns the session object.

Inputs: session_id (required), force. Returns: id, lease, proxy, engine, +14 more

spicrawl_usageUsage breakdownA
Read-onlyIdempotent
Inspect

Billing-grade usage for your org over a date window, from the daily rollup (usage_daily), grouped by day, project, engine, feature or key. group_by=key returns per API key (key_id, name, prefix, project_id) counters per metric per day, from usage_key_daily; keyless (dashboard) usage is not in it, and a project-scoped key sees only its own project's keys. Use it to answer 'how many requests/credits did I use, and where'. The window defaults to the last 30 days including today; to is EXCLUSIVE (a single day 2026-08-01 is from=2026-08-01, to=2026-08-02) and the window may not exceed 400 days. The response carries stale_seconds (age of the newest rollup row; null when the window is empty). Requires the key to have the read scope.

Inputs: to, from, metrics, group_by, +1 more. Returns: keys, groups, period, totals, +5 more

spicrawl_usage_reconciliationUsage reconciliationA
Read-onlyIdempotent
Inspect

Drift report for ONE day: re-derives usage from that day's request log and compares it with the billed daily rollup, so you can see whether the rollup missed or double-counted anything. Defaults to yesterday (UTC); today is still accumulating and would always show drift. Requires the read scope.

Inputs: day. Returns: day, drift, warnings, repairable, +2 more

spicrawl_usage_summaryUsage summaryA
Read-onlyIdempotent
Inspect

A compact current-period summary of your org's usage (totals as of now, from the daily rollup). No parameters. Use for a quick 'how much have I used so far'; use spicrawl_usage for a windowed breakdown. Requires the read scope.

Inputs: no named parameters. Returns: plan, period, credits, metrics, +6 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 27 tools

    Disambiguation4/5

    Most tools map to clearly distinct resources and actions, and descriptions explicitly clarify overlaps such as batch_results vs batch_task_content and session_get vs session_context. A few pairs remain close enough that an agent could hesitate, especially around batch status/results/content and usage variants, but the boundaries are documented.

    Naming Consistency4/5

    All names use the spicrawl_ prefix, snake_case, and domain grouping (batch, session, browser, usage, docs, request), making the set easy to scan. The action grammar is slightly mixed—some are verb_noun (session_create, docs_read) while others are noun_action (batch_list, batch_status)—but it is predictable overall.

    Tool Count2/5

    27 tools is heavy for an agent, exceeding the typical 25-tool ceiling even though the server spans several subdomains. Batch lifecycle alone has nine tools, and some surface area could likely be consolidated or conditionally exposed.

    Completeness4/5

    Coverage is broad: single scrape, full batch lifecycle, sessions with create/list/get/context/release/delete, browser control, usage, request logs, and docs. Minor gaps remain, such as project/key administration and explicit batch deletion, but core workflows are covered end to end.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.
      AGPL 3.0
    • A
      license
      A
      quality
      B
      maintenance
      Enables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.
      26
      1
      MIT
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.
      -