Skip to main content
Glama
spidra-mcp-server

spidra-mcp-server

Official
by spidra-io

Spidra MCP Server

The official Model Context Protocol (MCP) server for Spidra.

MCP is the standard that lets AI assistants use external tools. When you connect this server to an assistant like Claude Code, Claude Desktop, Cursor, Windsurf, VS Code, or Antigravity, the assistant can search the web, read pages, crawl sites, parse documents, run ready-made templates and watch pages for changes, on its own.

All you have to do is describe what you want in plain language, and the assistant picks the right Spidra tool, runs it, and works with the extracted data directly in the conversation.

Features

  • Search the web, news, images, videos, research papers and GitHub, and read each result's page in the same call

  • Read any page as Markdown, or get exactly the fields you describe as JSON, with a prompt or a JSON schema

  • Links from popular sites (Amazon, Google Maps, LinkedIn, YouTube, Zillow, Indeed and more) come back with their data already in named fields

  • Read 2 to 50 URLs at once, each with its own result

  • List a site's URLs in seconds, or crawl it for every page of one kind

  • Parse PDF, Word, Excel and PowerPoint files, by link or as a file

  • Run ready-made templates: Google Maps leads with emails, Amazon keyword rank, LinkedIn and Indeed jobs, Google Flights prices and more than 90 others

  • Save jobs as presets, and watch templates, presets or pages with monitors that alert by email, webhook, Slack or Discord

  • Browser actions before reading a page: click, type, select, scroll to the end, run JavaScript, loop over a list

  • Stealth Mode through residential proxies for sites that block automated visits or vary by country

  • Hosted at mcp.spidra.io with OAuth or an API key, or run it locally over stdio

  • Built to protect your credits: the narrowest tool first, no hidden second runs, and errors that say whether to retry

Related MCP server: browser-use-mcp

Before you start

You need two things:

  1. A Spidra account. Sign up at app.spidra.io. If you'll use an API key instead of signing in, create one under API Keys. Keys start with spd_.

  2. An MCP client. Any of the ones below works.

That's all for the hosted server. Running Spidra locally also needs Node.js 20 or newer (check with node --version); npx comes with Node.

Connect your client

There are two ways to connect:

  • Hosted (recommended). Point your client at https://mcp.spidra.io/mcp. There's nothing to install. You sign in with your Spidra account in the browser, or send an API key in a header.

  • Local. Your client runs npx -y spidra-mcp on your machine and passes your API key as an environment variable. This needs Node.js 20 or newer.

Every request is billed to your Spidra account exactly like a direct API call.

How signing in works

When your client connects, it opens a Spidra page in your browser. The page names the app asking for access (with a blue Verified badge and the app's real logo when its identity is verified, for example claude.ai or vscode.dev; an app that only names itself gets Name not verified and its initial) and says where you go back to afterwards. Click Approve, log in to Spidra if you aren't already, and the browser hands you back to your client. Most clients then refresh access in the background, so you sign in once.

If your client can't sign in this way, or you're setting up CI, use an API key. Create one at app.spidra.io under API Keys (keys start with spd_) and send it as Authorization: Bearer spd_....

Claude (claude.ai, Claude Desktop, mobile)

  1. Open Customize > Connectors, click +, then Add custom connector.

  2. Paste https://mcp.spidra.io/mcp and click Add.

  3. Ask Claude something that needs Spidra. Claude shows a Connect button the first time; click it and approve.

On Team and Enterprise plans an owner adds the connector under Organization settings > Connectors, and each member clicks Connect. Claude Desktop reads remote servers only from Connectors, not from claude_desktop_config.json. To run Spidra locally in Claude Desktop instead, see Local.

Claude Code

claude mcp add --transport http spidra https://mcp.spidra.io/mcp
claude mcp login spidra

claude mcp login opens the sign-in page and prints Authenticated with "spidra" when you're done. You can also run /mcp inside a session. With an API key instead:

claude mcp add --transport http spidra https://mcp.spidra.io/mcp \
  --header "Authorization: Bearer spd_YOUR_API_KEY"

Add -s user to either command to make Spidra available in every project.

Or install the Spidra plugin, which adds the same server plus a web-data skill that helps Claude pick the right tool and keep jobs small:

/plugin install spidra --marketplace spidra-io/spidra-mcp-server

The plugin lives in plugin/. Use one way or the other, not both, or Claude Code lists the server twice.

ChatGPT

  1. Go to Settings > Apps > Advanced settings and turn on Developer mode.

  2. Click Create app, name it Spidra, paste https://mcp.spidra.io/mcp, choose OAuth, and create it.

  3. ChatGPT opens the Spidra sign-in page. Approve, and the app is ready in new chats.

Cursor

Add to Cursor (opens Cursor), or add this to ~/.cursor/mcp.json (every project) or .cursor/mcp.json (one project):

{
  "mcpServers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp"
    }
  }
}

Then open Cursor Settings, find Spidra in the MCP servers list and click Needs login. If Spidra shows as connected but a tool call fails with "Unauthorized", use an API key:

{
  "mcpServers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp",
      "headers": {
        "Authorization": "Bearer spd_YOUR_API_KEY"
      }
    }
  }
}

VS Code (GitHub Copilot)

Run this in a terminal:

code --add-mcp '{"name":"spidra","type":"http","url":"https://mcp.spidra.io/mcp"}'

It adds Spidra to your user mcp.json. Start the server from that file (or from the tools picker in Copilot Chat), allow VS Code to sign in to Spidra when it asks, and approve in the browser. For a workspace with an API key, put this in .vscode/mcp.json. VS Code asks for the key once and keeps it in its secret storage:

{
  "inputs": [
    { "type": "promptString", "id": "spidra-key", "description": "Spidra API key", "password": true }
  ],
  "servers": {
    "spidra": {
      "type": "http",
      "url": "https://mcp.spidra.io/mcp",
      "headers": { "Authorization": "Bearer ${input:spidra-key}" }
    }
  }
}

Codex (CLI, IDE extension and app)

Add this to ~/.codex/config.toml, then run codex mcp login spidra:

[mcp_servers.spidra]
url = "https://mcp.spidra.io/mcp"
tool_timeout_sec = 300

tool_timeout_sec matters: Codex stops a tool call after 60 seconds by default, and a hard page or a batch can take longer. To use an API key instead of signing in, add bearer_token_env_var = "SPIDRA_API_KEY" to the same block and export SPIDRA_API_KEY in the shell that starts Codex.

Gemini CLI

gemini mcp add --transport http spidra https://mcp.spidra.io/mcp

Then run /mcp auth spidra inside Gemini CLI to sign in. With an API key instead:

gemini mcp add --transport http -H "Authorization: Bearer spd_YOUR_API_KEY" spidra https://mcp.spidra.io/mcp

Windsurf (Devin Desktop)

Open the MCP settings in Cascade and edit the raw config (~/.config/devin/mcp_config.json, or %APPDATA%\devin\mcp_config.json on Windows):

{
  "mcpServers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp",
      "transport": "http"
    }
  }
}

Click Authenticate next to Spidra when it shows "Needs authentication". Older Windsurf builds keep this file at ~/.codeium/windsurf/mcp_config.json and call the field serverUrl. For an API key, add "headers": { "Authorization": "Bearer spd_YOUR_API_KEY" }.

Cline

Open MCP Servers > Configure and add this to cline_mcp_settings.json. Leave "type" in: without it Cline uses the old SSE transport. Cline gives up on a tool call after 60 seconds unless you raise timeout:

{
  "mcpServers": {
    "spidra": {
      "type": "streamableHttp",
      "url": "https://mcp.spidra.io/mcp",
      "headers": { "Authorization": "Bearer spd_YOUR_API_KEY" },
      "timeout": 300
    }
  }
}

Use an API key in Cline for now. Its VS Code extension signs in through a callback address Spidra doesn't accept yet.

Zed

Add this under context_servers in Zed's settings.json. Zed opens the sign-in page when it connects:

{
  "context_servers": {
    "spidra": {
      "url": "https://mcp.spidra.io/mcp"
    }
  }
}

With an API key, add "headers": { "Authorization": "Bearer spd_YOUR_API_KEY" }. Zed skips sign-in when a header is set.

Goose

Run goose configure, choose Add Extension > Remote Extension (Streamable HTTP), name it spidra, paste https://mcp.spidra.io/mcp, and set the timeout to 300. Goose opens the sign-in page the first time it connects.

Antigravity

Add this to ~/.gemini/config/mcp_config.json, or .agents/mcp_config.json for one workspace. The field is serverUrl; Antigravity doesn't accept httpUrl. timeoutSeconds raises its 180-second limit per tool call:

{
  "mcpServers": {
    "spidra": {
      "serverUrl": "https://mcp.spidra.io/mcp",
      "timeoutSeconds": 300
    }
  }
}

Antigravity signs in on its own. For an API key, add "headers": { "Authorization": "Bearer spd_YOUR_API_KEY" }.

Any other client

Add a remote Streamable HTTP server with the URL https://mcp.spidra.io/mcp. If the client supports OAuth for MCP, it finds the sign-in settings itself. Otherwise send Authorization: Bearer spd_YOUR_API_KEY (or X-Spidra-Api-Key: spd_YOUR_API_KEY). If the client has a tool timeout setting, set it to 300 seconds.

Local

Every local setup runs npx -y spidra-mcp and passes your key in SPIDRA_API_KEY. Put the key in the client's env block, not in a .env file.

Claude Code:

claude mcp add spidra -e SPIDRA_API_KEY=spd_YOUR_API_KEY -- npx -y spidra-mcp

Codex (~/.codex/config.toml):

[mcp_servers.spidra]
command = "npx"
args = ["-y", "spidra-mcp"]
env = { SPIDRA_API_KEY = "spd_YOUR_API_KEY" }
tool_timeout_sec = 300

Claude Desktop, Cursor, Windsurf, Antigravity, Cline and most others take the same JSON under mcpServers. In Claude Desktop it goes in Settings > Developer > Edit Config; quit and reopen Claude afterward.

{
  "mcpServers": {
    "spidra": {
      "command": "npx",
      "args": ["-y", "spidra-mcp"],
      "env": {
        "SPIDRA_API_KEY": "spd_YOUR_API_KEY"
      }
    }
  }
}

VS Code uses servers instead of mcpServers, with "type": "stdio":

code --add-mcp '{"name":"spidra","type":"stdio","command":"npx","args":["-y","spidra-mcp"],"env":{"SPIDRA_API_KEY":"spd_YOUR_API_KEY"}}'

Gemini CLI and Zed take command, args and env under mcpServers (Gemini, in ~/.gemini/settings.json) or context_servers (Zed).

Using a staging or self-hosted Spidra API? See Configuration.

Self-hosting

Running over HTTP instead of stdio

By default the server talks to your client over stdio, which is what all the configs above use and what you want on a single machine. If you need an HTTP endpoint instead (for example, a tool that connects to MCP servers over the network), start the server like this:

env HTTP_STREAMABLE_SERVER=true SPIDRA_API_KEY=spd_YOUR_API_KEY npx -y spidra-mcp

Then connect to http://localhost:3000/mcp. On this transport the API key can also be sent per request using an Authorization: Bearer header, which is useful when one server instance serves more than one user. Bound to localhost (the default), the server only answers requests addressed to this machine, which protects it from DNS rebinding.

This is exactly how the hosted https://mcp.spidra.io/mcp endpoint runs in production: one stateless instance, keyed per request, so you don't need to set SPIDRA_API_KEY at all when using the hosted URL.

If you set HOST to anything other than localhost and also set SPIDRA_API_KEY, anyone who can reach the port can spend that key's credits. On a shared network, leave SPIDRA_API_KEY unset so every caller has to send their own key.

Protocol versions

Both transports serve the current MCP revision, 2026-07-28 (no handshake, version and capabilities sent with every request), and every earlier one from 2024-11-05 to 2025-11-25 (the initialize handshake). Clients don't need any setting for this: the server answers each request in the revision the client speaks.

Try it

Once connected, ask for web data in normal language. You never call the tools by name; the assistant does that. Some things to try:

"Search for the best espresso machines and summarize the top 3."

"Get me 20 dentists in Ikeja, Lagos from Google Maps with their phone numbers and websites."

"Compare the pricing pages of stripe.com and paddle.com and tell me which is cheaper for a small SaaS."

"Here are 12 product URLs. Get me the name, price and rating for each one as a table."

"Read pages 1 to 3 of https://arxiv.org/pdf/1706.03762 and summarize the method."

"Watch this Amazon product every day and tell me when the price drops."

If the assistant answers with real data, everything is working.

Configuration

These apply to the local/self-hosted server (npx, manual install, or your own HTTP instance). The hosted endpoint at mcp.spidra.io needs none of them, just your API key in a header.

Variable

Required

Description

SPIDRA_API_KEY

Yes

Your Spidra API key, starting with spd_

SPIDRA_API_URL

No

Override the API base URL, for staging or self-hosted setups. It must be https://, or http:// on localhost, so your key never travels in the clear; the server names the API host on stderr at startup

HTTP_STREAMABLE_SERVER

No

Set to true to serve HTTP at http://localhost:3000/mcp instead of stdio

PORT / HOST

No

Bind address for the HTTP transport. Defaults are 3000 and localhost

SPIDRA_MCP_DOTENV

No

Set to 1 to read a .env file on stdio. Off by default: clients start the server in your project folder, and a .env in a cloned repo could otherwise point SPIDRA_API_URL at someone else's server. Over HTTP, .env is read as before

On the HTTP transport (self-hosted or hosted), the API key can also be sent per request via an Authorization: Bearer header (or X-Spidra-Api-Key), which is useful when one server instance serves more than one user. If a request carries both a header key and a logged-in OAuth session, the header key wins.

OAuth sign-in is only for the hosted endpoint, where Spidra's backend is the identity provider. It turns on when all of these are set:

Variable

Description

MCP_PUBLIC_URL

The server's public origin, e.g. https://mcp.spidra.io. The OAuth issuer; the protected resource is <MCP_PUBLIC_URL>/mcp

MCP_OAUTH_CLIENT_ID / MCP_OAUTH_CLIENT_SECRET

This server's client credentials at the backend's /api/oauth/* endpoints

SPIDRA_API_URL

The backend API, which also hosts the OAuth endpoints

MCP_OAUTH_SIGNING_KEY

Recommended. Key for the sealed client IDs, codes and tokens. Without it, the key is derived from the client secret, so rotating the secret signs everyone out

MCP_OAUTH_SIGNING_KEYS

Optional. Several keys, comma-separated, newest first, for a rotation: new tokens use the first, and the rest still open existing ones

MCP_OAUTH_EXTRA_REDIRECT_URIS

Optional. Comma-separated exact redirect URIs to accept besides loopback addresses and the built-in client callbacks

With OAuth on, every MCP request without credentials gets a 401 with the sign-in challenge, from the first one, which is when Cursor, VS Code, Claude and ChatGPT start sign-in. The discovery documents and sign-in pages stay open. Nothing about OAuth is kept in memory, so restarts and deploys don't sign anyone out, and a backend hiccup answers 503 (retry) rather than anything that makes a client drop the session.

The HTTP server also limits requests per minute: MCP_RATE_ANONYMOUS (60 per IP without credentials), MCP_RATE_AUTHENTICATED (600 per caller) and MCP_RATE_SIGN_IN (30 per IP on the sign-in pages). claude.ai, the Claude apps and ChatGPT call from their own servers, so all their users share a few addresses; those addresses get MCP_RATE_PLATFORM_ANONYMOUS (3,000) and MCP_RATE_PLATFORM_SIGN_IN (600) instead. Anthropic's ranges are built in, and ChatGPT's are loaded from https://openai.com/chatgpt-connectors.json at startup and every 6 hours (MCP_OPENAI_EGRESS_URL points elsewhere, or off). A platform address is only recognised from Cloudflare's CF-Connecting-IP or the socket, never from X-Forwarded-For. On SIGTERM it stops taking new requests and lets calls in flight finish, for up to MCP_DRAIN_MS (50,000), so set PM2's kill_timeout above that.

How to choose a tool

The same guidance is built into the tool descriptions, so the assistant follows it on its own.

  • You don't know the URL: spidra_search. You know the site but not the page: spidra_map, then read the pages you need.

  • One URL: spidra_scrape. Two or more, even to compare two: spidra_batch_scrape, so each gets its own result.

  • Many pages of one kind on a site, URLs unknown: spidra_crawl.

  • A document, not a web page: spidra_parse.

  • A common job on a well-known site: spidra_find_templates, then spidra_run_template. Usually cheaper and more complete than building it from scrapes.

  • Something to keep an eye on: spidra_create_monitor.

Tools that don't wait to the end return an id. The assistant checks it with spidra_get_job instead of starting the job again.

All tools

Group

Tool

What it does

Waits?

Find

spidra_search

Web, news, images, videos, research papers, GitHub

Yes

spidra_map

A site's URLs, optionally ranked by a search

Yes

Read pages

spidra_scrape

One URL as Markdown or JSON, with optional browser actions

Yes

spidra_batch_scrape

2 to 50 URLs, one result each

With wait and up to 10 URLs

spidra_crawl

Every page of one kind on a site

No, returns a jobId

spidra_crawl_pages

A crawl's pages, paged, with file links

Yes

spidra_crawl_report

Why a crawl skipped pages

Yes

spidra_crawl_extract

New extraction over a finished crawl, without re-crawling

No, returns a jobId

spidra_retry_crawl_page

Redo one crawled page's extraction

Yes

Documents

spidra_parse

One or many documents to Markdown or JSON, or the cost with dryRun

Up to about 50 seconds

Ready-made

spidra_find_templates

Search the templates

Yes

spidra_get_template

A template's inputs, columns and limits

Yes

spidra_run_template

Run one, or price it with estimate

Up to about 45 seconds

spidra_list_presets

Saved presets, or one in full

Yes

spidra_save_preset

Save or change a preset

Yes

spidra_run_preset

Run a preset, with optional overrides

Depends on the job

spidra_delete_preset

Delete a preset

Yes

Monitors

spidra_create_monitor

Create, price with estimate, or send a test alert

Yes

spidra_list_monitors

All monitors, or one in full

Yes

spidra_update_monitor

Change, pause or resume

Yes

spidra_delete_monitor

Delete a monitor and its history

Yes

spidra_check_monitor

Run a check now

No, runs in the background

spidra_monitor_changes

What each check found

Yes

Jobs

spidra_get_job

Status and result of any scrape, search, batch, crawl, parse or template run

Yes

spidra_cancel_job

Cancel a batch, crawl or search

Yes

spidra_retry_batch

Re-run a batch's failed URLs

Yes

Account

spidra_account

Credits left, plan, bandwidth, monitor limits, usage

Yes

spidra_logs

Past runs of any tool, or one in full

Yes

Every API capability is covered except three that don't suit an assistant: downloading a crawl as a ZIP file, uploading files over 10 MB (the assistant can pass a link instead), and deleting a stored document early (they're deleted after 24 hours anyway).

Output format

When you need specific fields, ask for JSON and describe them, or give a JSON schema. The assistant gets a small, focused answer instead of a whole page, which is faster and cheaper. Define every field in a schema: an object with no properties comes back empty.

Template rows come back with just the template's table columns by default (name, phone, website and so on). Ask for full detail when you need every field; a Google Maps place has about 50.

Prompts

The server also offers ready-made prompts, which appear in your client's prompt menu (for example / in Claude Code). Each one asks for a few details, then walks the assistant through the right tools, with a cost check first:

Prompt

What it does

Arguments

find_local_leads

Businesses of one kind in an area, from Google Maps

business, area, optional count, contacts

track_prices

Reads current prices, then sets up a monitor for changes

urls, optional schedule, alertWhen

research_topic

Searches, reads the best sources and answers with links

topic, optional sources, depth

crawl_docs

Turns a docs site, or one section, into Markdown

url, optional section, maxPages

extract_document_table

Pulls a table out of a PDF, Word, Excel or PowerPoint file

url, table, optional pages

use_template

Finds and runs the ready-made template for a common job

task

watch_page

Alerts when a page changes

url, optional what, schedule

Credits and how this server protects them

Pages cost 1 credit each, plus AI credits when extraction runs, plus 5 credits per CAPTCHA solved. Search costs 1 credit per 10 results per source. Templates, presets and monitor checks cost what the job behind them costs. Failed pages and failed jobs cost nothing. Agent loops can burn credits quickly, so this server is built to prevent that:

  • Tool descriptions steer the assistant to the narrowest tool and small limits, and spidra_account lets it check the balance before a big job.

  • Templates, monitors and parse can be priced first, for free (estimate and dryRun).

  • spidra_scrape runs once. If a page comes back nearly empty, the result says so and suggests Stealth Mode, rather than quietly paying for a second run.

  • Jobs that don't finish in the call return an id with instructions to check it, never to start again. Duplicate submissions within a few seconds are also merged on the server.

  • Errors say what to do: wait this long (rate limits), fix these fields (validation), or don't retry (out of credits, overdue payment, a plan limit).

  • Cancelling returns the credits held for work that hadn't started.

  • Tools that can spend a lot in one call (batch scrape, crawl, parse, run template, run preset) aren't marked read-only, so clients that ask before acting ask first.

  • No call waits longer than about 45 seconds. Past that you get the job's id, so a client that gives up at 60 seconds doesn't lead to a second run. While it waits, the server sends progress to clients that ask for it, and a call you cancel stops waiting at once.

Output size

Tool results are capped so they fit in the assistant's context: a whole result at 80,000 characters (about 20,000 tokens), with long strings cut to 20,000 characters each, or shorter when that's what it takes to fit. Every cut says how to get the rest. Big lists page with offset and limit, template rows come back trimmed to their columns, and crawl results leave out per-page file links, which spidra_crawl_pages provides. Results also come back as structuredContent for clients that read fields directly, and every tool except spidra_logs publishes an outputSchema for it. The schemas are loose (every field optional, extra fields allowed), so a new API field never fails a call. When a result has too many items to fit even with short strings, lists are cut to their first items with a note, and it stays valid JSON.

Error handling

Errors come back as readable messages, not stack traces, and each one carries guidance the assistant can act on:

{
  "content": [
    {
      "type": "text",
      "text": "Rate limited (TOO_MANY_PENDING_JOBS): You have too many jobs queued. Wait 30 seconds before retrying."
    }
  ],
  "isError": true
}

Transient network failures and 5xx responses are retried automatically with backoff before you ever see an error, courtesy of the underlying Spidra Node SDK.

What changed in 0.8.1

  • Claude and ChatGPT users no longer share one rate limit. claude.ai, the Claude apps and ChatGPT reach the hosted server from a few addresses of their own, so the per-IP limits on sign-in and the first connection were shared by everyone on those apps. Those addresses now get their own, larger limits. Clients on your own machine (Claude Code, Cursor, VS Code and others) are unchanged.

  • A Claude Code plugin. plugin/ bundles this server with a web-data skill; install it with /plugin install spidra --marketplace spidra-io/spidra-mcp-server.

  • Cheaper page reading. spidra_scrape now says that AI extraction grows with a page's length (20 credits or more on a long docs page, not "1 or 2"), and that reading a page as Markdown with no prompt costs 1 credit. In evals, assistants had been summarizing long pages with a prompt for 24 to 32 credits.

  • Tool descriptions no longer list "newer" sites. New sites and templates are added on Spidra's side all the time and need no new version of this server: spidra_find_templates always returns the current list.

What changed in 0.8.0

  • The current MCP revision, 2026-07-28, works. Clients that speak it (current Claude apps and others) were refused by 0.7.0 with "Unsupported protocol version". The server now runs on the official MCP SDK 2.x and answers both the new revision and every 2025-era one, over stdio and HTTP.

  • OAuth sign-in rebuilt: it starts on a 401 the way Claude and ChatGPT expect, supports Client ID Metadata Documents (Claude Code, ChatGPT) as well as dynamic registration, sends the RFC 9207 iss parameter, requires PKCE with S256, and survives server restarts. Redirect addresses are checked strictly.

  • Security: a .env file in your project folder is no longer read on stdio. Clients start npx spidra-mcp in the open project's folder, and in 0.7.0 a .env there (one shipped in a cloned repo, say) could set SPIDRA_API_URL and send your key to another server. SPIDRA_API_URL must now also be https://, or http:// on localhost.

  • Sign-in in Chrome and Safari works, with a redesigned consent page that shows the app's real logo only when its identity is verified. On the hosted server, sign-in now starts when the client first connects, which Cursor needs.

  • spidra_get_job can wait: pass waitSeconds (up to 45) and it answers when the job finishes, instead of the assistant calling it over and over. No call waits longer than about 45 seconds, waits send progress, and cancelling a call stops it.

  • Output schemas and prompts: every tool except spidra_logs publishes an outputSchema, and seven prompts guide common jobs (see Prompts).

  • Tools that can spend a lot in one call ask for confirmation in clients that honor it. Restarts let calls in flight finish.

  • Long-running calls over HTTP stream keepalives, so proxies don't cut them off. Results are compact JSON and stay valid JSON when they're shortened. spidra_logs and spidra_crawl_report take their filters as fixed values, and the crawl report lists the outside_start_path reason.

  • Tool names and arguments are otherwise unchanged.

What changed in 0.7.0

  • New: templates, presets and monitors (spidra_find_templates, spidra_get_template, spidra_run_template, the preset tools and the monitor tools), spidra_account for the credit balance, and spidra_logs for every tool's history.

  • One tool for job status: spidra_get_job replaces spidra_check_scrape_status, spidra_check_search_status, spidra_check_batch_status, spidra_check_parse_status and spidra_check_crawl_status. spidra_cancel_job replaces spidra_cancel_batch and spidra_cancel_crawl. spidra_account replaces spidra_usage, and spidra_logs replaces spidra_scrape_logs. If you allow-list tools by name in your client, update the list.

  • spidra_scrape no longer runs a page twice. It used to retry a thin page in a browser with a proxy, which cost twice. Now it runs once and tells the assistant when Stealth Mode might help.

  • spidra_parse takes several documents at once and prices them with dryRun.

  • Results include structuredContent, error messages are clearer (out of credits, plan limits, validation errors by field), and a failed-scrape filter in the logs works again.

Troubleshooting

  • The assistant does not see any Spidra tools. Restart your client after adding the config. Most clients only read MCP configuration at startup. In Claude Code, run /mcp to check the connection status.

  • "No Spidra API key configured." For local/stdio setups, the SPIDRA_API_KEY variable is not reaching the server, make sure it is inside the env block of the server entry, not at the top level of the config file. For the hosted endpoint, check your header name and value (below).

  • Hosted endpoint returns a 401. Without a key, that's the sign-in prompt: calling a tool starts OAuth in clients that support it, and other clients need a key. With a key, the header is missing or misnamed: it must be exactly Authorization: Bearer spd_... (or X-Spidra-Api-Key: spd_...). A key that was revoked under API Keys fails on the tool call instead, with a message saying so.

  • OAuth login doesn't open a browser, or the client falls back to asking for a key. Not every MCP client supports OAuth yet, use the API key method for that client instead. If a browser window did open but the flow failed partway through, retry; if it keeps failing, confirm you're logged into app.spidra.io in that browser.

  • OAuth sign-in says "This sign-in has expired" or "can't be confirmed". The attempt took longer than 10 minutes, or the link was opened in a different browser from the one that started it. Start the connection again from your client and finish it in the same browser.

  • "Unknown client_id" after upgrading. Clients registered with 0.7.0 have to register again. Most do this on their own when you reconnect; otherwise remove and re-add the connector.

  • A tool call stops after exactly 60 seconds. That's your client's limit, not Spidra's. Codex and Cline default to 60 seconds; raise it to 300 (tool_timeout_sec in Codex, timeout in Cline, timeoutSeconds in Antigravity). The job keeps running either way, so ask for the result with spidra_get_job instead of starting it again.

  • Cursor shows Spidra as connected, but tools fail with "Unauthorized". Cursor didn't start the sign-in. Use the API key config in the Cursor section.

  • A scrape "timed out." The job is still running on the server and nothing is lost. The error includes the job ID, and the assistant fetches the result with spidra_get_job. Bot-protected sites can take a couple of minutes.

  • Results come back empty when using a schema. Check the schema: every field you want must be defined with a type. An object with no properties gives the AI nothing to fill in.

  • npx cannot find the package. Make sure you are on Node 20 or newer and that your network allows access to the npm registry. This only applies to local/self-hosted setups, the hosted endpoint doesn't use npx.

Development

git clone https://github.com/spidra-io/spidra-mcp-server.git
cd spidra-mcp-server
npm install
npm run build       # bundles to dist/index.js
npm test            # spawns the built server against a fake API: every tool, both protocol eras with the
                    # official SDK 2.x and 1.x clients over stdio and HTTP, and the OAuth flow end to end
npm run typecheck

npm test also clicks through the OAuth consent page in Chromium, WebKit and Firefox when Playwright's browsers are installed (npx playwright install chromium webkit firefox), and saves screenshots to SPIDRA_CONSENT_SHOTS (default: a temp folder).

tests/evals/ runs realistic tasks through a real model (Claude Code headless, with only this server's tools) and scores tool choice, arguments, calls, credits and the answer. It spends real credits and model usage, so point it at a local API and run a few tasks at a time: npm run build, then SPIDRA_EVAL_API_URL=http://localhost:4321/api SPIDRA_EVAL_API_KEY_FILE=<key file> node tests/evals/run.mjs --tasks credits_only,price_one (--list shows the tasks).

npm run check:tools calls every tool once against a real Spidra API, with no model involved, and prints ok or the error for each: a search, a map, a few pages, a short crawl and what reads it, a parse, a small template run, and a preset and a monitor that it creates and deletes. It costs a few credits. Run it with SPIDRA_CHECK_API_URL=http://localhost:4321/api SPIDRA_CHECK_API_KEY_FILE=<key file> npm run check:tools; against production it also asks for SPIDRA_CHECK_ALLOW_PRODUCTION=1.

To run your local build against a local Spidra API, set SPIDRA_API_URL=http://localhost:4321/api.

Contributions are welcome. Fork the repository, create a feature branch, make sure npm test passes, and open a pull request.

License

MIT

Available Tools

28 tools
spidra_accountA
Read-only

The user's plan, credits left this billing period, Stealth Mode bandwidth, and how many monitors the plan allows. Use it before a large batch, crawl or template run to check there's enough, or to answer "how many credits do I have".

  • "usage" adds credits, requests and tokens per day ("7d", "30d") or per week ("weekly").

  • "byTool": true adds credits per day split by tool (scrape, crawl, search, parse, map) for the last 7 days, or the "usage" range.

"credits.used" includes credits held by jobs still running; unused held credits come back when they finish.

Inputs: usage, byTool.

spidra_batch_scrapeA
Read-only

Read 2 to 50 known URLs, each on its own with its own result, using the same prompt or schema for all.

Use it for anything with more than one URL: product pages, listings, articles, or just two pages to compare. Waiting: with "wait": true and 10 URLs or fewer, it waits (up to a few minutes) and returns the finished batch. Otherwise it returns a "batchId" straight away; check it with spidra_get_job (type "batch") every 10 to 15 seconds until "status" is completed or cancelled. Don't start the batch again while it runs.

  • "output" is "markdown" (default) or "json"; a schema sets "json". Popular-site pages also get "structuredData".

  • A completed batch can still have failed items: check "failedCount" and each item's "error". spidra_retry_batch re-runs only the failed ones.

  • Browser actions aren't supported in a batch; use spidra_scrape for pages that need them.

Cost: per URL, the same as spidra_scrape. Failed URLs cost nothing.

Example: {"urls": ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"], "prompt": "Title and price", "output": "json", "wait": true}

Inputs: urls (required), wait, output, prompt, +6 more.

spidra_cancel_jobA
DestructiveIdempotent

Cancel a running batch, crawl or search.

  • "batch": URLs not yet started are stopped and their held credits returned; finished ones keep their results. A cancelled batch can't be retried.

  • "crawl": stops at its next check. If it had returned pages, it ends completed with those (charged as usual); if not, cancelled and free. spidra_crawl_pages still reads its pages.

  • "search": one not yet started is removed; a running one stops at its next checkpoint and then reports failed with "Search cancelled".

Inputs: id (required), type (required).

spidra_check_monitorA

Run a monitor's check now, outside its schedule. It runs in the background, costs what a scheduled check costs, and alerts like one. Once a minute at most per monitor. Read the result with spidra_monitor_changes after a little while.

Inputs: monitorId (required).

spidra_crawlA
Read-only

Start at one URL and follow links to the kind of pages you describe, optionally extracting data from each. Returns a "jobId" straight away.

Use it when you need many pages of one kind and don't know their URLs (every blog post, every product in a category, every page of docs). Don't use it when you know the URLs (spidra_batch_scrape) or only need a site's list of URLs (spidra_map, much cheaper). Then: check it with spidra_get_job (type "crawl") every 10 to 15 seconds until it's completed, failed or cancelled. Don't start it again while it runs. Stop a mistake with spidra_cancel_job.

  • "crawlInstruction" says which pages to return, e.g. "Individual book pages" or "Blog posts, not tag pages".

  • "transformInstruction" or "schema" extracts data from each page with AI. Leave both out for each page as Markdown, with no AI cost.

  • "maxPages" counts only pages returned (default 5). Pages read just to find them are free. Plan limits per crawl: Free 100, Starter 1,000, Builder 5,000, Pro 10,000.

  • The crawl stays under the start URL's path unless "crawlEntireDomain" is true.

  • A page you expected is missing? spidra_crawl_report says why each link was skipped.

Cost: 1 credit per page returned, plus AI credits with a transformInstruction or schema. A crawl that returns nothing is free.

Example: {"baseUrl": "https://books.toscrape.com", "crawlInstruction": "Individual book pages", "transformInstruction": "Title and price", "maxPages": 10}

Inputs: baseUrl (required), crawlInstruction (required), schema, cookies, +12 more.

spidra_crawl_extractA

Run a new extraction over every page a finished crawl saved, without visiting the site again. Much cheaper than crawling again: only AI credits are charged. Returns a new "jobId"; check it with spidra_get_job (type "crawl"). The original crawl doesn't change.

Inputs: jobId (required), transformInstruction (required).

spidra_crawl_pagesA
Read-only

Read a crawl's pages, in order, with each page's data and links to its Markdown and HTML (the links last 1 hour). Use it for crawls of more than 100 pages, since spidra_get_job returns only the first 100, or after a cancelled crawl to get the pages it finished.

Page through with "offset" and "limit" until "hasMore" is false. Keep "limit" small (20 to 50) to save context. Each page's "id" is what spidra_retry_crawl_page needs.

Inputs: jobId (required), limit, offset.

spidra_crawl_reportA
Read-only

Explain a finished crawl: how it read the instruction, how many pages it read only to find the wanted ones (free), and every link it didn't follow, counted by reason. Pass "reason" to list the URLs for one reason, each with the page it was found on.

Use it when a page you expected is missing, then adjust crawlInstruction, includePaths or maxPages and crawl again. Reasons: not_relevant, outside_domain, include_pattern, exclude_pattern, file_type, other_version_or_language, duplicate_page, redirected_off_site, robots_txt, action_link. The report is empty while the crawl runs.

Inputs: jobId (required), limit, offset, reason.

spidra_create_monitorA

Watch a template, a preset or a list of pages on a schedule and get an alert when something changes: new or removed items, changed fields (like a price), or page text changes.

  • The first check runs straight away and records the starting point without alerting.

  • "estimate": true returns the credits per check and per month, for free, without creating it.

  • "testNotification": true sends a sample alert to the "notify" channels, without creating anything, to check a webhook, Slack or Discord URL.

  • The plan limits how many monitors the user can have and how often they check (spidra_account shows it).

  • Confirm the schedule and channels with the user before creating, since each check costs credits.

Cost: each check costs what running its sources costs. A "goal" adds a little AI on checks where something changed.

Example: {"name": "Cable price", "sources": [{"kind": "urls", "urls": ["https://www.amazon.com/dp/B0CG1LGWR6"]}], "schedule": {"every": "1d", "at": "09:00", "timezone": "Africa/Lagos"}, "alertOn": "changed", "watchFields": ["price"], "goal": "Only tell me if the price drops"}

Inputs: goal, name, rule, notify, +7 more.

spidra_delete_monitorA
DestructiveIdempotent

Delete a monitor and its check history. To stop checks but keep the history, pause it with spidra_update_monitor instead. Confirm with the user before deleting.

Inputs: monitorId (required).

spidra_delete_presetA
DestructiveIdempotent

Delete a saved preset. Runs already made from it stay in the history. A monitor that watches it starts failing until its sources are changed. Confirm with the user before deleting.

Inputs: presetId (required).

spidra_find_templatesA
Read-only

Find ready-made templates: jobs that turn a few inputs into a table of results, such as Google Maps leads with emails, Amazon keyword rank, LinkedIn or Indeed jobs, YouTube transcripts, Zillow homes or Google Flights prices. Free.

Use it first when the task is a common data job on a well-known site. A template is usually cheaper and more complete than writing scrapes yourself. Then: spidra_get_template for its inputs, and spidra_run_template to run it.

Filter with "q" (words, e.g. "reviews"), "site" (e.g. "google-maps", "amazon") or "category". With no filter you get every template; the response also lists the site keys and categories to filter by.

Inputs: q, site, category.

spidra_get_jobA
Read-only

Get any job's status, and its result once finished. Use it for the ids other tools return when they don't wait to the end.

  • "scrape", "search": a jobId. Readable for about an hour after it finishes (failed ones 24 hours); after that, use spidra_logs.

  • "batch": a batchId. A completed batch can still have failed items; check "failedCount".

  • "crawl": a jobId. Returns the first 100 pages; spidra_crawl_pages reads the rest.

  • "parse": a jobId. "includePages" adds each page's Markdown.

  • "parse_batch": a batchId.

  • "template_run": a runId. Rows come back with the template's columns ("detail": "full" for every field) and page with "offset" and "limit".

Check every 10 to 15 seconds while it runs. Statuses: queued, waiting, active and running are still in progress; completed, partial, failed and cancelled are final.

Inputs: id (required), type (required), limit, detail, +2 more.

spidra_get_templateA
Read-only

Get what a template needs and what it returns: its inputs (send them by "key" to spidra_run_template; "required" ones must be set, others have defaults), its columns, and its limits in plain words. Free.

Inputs: slug (required).

spidra_list_monitorsA
Read-only

List the user's monitors with their schedule, last check and next check. Pass "monitorId" to get one in full. What changed is in spidra_monitor_changes.

Inputs: monitorId.

spidra_list_presetsA
Read-only

List the user's saved presets, newest first: saved template runs (a template and its inputs) and saved tool requests. Pass "presetId" to get one in full. Run one with spidra_run_preset.

Inputs: page, limit, presetId.

spidra_logsA
Read-only

Find past runs, newest first, or open one by its id. Every run is logged, including ones made in the dashboard, and logs last after the job itself can no longer be read with spidra_get_job.

  • "tool": scrape, search, map, parse, crawl, batch or template.

  • "id": open one log in full, with its result. For scrape, search and map this is the log's "uuid" (not the job's id); for crawl and template it's the job's id. A parse log's "jobId" opens with spidra_get_job (type "parse"), and a batch with spidra_get_job (type "batch").

  • "status" and "searchTerm" filter scrape, search, map and parse logs. Scrape: success, error, in_progress. Search: also cancelled. Map: success, empty, error. Parse: queued, running, completed, failed.

Inputs: tool (required), id, page, limit, +2 more.

spidra_mapA
Read-only

List a website's page URLs without scraping them, from its sitemaps and the links on its pages. Waits and returns the URLs, usually in a few seconds.

Use it when you need the right pages on a site before reading them ("which page on stripe.com covers refunds?"), or a full list of a site's URLs. Then: pick the URLs you need and read exactly those with spidra_scrape or spidra_batch_scrape. That's usually cheaper than spidra_crawl when you only need a few pages.

  • "search" keeps only matching URLs, best matches first. Keep "limit" small when you only need a few.

  • "total" is how many URLs matched; "scanned" is how many were found in all.

Cost: 1 credit per call. Free when no URLs are found.

Example: {"url": "https://docs.python.org/3/", "search": "asyncio", "limit": 20}

Inputs: url (required), limit, search, cookies, +4 more.

spidra_monitor_changesA
Read-only

A monitor's checks, newest first, with a one-line summary of each. Pass "checkId" for everything one check found, per source: new, changed and removed items (with old and new values), and page text changes. A source's first check has "baseline": true and only records the starting point. Checks are kept for 30 days.

Inputs: monitorId (required), page, limit, checkId.

spidra_parseA
Read-only

Turn documents into Markdown, page by page, and into JSON with a "prompt" or "schema". PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, CSV and HTML files work, and scanned pages are read with AI.

Use it for reports, papers, invoices, contracts and spreadsheets, online or given to you as a file. Don't use it for web pages (spidra_scrape).

Send the documents one way:

  • "url": a link to one document.

  • "fileBase64" and "fileName": a file you have, base64-encoded (up to about 10 MB).

  • "fileId": a document parsed in the last 24 hours, to parse again with other options.

  • "urls" or "fileIds": several documents at once (up to 50), each parsed with the same options.

  • "pages" ("1-5, 9" or "10-") or "maxPages" parses only part of a long PDF.

  • "dryRun": true returns the page count and cost, for free, without parsing.

  • It waits up to about 50 seconds. A long document or a batch still running returns an id; check it with spidra_get_job (type "parse" or "parse_batch").

Cost: 1 credit per page parsed, plus AI credits with a prompt or schema. Formats without pages count one page per 3,000 characters. Failed pages and documents cost nothing.

Example: {"url": "https://arxiv.org/pdf/1706.03762", "pages": "1-3"}

Inputs: url, mode, urls, pages, +9 more.

spidra_retry_batchA

Run a batch's failed URLs again with the batch's original options; the rest are left alone. Use it after spidra_get_job shows "failedCount" above 0. Cookies are never stored, so pass "cookies" again if the batch used them. Then check the batch with spidra_get_job as before. A URL that fails again costs nothing.

Inputs: batchId (required), cookies.

spidra_retry_crawl_pageA

Run a crawl's extraction again on one saved page and return the new data. It doesn't visit the site, and the page's stored data is replaced. Use it when one or two pages came back empty or wrong. Get "pageId" from spidra_crawl_pages (each page's "id"). Only crawls with a transformInstruction or schema can be retried. Costs only the AI credits for that page.

Inputs: jobId (required), pageId (required).

spidra_run_presetA
Read-only

Run a saved preset. "overrides" changes template inputs, or request fields of a tool preset, for this run only.

A template preset returns its template run (check a running one with spidra_get_job, type "template_run"). A tool preset returns that tool's own answer: map links straight away, or a scrape, batch, crawl or search job id to check with spidra_get_job. Costs what the same job costs when run directly.

Inputs: presetId (required), overrides.

spidra_run_templateA
Read-only

Run a template and get its rows. Most runs finish in this call (it waits up to about 45 seconds). A longer one returns a "runId"; check it with spidra_get_job (type "template_run") every 10 to 15 seconds until "status" isn't "running".

  • Get the inputs from spidra_get_template first. Send them in "inputs" by key.

  • "estimate": true checks the inputs and returns the credits it should cost, for free, without running.

  • Rows come back with the template's columns. "detail": "full" returns every field. Rows page with "offset" and "limit" (default 50).

  • A "partial" run had some pages fail; its rows are still there, with the failures listed.

Cost: what the same job costs with the tool behind it, usually 1 credit per page read. Failed pages and failed runs cost nothing.

Example: {"slug": "google-maps-leads", "inputs": {"query": "dentist", "areas": ["Lekki, Lagos"]}}

Inputs: slug (required), inputs (required), limit, detail, +2 more.

spidra_save_presetA

Save a job so it can run again with one call, or change a saved one.

  • A template preset: "kind": "template", the template's "template" slug and its "inputs".

  • A tool preset: "kind": "tool", the "tool" (scrape, batch, crawl, map, search or parse) and "config", the request body exactly as that tool takes it. Cookies are never stored.

  • To change one, pass its "presetId" and only the fields that change ("name", "description", "inputs" or "config").

Presets can also be watched for changes with spidra_create_monitor.

Inputs: kind, name, tool, config, +4 more.

spidra_scrapeA

Read one known URL and get its content as Markdown, or the data you describe as JSON. Waits and returns the result, usually in a few seconds (up to a minute for hard pages).

Use it for one URL. Links to PDF, Word, Excel and PowerPoint files work too. Don't use it for more than one URL, even to compare two: spidra_batch_scrape gives each URL its own result. To find pages on a site, use spidra_map or spidra_crawl.

  • No "prompt" or "schema": the page as Markdown.

  • "prompt": what you want, in plain words; the AI picks the shape. Add "schema" for exact fields and types (null where the page has nothing).

  • Popular sites (Amazon, Google Maps, LinkedIn, YouTube, Zillow, Indeed and others): with "output": "json" and no prompt, "content" is the page's data in named fields, no AI needed. Otherwise that data is in "structuredData".

  • "actions" runs browser steps first (click, type, scroll, forEach to go through a list); "instruction" does one task described in plain words instead. Never both. Steps run once and are never retried automatically, since a step like a form submission has effects.

  • If the result has "lowContent": true, the page came back nearly empty, usually because the site blocks automated visits. Call again with "useProxy": true.

Cost: 1 credit, plus AI credits for the tokens used with a prompt or schema (often 1 or 2), plus 5 per CAPTCHA solved. A page that can't be read costs nothing.

Browser actions: up to 50 steps (click, type, check, uncheck, hover, select, press, wait, waitFor, scroll, scrollUntilEnd, evaluate, screenshot, forEach), with waits adding up to 60 seconds at most. "selector" is CSS, XPath, or plain words ("the Sign in button"); prefer CSS when you know it. A failed step fails the job (free) with "Action N (...) failed: "; mark steps that may not be there, like cookie banners, "optional": true. forEach goes through a list of items and must be the single, last action. "actionResults" reports each step as ok, skipped or failed.

Example: {"url": "https://books.toscrape.com", "prompt": "Return the title and price of every book", "output": "json"} Example with actions: {"url": "https://quotes.toscrape.com/login", "actions": [{"type": "type", "selector": "#username", "value": "demo"}, {"type": "type", "selector": "#password", "value": "demo"}, {"type": "click", "selector": "input[type='submit']"}, {"type": "waitFor", "selector": "a[href='/logout']"}], "prompt": "List the first 3 quotes with their authors", "output": "json"}

Inputs: url (required), output, prompt, schema, +10 more.

spidra_update_monitorA

Change a monitor, or pause and resume it. Send only what changes; the fields are the same as spidra_create_monitor. "status": "paused" stops checks and keeps the history; "active" starts them again. Changing "sources" restarts the comparison for new sources, and "notify" replaces all channels.

Inputs: monitorId (required), goal, name, rule, +6 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 28 tools

    Disambiguation4/5

    Core scraping, search, map, parse, template, and monitor tools have explicit use/don't-use guidance, so major choices are clear. Some boundaries remain around job/result retrieval (spidra_get_job vs spidra_crawl_pages vs spidra_logs) and retry tools, requiring description reading.

    Naming Consistency3/5

    All tools use the spidra_ prefix, which helps grouping. However patterns mix bare verbs (spidra_scrape, spidra_search, spidra_map), verb_noun (spidra_get_job, spidra_create_monitor), noun_verb (spidra_batch_scrape), and noun-only names (spidra_account, spidra_logs).

    Tool Count3/5

    28 tools is heavy for an MCP surface and exceeds the typical 3-15 range. The categories are coherent and many tools are justified by the platform's breadth, but the set is still at the borderline-to-too-many end.

    Completeness4/5

    The surface covers single/batch scraping, crawling, URL mapping, search, document parsing, templates, presets, monitors, job control, account, and logs. Lifecycle coverage is strong; only minor gaps like consolidated job listing or broader job management remain.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers