Skip to main content
Glama
OfficialSpicrawl

Spicrawl MCP Server

Spicrawl MCP Server: web scraping for Claude, Cursor and AI agents

To let Claude, Cursor or another AI agent scrape websites, connect the Spicrawl MCP server. Add the hosted endpoint https://mcp.spicrawl.com/mcp (or run npx -y @spicrawl/mcp locally) with a Spicrawl API key, and the agent can call spicrawl_scrape to read any URL as Markdown, text, HTML or JSON. It renders JavaScript pages, extracts structured data, scrapes up to 10,000 URLs in one batch job, takes screenshots and keeps logged-in browser sessions, all as native spicrawl_* tools in Claude Code, Claude Desktop, Cursor, VS Code, Windsurf, Gemini CLI and Codex. The free tier includes 1,000 credits per month.

npm version npm downloads license CI MCP Registry Glama Dify Marketplace Cursor Directory

Docs · Get an API key · TypeScript SDK · CLI · Agent plugins · Changelog

Install from: npm · Smithery · Glama · MCP Registry · Dify plugin · Cursor Directory

Quickstart

  1. Create an API key at app.spicrawl.com. Keys start with spicrawl_live_ or spicrawl_test_.

  2. Connect the hosted endpoint, https://mcp.spicrawl.com/mcp (MCP Streamable HTTP, nothing to install):

    export SPICRAWL_API_KEY=spicrawl_live_...
    claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
      --header "Authorization: Bearer $SPICRAWL_API_KEY"

    Codex: codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

  3. Or run it locally over stdio (Node.js 20+), for clients that only launch local servers:

    claude mcp add spicrawl --env SPICRAWL_API_KEY=$SPICRAWL_API_KEY -- npx -y @spicrawl/mcp
  4. Ask your agent: "Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its price." It calls spicrawl_scrape and gets the page back as Markdown.

Both transports serve the same spicrawl_* tools (25 are defined; the hosted endpoint lists 24 until remote browsers launch, see Tools), and every call runs under your key with the same scopes, limits and credits as a direct API call.

Related MCP server: snoopscan

What it does

  • Scrape any URL to Markdown, text, HTML, a JSON envelope or a printed PDF with one tool, spicrawl_scrape. Markdown is the default and strips the page to its main content.

  • Render JavaScript with render: true (3 credits). A plain fetch costs 1 credit, screenshots, PDFs and actions use a full browser (8 credits), and mode: "auto" escalates from fetch to browser and bills only the rung that worked (price table). You never pick the rendering yourself; Spicrawl chooses the cheapest that can do what the request asks.

  • Extract structured data with a CSS/XPath selector map (extract) or from the JSON-LD, OpenGraph, Twitter Card and microdata a page publishes (autoparse).

  • Drive the page before capture with 8 typed browser actions: click, fill, wait_for, wait_for_navigation, scroll (including infinite scroll), select, evaluate and screenshot.

  • Return screenshots and PDFs the model can see: screenshots arrive as MCP image blocks (up to 5 MB of base64 each) and printed pages as application/pdf resource blocks (up to 10 MB).

  • Batch scrape thousands of URLs as one async job, up to 10,000 items per call, with 9 tools to submit, poll, page through results, retry, cancel and keep a job open so you can append URLs as you discover them.

  • Keep a logged-in browser with persistent sessions: cookies, storage and a fixed rendering tier reused across scrapes.

  • Answer "what did that cost?" from usage and request-log tools, and read the Spicrawl docs from inside the agent.

Use cases

  • Scrape docs into a RAG knowledge base. Read documentation pages as clean Markdown with spicrawl_scrape (main_content_only is on by default for Markdown), or queue up to 10,000 URLs in one spicrawl_batch_submit job. Spicrawl returns the Markdown; chunking and embedding are yours. See the knowledge-base recipe.

  • Monitor prices. Pull a product's title, price and availability with an extract selector map or autoparse, and set cache: false for a fresh reading. A plain-fetch check costs 1 credit, a rendered one 3. Run it on your own schedule. See price monitoring.

  • Extract job postings. Collect listings from job boards, including infinite-scroll pages (render: true with a scroll action), as structured fields. See job postings.

  • Read JavaScript-heavy pages. When a plain fetch returns an empty shell, call spicrawl_scrape with render: true (3 credits) so the page builds itself first.

Check each site's terms and robots.txt before you scrape it.

Which tool to use

  • If you need one page's content or data now, use spicrawl_scrape.

  • If the page is blank or a skeleton without JavaScript, call spicrawl_scrape again with render: true.

  • If you know the page's structure, use extract with CSS selectors; if you don't, try autoparse.

  • If you have more than about 20 URLs, use spicrawl_batch_submit, then spicrawl_batch_status and spicrawl_batch_results.

  • If you discover URLs as you go, submit with open: true, append with spicrawl_batch_add_items and finish with spicrawl_batch_close.

  • If you must log in first, use spicrawl_session_create and pass its id as session_id to spicrawl_scrape.

  • If a call failed and you need to know why, use spicrawl_request_get with the request id, or spicrawl_requests_list with only_errors.

  • If you are unsure of a parameter or an error code, use spicrawl_docs_search before guessing.

Tools

25 tools, grouped by task. The hosted endpoint lists 24 until remote browsers launch: its keyless tools/list and the server card leave out spicrawl_browser_connect_url (Coming soon), while a session opened with your key and the stdio package (npx -y @spicrawl/mcp) list all 25. Tools reject unknown arguments, so a misspelt field fails with an error naming it instead of being silently dropped. Every tool sets a title and explicit readOnlyHint, destructiveHint and openWorldHint annotations, so a client knows which calls to confirm: spicrawl_scrape is not read-only (its method and actions can submit forms), and spicrawl_batch_cancel, spicrawl_session_release and spicrawl_session_delete are destructive. Every tool also declares an outputSchema and returns matching structuredContent beside the same text it always returned, so a client can read fields (credits, status, next_cursor, ...) without parsing text. The schemas list what an agent can rely on and stay open: fields the API adds later pass through, and statuses and tiers are plain strings. A tool that returns a document or a JSON array (spicrawl_docs_read, spicrawl_batch_task_content) puts it under content in structuredContent. Full argument reference: docs.spicrawl.com/agents/mcp.

Scrape a website to Markdown

Tool

What it does

Key parameters

spicrawl_scrape

Retrieve one URL as markdown (default), text, html, json or pdf, with optional rendering, extraction, links, screenshots and browser actions.

url, format, render, mode, main_content_only, include_tags, exclude_tags, extract, autoparse, links, screenshot, actions, wait_for, session_id, proxy, cache, max_cost

Batch scrape many URLs

Tool

What it does

Key parameters

spicrawl_batch_submit

Queue many URLs as one async job with shared settings; returns the job with estimated_credits.

urls or items, render, format, open, max_attempts, credit_budget, max_cost

spicrawl_batch_list

List the project's batch jobs, newest first.

status, limit (max 200), cursor

spicrawl_batch_status

One job's status and progress.

job_id

spicrawl_batch_results

A page of finished items; works while the job runs.

job_id, status, limit (max 5000), cursor

spicrawl_batch_task_content

The full payload of one item.

job_id, seq

spicrawl_batch_cancel

Cancel a job; finished items keep their results.

job_id

spicrawl_batch_retry

Re-queue every failed item.

job_id

spicrawl_batch_add_items

Append URLs to a job submitted with open: true.

job_id, urls or items

spicrawl_batch_close

Stop an open job accepting items so it can complete.

job_id

Browser sessions

Tool

What it does

Key parameters

spicrawl_session_create

Create a persistent browser identity (cookies, storage, a fixed rendering tier).

engine, ttl_seconds, session_context

spicrawl_session_list

List sessions, newest first.

status, engine, limit, cursor

spicrawl_session_get

One session's status, rendering tier, exit, usage and expiry.

session_id

spicrawl_session_context

Dump a live session's cookies and storage (contains secrets).

session_id

spicrawl_session_release

End a session; its record stays as released.

session_id

spicrawl_session_delete

Permanently delete a session and its state.

session_id

Usage and request history

Tool

What it does

Key parameters

spicrawl_usage

Billing-grade usage over a date window (read scope).

from, to, group_by, metrics

spicrawl_usage_summary

Current-period totals (read scope).

none

spicrawl_usage_reconciliation

One day's drift between the request log and the billed rollup (read scope).

day

spicrawl_requests_list

Recent API calls with status, error code, rendering tier, latency and credits.

only_errors, status, limit, all_projects

spicrawl_request_get

The full log record of one call.

id

Docs

Tool

What it does

Key parameters

spicrawl_docs_search

Search the Spicrawl docs; an error code also links to its entry.

query, limit (1-20, default 8)

spicrawl_docs_read

Read one docs page as Markdown (truncated over 60,000 characters).

path

spicrawl_docs_index

The docs' llms.txt: every page with its title and .md link.

none

Coming soon

spicrawl_browser_connect_url will mint a single-use CDP WebSocket URL for driving a Spicrawl-hosted browser from any CDP client; remote browsers are not available yet. It is the 25th tool and the only one the hosted keyless listing and the server card leave out (24 tools there). The schemas also accept ai_extract, stealth, extract_preset, premium_proxy, proxy_country and sticky_key ahead of launch. Agents should not send them yet. To list only what works today, set SPICRAWL_MCP_HIDE_UNAVAILABLE=1 (see Environment variables).

Example prompts

  • "Scrape https://example.com/pricing as markdown and summarise the plans."

  • "Get the product name, price and stock status from this URL." (spicrawl_scrape with extract or autoparse)

  • "Take a full-page screenshot of https://example.com after the cookie banner is gone." (render, actions, screenshot_fullpage)

  • "Scrape these 500 URLs and give me each page title." (spicrawl_batch_submit, then spicrawl_batch_results)

  • "Why did my last scrape fail?" (spicrawl_requests_list with only_errors, then spicrawl_request_get)

  • "How many credits have I used this month?" (spicrawl_usage_summary)

Set up your MCP client

Put your key in SPICRAWL_API_KEY and reference it from the config where the client allows, so the key never lands in a committed file.

Hosted endpoint, stored for you only (add --scope user for every project):

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

To share it with a team, commit .mcp.json at the project root. Claude Code expands ${VAR}, so each teammate's own key is used:

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

Or install the plugin, which bundles the MCP server and the Spicrawl agent skill:

claude plugin marketplace add OfficialSpicrawl/agent-plugins
claude plugin install spicrawl@spicrawl-plugins

Check with claude mcp list, or /mcp in a session. More: docs.spicrawl.com/agents/claude-code.

Claude Desktop's claude_desktop_config.json starts local (stdio) servers only, and custom connectors cannot send an Authorization header, so run the package locally. The file is at ~/Library/Application Support/Claude/claude_desktop_config.json on macOS and %APPDATA%\Claude\claude_desktop_config.json on Windows:

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

Restart Claude Desktop after saving.

Use the Add to Cursor button above, or add this to .cursor/mcp.json in the project (or ~/.cursor/mcp.json for every project). Cursor resolves ${env:NAME} in url and headers, so start Cursor from an environment where SPICRAWL_API_KEY is set:

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

More: docs.spicrawl.com/agents/cursor.

Use the Install in VS Code button above, or add this to .vscode/mcp.json. VS Code uses the servers key, and the input variable makes it ask for the key once and store it securely, so the file is safe to commit:

{
  "inputs": [
    { "type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true }
  ],
  "servers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
    }
  }
}

VS Code does not expand ${env:...} inside headers, so use the input variable. More: docs.spicrawl.com/agents/github-copilot.

Windsurf is now named Devin Desktop. For the default Devin Local agent, add the server in local scope (saved to the gitignored .devin/mcp_config.local.json):

devin mcp add spicrawl https://mcp.spicrawl.com/mcp -H "Authorization: Bearer $SPICRAWL_API_KEY"

For the legacy Cascade agent, edit ~/.config/devin/mcp_config.json (%APPDATA%\devin\mcp_config.json on Windows). Cascade reads the URL from serverUrl and replaces ${env:...}:

{
  "mcpServers": {
    "spicrawl": {
      "serverUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

More: docs.spicrawl.com/agents/windsurf.

Gemini CLI strips variables whose names contain KEY, TOKEN or AUTH before expanding headers, so export the key under another name:

export SPICRAWL_MCP_BEARER="$SPICRAWL_API_KEY"

Then add to ~/.gemini/settings.json (or .gemini/settings.json for one project). Streamable HTTP uses the httpUrl key:

{
  "mcpServers": {
    "spicrawl": {
      "httpUrl": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer $SPICRAWL_MCP_BEARER" }
    }
  }
}

Or: gemini mcp add --scope user --transport http --header 'Authorization: Bearer $SPICRAWL_MCP_BEARER' spicrawl https://mcp.spicrawl.com/mcp. Run /mcp to check. More: docs.spicrawl.com/agents/gemini-cli.

Codex reads the key from the environment each time it connects, so the key never lands in a file:

codex mcp add spicrawl --url https://mcp.spicrawl.com/mcp --bearer-token-env-var SPICRAWL_API_KEY

The equivalent ~/.codex/config.toml (or .codex/config.toml in a trusted project):

[mcp_servers.spicrawl]
url = "https://mcp.spicrawl.com/mcp"
bearer_token_env_var = "SPICRAWL_API_KEY"

Local stdio instead:

[mcp_servers.spicrawl]
command = "npx"
args = ["-y", "@spicrawl/mcp"]
env = { SPICRAWL_API_KEY = "spicrawl_live_..." }

More: docs.spicrawl.com/agents/codex.

A gateway that passes a connection parameter through as a header cannot always send Authorization: Bearer <key>: Smithery maps a parameter to a header but sends the raw value, with no Bearer prefix, and a client often cannot name its source authorization. /mcp takes the key in any of these headers:

Header

Value

x-api-key

spicrawl_live_... (recommended for a gateway)

api-key

spicrawl_live_...

Authorization

spicrawl_live_... (the raw key, no Bearer ), or Bearer spicrawl_live_...

In Smithery, make the API key a connection parameter, map it to the x-api-key header, and mark it secret. Or map it to Authorization and send the raw key; the server accepts that when the whole value is a Spicrawl key.

The key can be left optional in the connection schema (no required): initialize, ping and tools/list on /mcp answer without a key, with the same tool list as the server card (24 tools, without spicrawl_browser_connect_url), so a directory can list the tools. Every tool call still needs a valid key and is refused with 401 otherwise; no session is opened and nothing is billed or read without one. Say in the parameter's description that it is needed to run any tool.

If more than one is sent, the first that holds a well-formed key wins, in the order Authorization: Bearer, raw Authorization, x-api-key, api-key. A key in the query string is not read, because URLs end up in access logs. /chatgpt/mcp takes OAuth access tokens only and accepts none of these.

Setting

Value

Transport

Streamable HTTP

URL

https://mcp.spicrawl.com/mcp

Header

Authorization: Bearer <your key> (or x-api-key: <your key>)

A stdio-only client runs npx -y @spicrawl/mcp with SPICRAWL_API_KEY in its environment. The server accepts API keys only, not OAuth. Setup pages for more clients (OpenCode, Zed, JetBrains, Cline, Goose and others): docs.spicrawl.com/agents/mcp.

Environment variables

Variable

Default

Used by

Purpose

SPICRAWL_API_KEY

none (required)

stdio

Your API key. The HTTP server ignores it and uses each caller's bearer token.

SPICRAWL_BASE_URL

https://api.spicrawl.com (stdio), http://127.0.0.1:8080 (HTTP)

both

The Spicrawl API the tools call.

SPICRAWL_PUBLIC_BASE_URL

SPICRAWL_BASE_URL

both

The API as the end user reaches it, used only in URLs handed back.

SPICRAWL_DOCS_URL

https://docs.spicrawl.com

both

Docs base URL for the docs tools. Your key is never sent there.

SPICRAWL_DOCS_HOST

none

both

Legacy: an origin whose docs are at /docs. Read only when SPICRAWL_DOCS_URL is unset.

SPICRAWL_MCP_ADDR

127.0.0.1:8090

HTTP

Listen address, host:port.

SPICRAWL_MCP_SESSION_IDLE_MS

1800000 (30 min)

HTTP

Idle MCP sessions are dropped after this long.

SPICRAWL_MCP_HIDE_UNAVAILABLE

off

both

1, true, yes or on leaves out what is announced but not available yet: spicrawl_browser_connect_url is not registered (24 tools instead of 25) and every argument marked "Coming soon" is dropped from the schemas (ai_extract, stealth, extract_preset, premium_proxy, proxy_country, sticky_key, and the session rotate_ip, region_pool and fingerprint). Read when the server starts a session, so restart the process after changing it. The keyless tools/list and the server card behave as if it were always on.

SPICRAWL_CHATGPT_ENABLED

off

HTTP

1, true, yes or on turns on the ChatGPT endpoint. Off, /chatgpt/mcp, its metadata and the OpenAI challenge all answer 404.

SPICRAWL_CHATGPT_RESOURCE

https://mcp.spicrawl.com/chatgpt/mcp

HTTP

The OAuth resource identifier. Tokens must be issued for exactly this aud; the metadata is published at /.well-known/oauth-protected-resource + its path.

SPICRAWL_OAUTH_ISSUER

https://app.spicrawl.com

HTTP

The authorization server, listed in the metadata. Tokens are introspected at <issuer>/api/oauth/introspect.

SPICRAWL_OAUTH_INTROSPECT_SECRET

none

HTTP

This server's bearer secret at the introspection endpoint. Unset, every token is answered 503.

SPICRAWL_CHATGPT_ROOT_PRM

off

HTTP

Also serve the metadata at the root /.well-known/oauth-protected-resource. Off by default because /mcp takes API keys and must not advertise OAuth.

SPICRAWL_OPENAI_APPS_CHALLENGE

none

HTTP

The OpenAI domain-verification token served at /.well-known/openai-apps-challenge.

SPICRAWL_OPENAI_APPS_CHALLENGE_FILE

none

HTTP

A file holding that token (surrounding whitespace and the trailing newline are dropped). Read when SPICRAWL_OPENAI_APPS_CHALLENGE is unset.

With no docs variable set, a server pointed at a self-hosted API reads that API's own docs at <SPICRAWL_PUBLIC_BASE_URL or SPICRAWL_BASE_URL>/docs.

Self-host the MCP server over HTTP

The hosted endpoint runs this package's Streamable HTTP server. To run your own:

SPICRAWL_MCP_ADDR=127.0.0.1:8090 \
SPICRAWL_BASE_URL=https://api.spicrawl.com \
npx -y @spicrawl/mcp --http
  • SPICRAWL_BASE_URL defaults to http://127.0.0.1:8080 in HTTP mode (a Spicrawl API on the same host), not to the public API. Set it to https://api.spicrawl.com to front the public API.

  • MCP is served at POST/GET/DELETE /mcp; liveness at GET /healthz (no auth).

  • GET /.well-known/mcp/server-card.json (no auth, Cache-Control: public, max-age=3600) is a static server card for directories such as Smithery that cannot scan /mcp without a key: serverInfo, the transport, bearer auth and the tool list. The tools are read from the same definitions as tools/list, as if SPICRAWL_MCP_HIDE_UNAVAILABLE were on: 24 tools, without spicrawl_browser_connect_url. A session opened with a key lists that tool too, unless the variable is set.

  • Without any key, /mcp answers only discovery: initialize, ping, the initialized notification and tools/list (the same 24-tool list as the server card; no Mcp-Session-Id is issued and nothing is sent upstream). Any other request, a tools/call included, is 401, as is a request that sends a malformed key or names a session. A session-less GET is 405.

  • Each caller sends their own key as Authorization: Bearer spicrawl_live_..., or, for a gateway that cannot send the Bearer prefix, as a raw Authorization: spicrawl_live_..., x-api-key or api-key header (see "Smithery and other gateways" above; never a query parameter). The key is checked against the API before a session opens (401 if refused, 503 with Retry-After if the API cannot be reached), and a session is pinned to the key that opened it.

  • The server speaks plain HTTP. Put it behind TLS before sending keys across a network.

ChatGPT endpoint (OAuth)

The same process can serve a second, restricted MCP endpoint for ChatGPT apps. It is off unless SPICRAWL_CHATGPT_ENABLED is on, and /mcp is unchanged either way.

Route

What it does

POST /chatgpt/mcp

MCP Streamable HTTP, stateless: no Mcp-Session-Id, a fresh server per request, JSON responses. Other methods answer 405.

GET /.well-known/oauth-protected-resource/chatgpt/mcp

Protected-resource metadata (RFC 9728): resource, authorization_servers (the issuer), scopes_supported: ["scrape","batch"], resource_documentation.

GET /.well-known/oauth-protected-resource

The same document, only with SPICRAWL_CHATGPT_ROOT_PRM.

GET /.well-known/openai-apps-challenge

The verification token as text/plain, exact bytes, no trailing newline; 404 when none is configured.

Authentication takes OAuth access tokens (spicrawl_oat_…) only:

  • No token: 401 with WWW-Authenticate: Bearer resource_metadata="https://mcp.spicrawl.com/.well-known/oauth-protected-resource/chatgpt/mcp", scope="scrape batch".

  • A bad, inactive, expired or wrong-audience token, or an API key (spicrawl_live_…/spicrawl_test_…, never accepted here): the same 401 plus error="invalid_token", error_description="…".

  • A tool call the token's scopes do not cover: 403 with error="insufficient_scope". spicrawl_scrape needs scrape, the batch tools need batch, the docs tools need only a valid token.

  • The authorization server unreachable or refusing this server's secret: 503 with Retry-After.

Each token is checked with POST <issuer>/api/oauth/introspect (Authorization: Bearer <SPICRAWL_OAUTH_INTROSPECT_SECRET>, form body token=…). Only active: true with aud equal to the resource is accepted. A positive answer is cached by the token's SHA-256 for at most 30 seconds and never past its exp. Tools run upstream under the api_key the introspection returns; neither it nor the token is logged. If the API refuses that key mid-call, the tool returns an error whose _meta["mcp/www_authenticate"] carries a fresh invalid_token challenge, and the token is dropped from the cache.

The endpoint lists six tools, each with explicit readOnlyHint/destructiveHint/openWorldHint, a title, and securitySchemes: [{type: "oauth2", scopes: [...]}] (also in _meta.securitySchemes):

Tool

Scopes

Arguments

spicrawl_scrape

scrape

url (http/https only), format (markdown/text/html/json), render, main_content_only, include_tags, exclude_tags, links, extract (CSS selectors), autoparse, wait_for, cache, cache_ttl, max_cost (default 5). Always an HTTP GET.

spicrawl_batch_submit

batch

urls (1 to 25), format, render, main_content_only, max_cost, credit_budget (default 50).

spicrawl_batch_status

batch

job_id.

spicrawl_batch_results

batch

job_id, status, limit (default 25, max 100), cursor.

spicrawl_docs_search, spicrawl_docs_read

none

As on /mcp.

spicrawl_scrape is read-only here, unlike on /mcp: this endpoint only sends a GET, with no method, body, headers or actions. spicrawl_batch_submit is not read-only, because it creates a job.

spicrawl_batch_results fills in the content of a succeeded item that the results page lists without it, which happens while the job is still finishing, from GET /v1/batch/{id}/tasks/{seq}/content. It fetches at most 25 items and 2 MiB per page. An item that can't be read yet comes back unavailable, and one past the size cap omitted.

Every other argument is refused, not dropped. Results keep the content, the site's status and the credits charged; request ids, timestamps, engine and proxy details, storage references and project ids are left out.

Installed globally (npm install -g @spicrawl/mcp), the command is spicrawl-mcp (alias spicrawl-mcp-server), with --http, --version and --help.

FAQ

Which MCP server scrapes JavaScript sites?

The Spicrawl MCP server does. Call spicrawl_scrape with render: true (3 credits) and the page loads in a browser, runs its scripts and comes back as finished Markdown, text, HTML or JSON. Add wait_for with a CSS selector for content that appears late, or set mode: "auto" to start with a plain fetch and move up to a stronger rendering only when needed. Spicrawl picks the rendering for each request and bills the tier that served it (fetch, js or browser, reported as engine in the JSON result); there is no argument to choose it.

How do I give Claude web access?

Connect the Spicrawl MCP server to Claude with an API key. In Claude Code:

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Claude can then read any URL you give it by calling spicrawl_scrape. Spicrawl reads pages by URL; it is not a search engine. For Claude Desktop, see the install answer below.

Is there a free tier?

Yes. During the beta every organization is on one plan of 1,000 credits per month. A plain fetch costs 1 credit, JavaScript rendering 3, and full-browser rendering (screenshots, PDFs, actions) 8. Failed requests cost 0, and a cache hit is billed at the same price as the fetch that stored it. When the month's credits run out, calls fail with ERR::LIMIT::QUOTA_EXCEEDED until it resets. See Credits.

Does it handle bot-protected pages?

Only with your own proxy. Pass it in proxy (an http:// URL; residential proxies work best). When a site answers with a Cloudflare managed challenge, Spicrawl detects it, moves the request to full-browser rendering on the same proxy exit and returns the real page, for 8 credits. If every step is challenged you get ERR::UPSTREAM::CHALLENGE and pay 0. Success is likely, not guaranteed, and Spicrawl does not solve CAPTCHAs. On the default shared pool a challenged page returns that error. See Scrape Cloudflare-protected pages.

How do I install it in Cursor or Claude Desktop?

Cursor: click Add to Cursor above, or add this to .cursor/mcp.json and start Cursor with SPICRAWL_API_KEY set:

{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${env:SPICRAWL_API_KEY}" }
    }
  }
}

Claude Desktop starts local (stdio) servers only, so run the package. Add this to claude_desktop_config.json and restart the app:

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": { "SPICRAWL_API_KEY": "spicrawl_live_..." }
    }
  }
}

Get a key at app.spicrawl.com. Steps for Claude Code, VS Code, Windsurf, Gemini CLI and Codex are under Set up your MCP client.

Can it crawl a whole website?

There is no single crawl tool. Scrape a page with links: true to get its links, then feed them to an open batch job (open: true and spicrawl_batch_add_items).

Other questions

Is there a hosted Spicrawl MCP server?

Yes: https://mcp.spicrawl.com/mcp, MCP Streamable HTTP, authenticated with Authorization: Bearer <API key>, so there is nothing to install.

Does it support OAuth?

Not on /mcp: it accepts Spicrawl API keys only, so turn OAuth off for it in clients that try it on a 401. The separate ChatGPT endpoint at /chatgpt/mcp takes OAuth access tokens only, when the operator turns it on.

Should I use the MCP server, the SDK or the CLI?

Use the MCP server when an AI agent should call Spicrawl as tools. Use @spicrawl/sdk in your own TypeScript or JavaScript code, @spicrawl/cli in a terminal or shell script, and the REST API from any other language.

Troubleshooting

Symptom

Fix

HTTP 401 or "Unauthorized: send your Spicrawl API key"

The header is missing, or the key is malformed, unknown, revoked or expired. Check that the variable is set in the environment the client started from. A gateway that cannot send Bearer can use an x-api-key header; a key in the URL is never read.

The header contains a literal ${SPICRAWL_API_KEY}

The client did not expand it. Use ${VAR} in Claude Code, ${env:VAR} in Cursor, ${input:id} in VS Code, bearer_token_env_var in Codex.

HTTP 503 with Retry-After on connect

The server could not reach the API to check your key. Retry; your key is fine.

ERR::AUTH::INSUFFICIENT_SCOPE from a spicrawl_usage* tool

Grant the key the read scope in the dashboard.

ERR::LIMIT::QUOTA_EXCEEDED (HTTP 402)

The monthly credit allowance is used up. Not retryable until it resets.

"Unknown argument" on spicrawl_scrape

Use the tool's names: render, not js_render; format, not response_format.

Tools do not appear

Restart the client, and check the config key (servers in VS Code, mcpServers elsewhere).

npx not found (stdio)

Install Node.js 20 or later.

curl -i https://mcp.spicrawl.com/mcp without a key answers 401: the endpoint is reachable.

Publishing to the MCP Registry

server.json lists both the npm/stdio package and the hosted Streamable HTTP endpoint under io.github.OfficialSpicrawl/mcp. API keys are requested from the person installing the server; no key belongs in the manifest.

The existing release workflow publishes to the MCP Registry after its npm job succeeds, on pushes to main or manual runs from main. Registry authentication uses GitHub OIDC (id-token: write), so it needs no new secret, device login or DNS record. The npm release keeps using the existing NPM_TOKEN secret.

For a release, run npm version --no-git-tag-version X.Y.Z, update the changelog, and submit the changes to main. The npm version hook synchronizes both versions in server.json. If you edit package.json by hand, run npm run registry:sync. CI checks the manifest against its official $schema with Ajv and rejects name or version drift before npm publication. To run that check locally:

npm ci
npm run registry:validate
# Offline: pass a downloaded copy of the exact server.json $schema URL.
npm run registry:validate -- /path/to/server.schema.json

The registry job checks that the published npm package has the matching mcpName before publishing. An existing registry version is skipped; lookup failures stop the job. To retry a registry failure or list the already-published current npm version, run the release workflow on main again. Use a new version for metadata changes to an existing listing. The publisher binary is pinned and its download is checked against the release checksums; update its version in release.yml when upgrading the CLI.

Verify publication at https://registry.modelcontextprotocol.io/v0.1/servers?search=io.github.OfficialSpicrawl/mcp. The workflow must be merged into OfficialSpicrawl/mcp before OIDC can publish that namespace. Fork pull requests validate and test, but never run the registry job.

Official references: publishing quickstart, GitHub Actions, and remote servers.

License

Apache-2.0

Available Tools

25 tools
spicrawl_batch_add_itemsAdd URLs to an open batch jobAInspect

Append URLs to a batch job that was submitted with open: true. Same item vocabulary as spicrawl_batch_submit (EITHER urls or items, not both, plus shared settings for these new items). Fails with CONFLICT if the job is closed or finished. Not idempotent: calling twice adds the URLs twice. Returns { job, items_added, items_dispatched }.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsNoShorthand: URLs scraped with the shared settings and no per-item overrides. Mutually exclusive with `items`.
waitNoMilliseconds to wait after load (rendered scrapes).
itemsNoLong form: one /v1/scrape request object per item. Besides `url` and `external_id`, any /v1/scrape field (e.g. `js_render`, `wait_for`, `response_format`) overrides the shared setting for that item only. Mutually exclusive with `urls`.
formatNoOutput format for every item (`response_format`). spicrawl_batch_submit defaults to 'markdown', the same as spicrawl_scrape; an item's own `response_format` overrides it.
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.
renderNoRender every URL with a browser (`js_render`).
max_costNoCeiling in credits for ONE item (not the whole job — see credit_budget).
wait_forNoCSS selector to wait for before capturing (rendered scrapes).
premium_proxyNoComing soon: not available yet, do not send. Use residential managed-pool exits for every URL.
proxy_countryNoComing soon: not available yet, do not send. ISO-3166 alpha-2 exit country; requires premium_proxy.
custom_headersNoExtra request headers sent with every URL.
block_resourcesNoResource types to block while rendering, e.g. ["image","font","media"].
main_content_onlyNoStrip each page to its main article.

Output Schema

ParametersJSON Schema
NameRequiredDescription
jobNoThe job as it stands after the call.
items_addedNoItems written to the job.
items_dispatchedNoOf those, how many reached the dispatch queue now; do not re-add the rest.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes beyond annotations by disclosing the non-idempotency ('calling twice adds the URLs twice'), the CONFLICT failure for closed/finished jobs, and the exact return shape. Annotations already cover idempotentHint=false and openWorldHint, so credit here is for the conflict semantics and return fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four dense sentences with zero filler: purpose, vocabulary constraint, failure condition, idempotency warning, and return shape. The critical non-idempotency warning is front-loaded enough to be caught.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the operation's key gotchas (mutual exclusivity, CONFLICT on closed jobs, non-idempotency) despite an output schema existing. An agent has everything needed to invoke correctly and interpret the response.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema carries the parameter burden. The description adds value by flagging the mutual exclusivity of `urls`/`items` and that shared settings apply to new items, but this largely restates what the schema already documents.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (append) and resource (URLs to a batch job), and scopes it precisely to jobs submitted with `open: true`, which distinguishes it from spicrawl_batch_submit and other batch siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly names the sibling it shares item vocabulary with (spicrawl_batch_submit) and states the precondition (job must be open) plus failure mode (CONFLICT). Lacks explicit 'when to use this vs spicrawl_batch_retry' guidance, but the open-job scoping is a clear routing signal.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_cancelCancel a batch jobA
DestructiveIdempotent
Inspect

Cancel a batch job: items not yet started are cancelled and will never run; items already finished keep their results; items already running finish first, so the job comes back cancelling and moves to cancelled when they drain. Cannot be undone — resubmit to run the URLs again. Returns the job object.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesJob id. Pass it as `job_id` to the other spicrawl_batch_* tools.
nameNoThe label given at submission, if any.
openNoTrue while the job accepts appended items (it will not complete until closed).
paramsNoThe job-level scrape settings as stored.
statusNoLifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.
priorityNoEffective scheduling priority.
progressNoLive item counters and the credits charged so far.
warningsNoDecisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued).
error_codeNoMachine code of a job-level failure.
project_idNoProject the API key belongs to.
started_atNoISO-8601 time the first item started.
status_urlNoAPI path to poll for this job: /v1/batch/{id}.
concurrencyNoEffective per-job concurrency after the server's clamping.
finished_atNoISO-8601 time the job reached a terminal status.
results_urlNoAPI path of the job's results: /v1/batch/{id}/results.
total_itemsNoNumber of items in the job.
max_attemptsNoEffective attempts per item.
submitted_atNoISO-8601 time the job was accepted.
error_messageNoHuman-readable job-level failure reason.
estimated_creditsNoCredits reserved at submission (a hold, not a charge; compare with progress.credits_charged).
failure_thresholdNoFailed-item count that aborts the job, when set.
results_expire_atNoISO-8601 deadline after which results answer 410 Gone.
cancel_requested_atNoISO-8601 time a cancel was accepted.
estimated_credits_microNoSame as estimated_credits in micro-credits.

TDQS

A4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations (destructiveHint, idempotentHint, readOnlyHint=false) by specifying exactly what is destroyed (items not yet started), what is preserved (finished items keep results), and how in-flight items are handled. It also discloses irreversibility, the 'cancelling' -> 'cancelled' state transition, and the recovery path (resubmit), which annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and packed with high-value behavioral detail in a compact form; no filler sentences. The trailing 'Returns the job object' is mildly redundant given an output schema exists, a minor inefficiency rather than a flaw.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter mutation tool with full annotation coverage and an output schema, the description supplies everything else an agent needs: scope of destruction, state-machine behavior, irreversibility, and remediation. No meaningful gap remains.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter, job_id, and the schema documents it fully (100% coverage, including its provenance from spicrawl_batch_submit/spicrawl_batch_list). The description adds no format, constraint, or edge-case detail about the id, so baseline 3 is appropriate when the schema already carries the parameter contract.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Cancel a batch job') and immediately scopes the semantics to per-item lifecycle behavior, so an agent knows exactly what operation this is. It never names or contrasts the adjacent spicrawl_batch_close sibling, which is the one plausible confusion, so it falls short of full sibling differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: the description communicates that this stops a job and that running items must drain, which tells an agent the appropriate operational moment. However there is no explicit when-to-use/when-not, no prerequisites, and no mention of when spicrawl_batch_close or spicrawl_batch_retry would be preferred instead.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_closeClose an open batch jobA
Idempotent
Inspect

Stop an open batch job accepting items, so it completes once its queued work drains. Idempotent (closing a closed job succeeds). Returns the job object.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesJob id. Pass it as `job_id` to the other spicrawl_batch_* tools.
nameNoThe label given at submission, if any.
openNoTrue while the job accepts appended items (it will not complete until closed).
paramsNoThe job-level scrape settings as stored.
statusNoLifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.
priorityNoEffective scheduling priority.
progressNoLive item counters and the credits charged so far.
warningsNoDecisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued).
error_codeNoMachine code of a job-level failure.
project_idNoProject the API key belongs to.
started_atNoISO-8601 time the first item started.
status_urlNoAPI path to poll for this job: /v1/batch/{id}.
concurrencyNoEffective per-job concurrency after the server's clamping.
finished_atNoISO-8601 time the job reached a terminal status.
results_urlNoAPI path of the job's results: /v1/batch/{id}/results.
total_itemsNoNumber of items in the job.
max_attemptsNoEffective attempts per item.
submitted_atNoISO-8601 time the job was accepted.
error_messageNoHuman-readable job-level failure reason.
estimated_creditsNoCredits reserved at submission (a hold, not a charge; compare with progress.credits_charged).
failure_thresholdNoFailed-item count that aborts the job, when set.
results_expire_atNoISO-8601 deadline after which results answer 410 Gone.
cancel_requested_atNoISO-8601 time a cancel was accepted.
estimated_credits_microNoSame as estimated_credits in micro-credits.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare idempotentHint=true and destructiveHint=false, and the description reinforces idempotency with a concrete case ('closing a closed job succeeds'). It also adds the non-obvious asynchronous semantic that the job does not stop immediately but drains its queue, which annotations do not convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, each carrying distinct information: the action, the idempotency guarantee, and the return value. Nothing is padded or restated.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter mutation with full annotation coverage and an output schema, the description covers behavior, idempotency, and drain semantics adequately. The only gap is not routing the agent between this tool and spicrawl_batch_cancel.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single job_id parameter is documented with its origin ('returned by spicrawl_batch_submit or spicrawl_batch_list'). The description adds no further parameter meaning, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a precise verb and resource ('Stop an open batch job accepting items') plus the post-condition ('completes once its queued work drains'). This separates it from the sibling spicrawl_batch_cancel, which an agent would otherwise confuse for immediate termination.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The graceful-drain framing implies the when ('you want the job to finish its queued work rather than abort it'), but no alternative is named outright. The sibling spicrawl_batch_cancel is never mentioned, so the choice between graceful close and hard cancel is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_listList batch jobsA
Read-onlyIdempotent
Inspect

List this API key's project's batch jobs, newest first, with status and progress. Returns { batches: [...], next_cursor? }; pass next_cursor back as cursor for the next page.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoPage size (default 50, max 200).
cursorNo`next_cursor` from a previous page, verbatim.
statusNoOnly jobs in this status.

Output Schema

ParametersJSON Schema
NameRequiredDescription
batchesYesJobs, newest first (empty when there are none).
next_cursorNoPass back as `cursor` for the next page; absent on the last page.

TDQS

A3.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the safe-read profile (readOnly, idempotent, non-destructive), so the bar is lower. The description adds real context beyond them: project scoping, newest-first ordering, and the cursor-based pagination contract, which an agent needs to page correctly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with purpose and scope, then the return/pagination contract. No filler; every clause earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so explaining return values is optional, yet the description still sketches the shape and pagination loop usefully. It is complete for calling the tool; the only gap is that sibling boundaries (vs batch_status) are not drawn.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all three parameters are already documented, including the status enum. The description reinforces the cursor linkage but adds no syntax or meaning beyond what the schema states, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('List ... batch jobs') with clear scope (this API key's project) and ordering (newest first). However, it does not differentiate itself from the sibling spicrawl_batch_status, which likely also surfaces job status, so an agent may hesitate between the two.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit when-to-use versus alternatives is given. The pagination instruction ('pass next_cursor back as cursor') is procedural help, not guidance on selecting this tool over batch_status or batch_results.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_resultsRead a batch job's resultsA
Read-onlyIdempotent
Inspect

Return a page of FINISHED items of a batch job (one record per item: seq, url, external_id, status, error, and content, or a result_url API path (GET /v1/batch/{id}/tasks/{seq}/content, same key) for large bodies). Unfinished items are not listed yet. Returns { results: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Results expire after the job's retention window (410 Gone).

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoPage size (default 500, max 5000).
cursorNo`next_cursor` from a previous page, verbatim.
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.
statusNoOnly items that finished with this status.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultsYesFinished items of the job, in seq order.
next_cursorNoPass back as `cursor` for the next page; absent on the last page.

TDQS

A4.4/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover the read-only/idempotent safety profile; the description goes well beyond them by disclosing that unfinished items are not yet listed, the exact paging contract, the expiry-and-410 behavior, and the large-body result_url fallback. This is exactly the behavioral context annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose leads the sentence, followed by pagination and retention details in priority order. Dense but efficient, though the first sentence packs several concepts into a nested parenthetical that could be split for faster scanning.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists, yet the description additionally explains the page shape and cursor handoff, plus what happens on expiry and where large bodies live. For a paginated read tool with three inputs, nothing an agent needs in order to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the description's parameter-level content (cursor round-trip, page size, status filtering via the job's items) is already stated in the schema. It restates the paging loop without adding new syntax or constraints, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Return) and resource (a page of FINISHED items of a batch job), and the scope word 'FINISHED' immediately distinguishes it from job-level status siblings and from per-task content fetches. An agent can identify the tool's role without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear operational context: results only appear once items finish, pages are walked by passing next_cursor back as cursor, and results expire (410 Gone). It also points at the result_url path for large bodies rather than the sibling spicrawl_batch_task_content. What's missing is an explicit when-to-use comparison against spicrawl_batch_status or spicrawl_batch_list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_retryRetry a batch job's failed itemsAInspect

Re-queue every FAILED item of a batch job so it runs again (succeeded items are untouched; a job with no failures is returned unchanged). Returns { job, items_reset, items_dispatched }.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
jobNoThe job as it stands after the call.
items_resetNoFailed items reset to queued (0 means nothing had failed).
items_dispatchedNoOf those, how many reached the dispatch queue now; the rest are recovered automatically.

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare the mutation profile (readOnlyHint=false, destructiveHint=false, idempotentHint=false), and the description adds real value on top: only failed items are re-queued, succeeded items are preserved, and a no-failure job is a no-op. It doesn't discuss re-dispatch semantics for a non-idempotent operation (e.g., duplicate side effects), which would have fully earned a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the action and its scope, followed by the return shape. Every clause carries information and none is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the description needn't detail return values, yet it still names the return fields. The mutation behavior is well covered; only deeper operational detail (auth/session requirements, effect on a currently running job) is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single job_id parameter is already documented in the schema as coming from spicrawl_batch_submit or spicrawl_batch_list. The description adds nothing about the parameter beyond what the schema states, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Re-queue'), resource ('FAILED item of a batch job'), and scope ('every FAILED item', 'succeeded items untouched'). An agent can distinguish this from spicrawl_batch_submit, spicrawl_batch_cancel, and spicrawl_batch_results without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear selection condition implicitly by scoping the operation to jobs with failures and noting a job with no failures returns unchanged. It doesn't explicitly name alternative tools (e.g., resubmit vs cancel) for when retry isn't the right move, so it stops short of full when/when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_statusCheck a batch job's statusA
Read-onlyIdempotent
Inspect

Return one batch job with its status and progress (total/completed/succeeded/failed/remaining, credits charged). Poll this after spicrawl_batch_submit; once status is completed, failed or cancelled, read spicrawl_batch_results.

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesJob id. Pass it as `job_id` to the other spicrawl_batch_* tools.
nameNoThe label given at submission, if any.
openNoTrue while the job accepts appended items (it will not complete until closed).
paramsNoThe job-level scrape settings as stored.
statusNoLifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.
priorityNoEffective scheduling priority.
progressNoLive item counters and the credits charged so far.
warningsNoDecisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued).
error_codeNoMachine code of a job-level failure.
project_idNoProject the API key belongs to.
started_atNoISO-8601 time the first item started.
status_urlNoAPI path to poll for this job: /v1/batch/{id}.
concurrencyNoEffective per-job concurrency after the server's clamping.
finished_atNoISO-8601 time the job reached a terminal status.
results_urlNoAPI path of the job's results: /v1/batch/{id}/results.
total_itemsNoNumber of items in the job.
max_attemptsNoEffective attempts per item.
submitted_atNoISO-8601 time the job was accepted.
error_messageNoHuman-readable job-level failure reason.
estimated_creditsNoCredits reserved at submission (a hold, not a charge; compare with progress.credits_charged).
failure_thresholdNoFailed-item count that aborts the job, when set.
results_expire_atNoISO-8601 deadline after which results answer 410 Gone.
cancel_requested_atNoISO-8601 time a cancel was accepted.
estimated_credits_microNoSame as estimated_credits in micro-credits.

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, so safety is covered; the description adds genuine behavioral context by enumerating the progress counters and credit charging, and by defining which statuses are terminal. It stops short of stating polling cadence, rate limits, or behavior for an unknown/expired job_id.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler. The return shape is front-loaded and the lifecycle guidance follows immediately, so the agent gets the essential contract in the first clause.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description need not explain the payload, and the polling workflow is fully covered. The only unaddressed edge is failure handling for an invalid or expired job_id, which is minor given the read-only, idempotent annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single job_id parameter is fully documented in the schema, including its provenance. The description adds nothing about the parameter itself, so the baseline 3 for schema-carried semantics applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Return one batch job with its status and progress') and pins the scope to a single job, which cleanly separates it from the list-shaped sibling spicrawl_batch_list. An agent knows exactly what this returns without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit sequencing: poll this after spicrawl_batch_submit, and once status is completed/failed/cancelled, switch to spicrawl_batch_results. Both the when-to-use and the alternative are named with the condition that selects them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_submitSubmit a batch scrape jobAInspect

Submit many URLs to be scraped as one asynchronous job, with shared settings applied to every URL. Give EITHER urls or items, not both (at least one URL, unless open is true). Returns the job object (id, status, estimated_credits, progress). Poll with spicrawl_batch_status, read with spicrawl_batch_results or spicrawl_batch_task_content. Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs. Set open: true to keep the job accepting more URLs via spicrawl_batch_add_items; an open job never finishes until you call spicrawl_batch_close.

ParametersJSON Schema
NameRequiredDescriptionDefault
nameNoA human label for the job.
openNoKeep the job accepting items after submission (append with spicrawl_batch_add_items, finish with spicrawl_batch_close).
urlsNoShorthand: URLs scraped with the shared settings and no per-item overrides. Mutually exclusive with `items`.
waitNoMilliseconds to wait after load (rendered scrapes).
itemsNoLong form: one /v1/scrape request object per item. Besides `url` and `external_id`, any /v1/scrape field (e.g. `js_render`, `wait_for`, `response_format`) overrides the shared setting for that item only. Mutually exclusive with `urls`.
formatNoOutput format for every item (`response_format`). spicrawl_batch_submit defaults to 'markdown', the same as spicrawl_scrape; an item's own `response_format` overrides it.
renderNoRender every URL with a browser (`js_render`).
max_costNoCeiling in credits for ONE item (not the whole job — see credit_budget).
priorityNoScheduling priority of the job relative to your other jobs.
wait_forNoCSS selector to wait for before capturing (rendered scrapes).
concurrencyNoMax items in flight at once for this job (may be capped by the server; see warnings).
max_attemptsNoPer-item retry attempts on failure.
credit_budgetNoCeiling on the WHOLE job, in credits; the run aborts when it would exceed this.
premium_proxyNoComing soon: not available yet, do not send. Use residential managed-pool exits for every URL.
proxy_countryNoComing soon: not available yet, do not send. ISO-3166 alpha-2 exit country; requires premium_proxy.
custom_headersNoExtra request headers sent with every URL.
block_resourcesNoResource types to block while rendering, e.g. ["image","font","media"].
failure_thresholdNoAbort the job once this many items have failed.
main_content_onlyNoStrip each page to its main article.
webhook_endpoint_idNoId of a registered webhook endpoint notified (with the job object) when the job completes.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesJob id. Pass it as `job_id` to the other spicrawl_batch_* tools.
nameNoThe label given at submission, if any.
openNoTrue while the job accepts appended items (it will not complete until closed).
paramsNoThe job-level scrape settings as stored.
statusNoLifecycle: queued, running, paused, cancelling, completed, failed or cancelled. Keep polling while queued/running/paused/cancelling; stop at completed, failed or cancelled.
priorityNoEffective scheduling priority.
progressNoLive item counters and the credits charged so far.
warningsNoDecisions made on the caller's behalf at submission (a clamped concurrency, items not yet queued).
error_codeNoMachine code of a job-level failure.
project_idNoProject the API key belongs to.
started_atNoISO-8601 time the first item started.
status_urlNoAPI path to poll for this job: /v1/batch/{id}.
concurrencyNoEffective per-job concurrency after the server's clamping.
finished_atNoISO-8601 time the job reached a terminal status.
results_urlNoAPI path of the job's results: /v1/batch/{id}/results.
total_itemsNoNumber of items in the job.
max_attemptsNoEffective attempts per item.
submitted_atNoISO-8601 time the job was accepted.
error_messageNoHuman-readable job-level failure reason.
estimated_creditsNoCredits reserved at submission (a hold, not a charge; compare with progress.credits_charged).
failure_thresholdNoFailed-item count that aborts the job, when set.
results_expire_atNoISO-8601 deadline after which results answer 410 Gone.
cancel_requested_atNoISO-8601 time a cancel was accepted.
estimated_credits_microNoSame as estimated_credits in micro-credits.

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare non-read-only, non-idempotent, non-destructive behavior. The description adds genuine context beyond that: the job is asynchronous, it returns a job object, and an open job 'never finishes until you call spicrawl_batch_close' — a lifecycle trap the annotations do not convey. It stops short of stating credit/auth side effects or that concurrent submission of the same URLs will duplicate work.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded in the first clause, followed by constraints, return shape, routing, and the alternative-tool guidance. Six sentences, each carrying distinct information, with no restatement of the name or title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 20-parameter, nested-object, async submission tool with a full output schema and 100% schema coverage, the description supplies exactly the missing layer: mutual exclusivity, open-job lifecycle, and the sibling tools to poll/read/extend with. No essential calling information is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds real meaning beyond the schema by stating the mutual exclusion of `urls` and `items` ('not both') and the minimum-content rule ('at least one URL, unless `open` is true'), which the schema does not express as a constraint.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource (submit many URLs as one asynchronous job) and immediately scopes it with 'shared settings applied to every URL'. It explicitly distinguishes itself from the scrape sibling and from batch_add_items/batch_close, so an agent can identify it without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States the selection condition outright: 'Use this instead of many spicrawl_scrape calls when you have tens or thousands of URLs.' It also names the downstream lifecycle tools (batch_status to poll, batch_results/batch_task_content to read, batch_add_items/batch_close for open jobs) and the exclusive condition for urls vs items.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_batch_task_contentRead one batch item's contentA
Read-onlyIdempotent
Inspect

Return the full payload one item of a batch job produced, by its seq (items are numbered from 0 in submission order). Cheaper than paging results when you need one URL's output. A CONFLICT error means either the item has not finished yet (poll spicrawl_batch_status and ask again) or it finished as failed (it will never produce content; the message says which). NOT_FOUND means no such seq.

ParametersJSON Schema
NameRequiredDescriptionDefault
seqYesThe item's sequence number (0-based, submission order).
job_idYesThe batch job id returned by spicrawl_batch_submit or spicrawl_batch_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
contentNoThe item's document when it is text (markdown, text or html) or any non-object JSON value. An item whose payload is a JSON object is returned as that object's own fields instead.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description adds a genuinely new layer: the error contract. It explains that CONFLICT is ambiguous between 'not finished yet' and 'finished as failed, will never produce content', and that the message disambiguates — this is behavior an agent cannot infer from the schema or annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three dense sentences, front-loaded with the primary purpose, then the cost/routing rationale, then the failure semantics. No filler, and each sentence carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need not be described, and the description instead covers the two non-obvious gaps — cost trade-off versus paging and the ambiguous CONFLICT failure mode. Nothing an agent needs to call and interpret this tool correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both `seq` and `job_id` are already documented in the schema, including the 0-based submission-order semantics that the description restates. The description adds no syntax or format detail beyond the structured fields, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (return), resource (one batch item's full payload), and the addressing key (`seq`), and explicitly contrasts itself with the sibling that pages results. An agent can identify the tool and its scope without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit selection rule ('Cheaper than paging results when you need one URL's output') that routes against spicrawl_batch_results, and additionally routes on failure states by telling the agent to poll spicrawl_batch_status on CONFLICT. When-to-use and the recovery alternative are both covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_browser_connect_urlBrowser (CDP) connect URLAInspect

Coming soon: remote browsers are not available yet, so minting fails; do not rely on this tool. For scripted page interaction today use actions on spicrawl_scrape. Mints a single-use connect URL for Spicrawl's remote browser (GET /v1/browser, a DevTools Protocol (CDP) WebSocket) so YOUR code can drive it with any CDP client, using the URL as the browser WebSocket endpoint. It calls POST /v1/browser/token and returns a ws:// or wss:// URL carrying a short-lived token, never the API key: the URL works ONCE, within 60 seconds, and only with the options given here (editing them in the URL is refused), so call this right before connecting and again for every new session. Prefer sending the key as an Authorization header when the client supports headers on the WebSocket upgrade: then connect to the plain /v1/browser URL with 'Authorization: Bearer ' and no token is needed. The endpoint exists only when the deployment was installed with --with-cdp, and the key must carry the browser scope; otherwise minting fails (404 or 403). Parameter errors are reported when minting.

ParametersJSON Schema
NameRequiredDescriptionDefault
proxyNoYour own proxy, as http://user:pass@host:port. Replaces our pool, so proxy_country, proxy_region and sticky_key cannot be combined with it. Kept out of the returned URL.
engineNoRendering tier of the remote browser: `browser` (full browser, full protocol; the default) or `js` (lighter JavaScript rendering with a smaller protocol surface). A tier your plan does not include is refused.
sticky_keyNoKeep one exit IP for the session under this key (2-64 letters, digits, hyphens). Omit to rotate (the default).
session_ttlNoSession lifetime in seconds, 60-900. Default 180.
proxy_regionNoBroader exit region/pool name; 'global' means no constraint. Mutually exclusive with proxy_country.
proxy_countryNoExit country, ISO-3166 alpha-2 lowercase (e.g. 'de'). Mutually exclusive with proxy_region.

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYesws:// or wss:// connect URL carrying a single-use token (never the API key); use it as the CDP browser WebSocket endpoint.
notesNoUsage notes for the connect URL.
expires_atNoWhen the token expires, RFC 3339 UTC (60 s after minting).
single_useYesAlways true: the URL works for one connection.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite annotations covering safety hints, the description adds substantial behavior beyond them: single-use URL, 60-second window, options locked into the token (URL edits refused), token never exposes the API key, endpoint requires --with-cdp, key needs `browser` scope, and 404/403 failure modes. This is unusually rich disclosure for a token-minting tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the critical 'coming soon, do not rely' warning, which is the most important thing an agent can read. The body is dense and every clause carries operational detail, though it is long and could be tightened slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description needn't explain return values, and it still covers auth, scope requirements, deployment prerequisite, token lifetime, single-use constraint, and failure codes. Nothing an agent needs to call or avoid this tool correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all six parameters including enums, patterns, ranges, and mutual exclusions. The description only echoes this at a high level ('only with the options given here') without adding syntax or semantics beyond the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Mints a single-use connect URL for Spicrawl's remote browser'), names the underlying endpoint, and distinguishes itself from the sibling alternative ('use `actions` on spicrawl_scrape'). An agent immediately knows what it does and that it is currently non-functional.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when NOT to use it ('Coming soon... do not rely on this tool') and routes to the working alternative for scripted page interaction today. It also gives the timing rule (call right before connecting, again per session) and the Authorization-header variant.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_docs_indexList the Spicrawl docsA
Read-onlyIdempotent
Inspect

The docs' llms.txt: every Spicrawl documentation page with its title, a one-line description and its Markdown URL. Use it to find the right page when a search does not, or to get an overview of what is documented; read a page with spicrawl_docs_read. No parameters.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYesWhere the document lives on the docs site.
contentYesThe index (llms.txt): one line per page with its title, description and Markdown URL.
truncatedYesTrue when the content was cut at the size cap; the content says so at its end.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered. The description adds useful context beyond them: the content is a static llms.txt index and what each entry carries, which tells the agent this is a cheap catalog lookup rather than a live query.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the resource and its contents, followed by when-to-use and the read follow-up. No filler and nothing repeated from the schema or annotations.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A zero-parameter catalog tool with an output schema, so return-value details need not be spelled out; the description still summarizes the entry shape. Nothing needed to select or invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Zero parameters, so the schema carries nothing to misinterpret and the baseline is 4. The description explicitly confirms 'No parameters', which matches the empty schema and removes any doubt about required input.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific resource (the Spicrawl docs llms.txt index) and exactly what it contains: every page with title, one-line description and Markdown URL. It names spicrawl_docs_read as the follow-up and implies spicrawl_docs_search, so an agent can distinguish it from its siblings without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit conditions: use it to find the right page when a search fails, or to get an overview. It names spicrawl_docs_read as the alternative for actually reading a page, though the search tool it complements (spicrawl_docs_search) is referenced only as 'a search' rather than by name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_docs_readRead a Spicrawl docs pageA
Read-onlyIdempotent
Inspect

Fetches one Spicrawl documentation page as Markdown. path may be a page path (guides/anti-bot), a docs URL from spicrawl_docs_search (url or md_url), or either with a #anchor (the whole page is returned; the anchor names the section to look at). Only pages on the Spicrawl docs (https://docs.spicrawl.com) can be read. Pages over 60000 characters are truncated, and the text says so. Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page.

ParametersJSON Schema
NameRequiredDescriptionDefault
pathYesPage path (`guides/anti-bot`, `errors#AUTH_INVALID_KEY`) or a full docs URL, as returned by spicrawl_docs_search.

Output Schema

ParametersJSON Schema
NameRequiredDescription
urlYesWhere the document lives on the docs site.
titleNoThe page's first heading, when it has one.
md_urlNoThe Markdown source it was read from.
contentYesThe document as Markdown.
sectionNoThe #anchor that was asked for, without the #. The whole page is returned.
truncatedYesTrue when the content was cut at the size cap; the content says so at its end.

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, open-world, so the safety profile is covered. The description still adds real operational detail beyond them: the 60,000-character truncation with an in-text notice, the domain restriction, and the anchor behavior (whole page returned, anchor merely names the section). It does not mention whether authentication is required, which is a minor remaining gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the core action, then accepted path forms, then limits and fallbacks. No sentence is redundant with the title or schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter read tool with an output schema, everything an agent needs is present: accepted input forms, domain restriction, truncation behavior, and escape hatches to sibling lookup tools. Return-value detail is correctly omitted since an output schema exists.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description exceeds it by explaining that `path` accepts three distinct forms (a bare page path, a url/md_url returned by spicrawl_docs_search, or either with a `#anchor`) and by clarifying the anchor's semantics. That is meaningfully more than the schema's one-line description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Fetches one Spicrawl documentation page as Markdown" gives a precise verb, resource, and output format. It is immediately distinguishable from spicrawl_docs_search (find pages) and spicrawl_docs_index (list all pages), so an agent can pick it without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly routes the agent: "Don't know the path? Use spicrawl_docs_search, or spicrawl_docs_index (llms.txt) for the list of every page." It also states the boundary condition "Only pages on the Spicrawl docs (https://docs.spicrawl.com) can be read," so both when-to-use and when-not-to-use are covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_request_getGet one requestA
Read-onlyIdempotent
Inspect

The full log record of one past API call by its request id. Every response carries its id in the X-Request-Id header (and error bodies carry it too), so this is the tool for debugging a failed or slow scrape: it shows status, error_code/error_detail, attempts, the engine and proxy (tier, country, provider, sticky) actually used, whether and how the target blocked it (vendor/signal/rule), per-stage latency spans and credits charged. Returns not_found both for unknown ids and for ones older than your retention window, and for another project's request unless your key has the read scope.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesThe request id (a ULID, from X-Request-Id or a spicrawl_requests_list row's `id`).

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesThe request id (X-Request-Id).
urlNoThe target URL as requested.
proxyNoWhat proxy was asked for versus what served. Never an exit IP or credentials.
spansNoLatency in milliseconds per stage; null means not measured. Stages do not sum to total_ms.
engineNoThe rendering tier that served it: fetch, js or browser. Absent when refused before a tier was chosen.
statusNoTerminal outcome: success, failed, timeout, blocked, rejected or cancelled. Only success is charged.
blockedNoPresent only when a bot challenge was detected.
attemptsNoDispatch attempts including retries.
bytes_inNo
featuresNoFeatures the request used; free-form.
url_hostNo
bytes_outNo
created_atNoWhen the request arrived, RFC 3339 UTC.
error_codeNoThe problem code the caller got, e.g. ERR::UPSTREAM::CHALLENGE. Absent on success.
project_idNo
http_statusNoThe target's HTTP status; absent when no response was received.
proxy_bytesNo
error_detailNo
credits_microNoCredits charged in micro-credits (1 credit = 1000000); 0 unless success.

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the readOnly/idempotent annotations by disclosing the auth requirement (needs `read` scope for other projects' requests) and precise failure semantics — not_found is returned both for unknown ids and for ids past the retention window. It also enumerates the content actually returned (status, error_code/detail, attempts, engine/proxy, block rules, latency spans, credits).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then the provenance of the id, the debugging use case, the record contents, and the failure conditions — every clause carries information, and the multi-item lists are compact rather than padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema already present, return shape needn't be explained, yet the description still completes the picture with auth prerequisites, retention-window behavior, and cross-project access rules. Nothing an agent needs to call and interpret this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single `id` parameter is already documented as a ULID from X-Request-Id or a list row. The description repeats rather than extends this (mentioning the X-Request-Id header), so it adds no meaning beyond the schema — baseline 3 for a fully-covered single param.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — 'The full log record of one past API call by its request id' — and the singular 'one ... by its request id' clearly contrasts with the plural spicrawl_requests_list sibling. An agent can distinguish this from the list tool and from session/batch tools without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly names the use case: 'this is the tool for debugging a failed or slow scrape.' It gives clear context for when to reach for it, but does not name an alternative or state an explicit when-not (e.g. preferring requests_list for scanning many).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_requests_listList recent requestsA
Read-onlyIdempotent
Inspect

Recent API calls in your key's project, newest first, each with project_id, status, engine, http_status, error_code/error_detail, attempts, URL, proxy used, latency spans (queue/engine/upstream/total ms), block detection and credits. Use it to find failing calls (only_errors=true) or to spot patterns across a target. all_projects=true lists every project in your organization instead (needs a key with the read scope; 403 ERR::AUTH::INSUFFICIENT_SCOPE otherwise). Records only exist within the retention window reported in page.retention_hours / page.retained_since; an empty page means nothing in that window. Paginate by passing page.next_before and page.next_before_id back as before and before_id while page.has_more is true, keeping all_projects the same. Not billing-grade: use spicrawl_usage for totals.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoRows per page, 1-200. Default 50.
beforeNoCursor: RFC 3339 timestamp, exactly as returned in page.next_before. Pass together with before_id.
statusNoOnly requests with this outcome.
before_idNoCursor: request id, exactly as returned in page.next_before_id.
only_errorsNoOnly requests whose status is anything other than 'success'.
all_projectsNoEvery project in your organization, not just the key's own. Needs the `read` scope. Keep it the same across pages.

Output Schema

ParametersJSON Schema
NameRequiredDescription
dataNoRequest records, newest first. Can be shorter than limit while page.has_more is true.
pageNoPaging cursor and the retention window.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the readOnlyHint/idempotentHint annotations by disclosing the retention window (page.retention_hours/retained_since), the meaning of an empty page, the exact 403 ERR::AUTH::INSUFFICIENT_SCOPE failure mode and required `read` scope, and the pagination contract. This is behavior an agent cannot infer from annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but front-loaded: core purpose first, then filtering, scope, retention, pagination, and the billing caveat. Every sentence earns its place, though the enumeration of returned fields is dense enough to slightly slow the opening.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a paginated, retention-bounded, scope-sensitive list tool, the description covers pagination mechanics, retention semantics, scope requirements, error behavior, and the distinction from the totals tool. An output schema exists, so omission of return formatting is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description still adds value by tying parameters together (paginate via page.next_before/next_before_id into before/before_id, keep all_projects stable across pages, only_errors semantics) and reinforcing intent behind status filtering.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Starts with a precise verb+resource+ordering statement ('Recent API calls in your key's project, newest first') and enumerates the fields returned, so the agent knows exactly what this list yields. It also distinguishes itself from sibling spicrawl_request_get (single call) and spicrawl_usage (totals) by naming the latter explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit use cases ('find failing calls (only_errors=true)' and 'spot patterns across a target') plus a strong when-not directive ('Not billing-grade: use spicrawl_usage for totals'). Also names the alternative scope mode (all_projects=true) with its precondition.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_scrapeScrape a URLAInspect

Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF. Handles the fetch vs. browser decision, charset decoding, and PDF-to-text for you. Optionally extract structured data with CSS selectors (extract) or autoparse (a natural-language ai_extract is coming soon), return discovered links, take a screenshot, or scope the DOM with include_tags/exclude_tags. Each successful call is billed in credits (a failure costs 0). Results are cached by default, and a cache hit is billed like the fetch that stored it (the cache saves time, not credits); set cache=false for time-sensitive pages. With the defaults it only reads the page. A method other than GET, HEAD or OPTIONS, and the actions that click, fill or run scripts on the page, can change state on the target site: set them only when the user asks for that. For markdown, text and HTML it returns { content }. With format: "json", or when extract, ai_extract, autoparse, links, screenshot or network_capture is set, it returns the JSON envelope: content, the site's status, credits, engine (the rendering tier: fetch, js or browser), warnings and data for extraction. Screenshots come back as image content blocks; each screenshots entry in the JSON keeps its metadata and says which block holds it. A PDF comes back as a resource content block, with its size, the rendering tier and the credits in the JSON.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe absolute URL to retrieve.
modeNoRouting mode. `auto` lets the platform escalate fetch -> browser as needed, billing only the rung that worked.
waitNoMilliseconds to wait after load before capturing (render only).
cacheNoServe a recent cached result if fresh (on by default). A hit is billed at the same price as the fetch that stored it: the cache saves time, not credits. Set false only when you need live data.
linksNoAlso return the page's discovered links as an absolute, de-duplicated list.
proxyNoYour own proxy URL to egress through. Takes precedence over the pool; no surcharge.
formatNoOutput format. 'markdown' (default) is best for feeding to an LLM; 'html' returns the raw document; 'json' returns a structured envelope; 'pdf' prints the page in a full browser (routed there for you) and returns the file as a resource content block.markdown
methodNoHTTP method sent to the target (default GET; no request body can be sent). Anything but GET, HEAD or OPTIONS can change state on the target site: use it only when the user asks.
renderNoRender with a real browser (executes JavaScript). Use for single-page apps or pages whose content is painted client-side. Costs more than the default fetch tier.
actionsNoBrowser steps run in order before capture (render only). Each step is {type, ...fields}. Examples: [{"type":"click","selector":"#load-more"},{"type":"wait_for","selector":".item:nth-child(20)","timeout_ms":5000}] · infinite scroll: [{"type":"scroll","to_bottom":true},{"type":"wait_for","ms":1000}] · login: [{"type":"fill","selector":"#user","value":"me"},{"type":"fill","selector":"#pass","value":"…","secret":true},{"type":"click","selector":"button[type=submit]","wait_for_navigation":true}]. A screenshot step switches the response to the JSON envelope (a FORMAT_COERCED warning says so).
extractNoSelector-based extraction: a map of field -> CSS/XPath selector, e.g. {"title":"h1","price":".amount"}. Returns structured `data`. Precise and cheap when you know the page structure.
stealthNoComing soon: not available yet, do not send. Stealth mode: render in a full browser hardened against bot detection. Default false.
max_costNoRefuse (rather than run) a request that would cost more than this many credits.
wait_forNoCSS selector to wait for before capturing (render only).
autoparseNoReturn the structured data the page publishes about itself (JSON-LD, OpenGraph, Twitter Card, microdata). No selectors; survives redesigns.
cache_ttlNoMax age in seconds of a cached copy to accept (0 forces a fresh fetch; capped at 48h). Freshness only: a hit is billed like a fetch.
parse_pdfNoTurn a PDF target into text/markdown (default true). false refuses PDFs.
ai_extractNoComing soon: not available yet, do not send. Model-driven extraction: describe what you want and let a model find it, no selectors. Refused with ERR::INTERNAL::UNAVAILABLE until it launches; use `extract` or `autoparse`.
screenshotNoCapture a screenshot (requires render).
session_idNoRun inside a session from spicrawl_session_create: same exit IP and browser state (cookies, localStorage) across calls.
sticky_keyNoComing soon: not available yet, do not send. Reuse the same managed-pool exit for every request carrying this key, without a session.
impersonateNoFetch tier only: present a real browser's TLS/JA3 + HTTP/2 fingerprint to clear passive bot gates. On by default; set false to send a plain client handshake.
exclude_tagsNoCSS selectors to remove (e.g. ['nav','.ad','#cookie']).
include_tagsNoCSS selectors to KEEP (everything else is dropped).
proxy_verifyNoVerify `proxy` is reachable before using it.
premium_proxyNoComing soon: not available yet, do not send. Use a residential exit from Spicrawl's managed pool instead of datacenter. Until then, pass your own `proxy`.
proxy_countryNoComing soon: not available yet, do not send. ISO-3166 alpha-2 country for a managed-pool exit, lowercase (e.g. 'us', 'de').
custom_headersNoExtra request headers sent to the target, e.g. {"Accept-Language":"de"}.
extract_presetNoComing soon: not available yet, do not send. A named extraction preset instead of an inline `extract` map; any value fails today. Put the rules in `extract`.
block_resourcesNoResource types the browser should not load, e.g. ["fonts","media","images"] (render only). Faster and cheaper.
network_captureNoRecord the API responses the PAGE fetched while rendering — often cleaner JSON than the DOM. Render only.
original_statusNoReturn the target's own HTTP status instead of 200 for a successful scrape.
wait_for_timeoutNoMax milliseconds to wait for `wait_for`.
main_content_onlyNoStrip the page to its main article, dropping nav/footer/aside. Defaults on for markdown.
screenshot_formatNoScreenshot image format.
screenshot_qualityNoScreenshot quality 1-100 (lossy formats).
screenshot_fullpageNoCapture the whole page, not just the viewport.
screenshot_selectorNoCapture only the element matching this CSS selector.
allowed_status_codesNoTarget statuses to treat as success instead of an error (e.g. [404] to capture a not-found page).

Output Schema

ParametersJSON Schema
NameRequiredDescription
pdfNoPresent for format=pdf: the file itself is a resource content block.
urlNoThe requested URL.
dataNoOutput of `extract` or `autoparse`.
linksNoThe page's links, absolute and de-duplicated, when `links` was set.
engineNoRendering tier that produced the page: fetch, js or browser.
statusNoThe target site's HTTP status (not this call's). Anything but 200/404/410 is charged 0.
contentNoThe page in the requested format (markdown, text or html); absent for a PDF.
creditsNoCredits charged for this call.
headersNoThe target's response headers.
networkNoResponses the page fetched, when `network_capture` was set.
warningsNoAdvisories about the call, e.g. an ignored option.
final_urlNoURL after redirects; may be empty.
truncatedNoThe target body exceeded the size cap and was cut.
diagnosticsNoWhich rendering tiers were tried, after an escalation.
screenshotsNoMetadata of each screenshot; the images are separate image content blocks.
empty_fieldsNo`extract` rules that matched nothing: usually a broken selector.
proxy_sourceNoWhose exit carried the request: pool, custom or direct.

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: credits billing per successful call with failures free, cache-hit billing parity (cache saves time, not credits), the warning that non-GET methods and click/fill/script actions can mutate the target site, and the tier escalation behavior. The annotations only say readOnly=false, openWorld=true, destructive=false, idempotent=false; the description explains why those are true and what the practical consequences are.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose, then options, then billing, then safety, then return shapes, so an agent can stop reading early. It is dense and long (heavy parentheticals), but with 39 parameters and several cross-cutting behaviors almost every clause carries non-redundant information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a highly complex open-world tool, the description covers cost model, caching semantics, state-change risk, format-dependent return shapes, screenshot/PDF content blocks, and the JSON envelope fields. Even with an output schema present, nothing an agent needs to invoke it safely is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds cross-parameter semantics the schema does not state as a whole: 'With format: json, or when extract, ai_extract, autoparse, links, screenshot or network_capture is set, it returns the JSON envelope.' It also flags unavailable parameters (ai_extract) and clarifies defaults, which helps an agent pick among the 39 options.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb and resource ('Retrieve a web page and return its content as markdown (default), text, HTML, a JSON envelope, or a printed PDF'), and the enumeration of output formats plus the fetch/browser handling makes it unmistakably the general-purpose single-URL scraper among its batch/session/usage siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete usage conditions: 'set `cache=false` for time-sensitive pages', 'set them only when the user asks for that' for state-changing methods and actions, and 'With the defaults it only reads the page.' It does not route the agent to siblings (e.g. batch_submit for multiple URLs), but within the tool the when-to-use guidance is explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_contextDump a session's stored contextA
Read-onlyIdempotent
Inspect

Return the full stored state of a LIVE session — cookies, localStorage and other credentials — plus its fingerprint and domain scores. The session_context object can be passed to spicrawl_session_create to clone the session. The result contains secrets (login cookies): do not echo it to the user or logs unless asked.

ParametersJSON Schema
NameRequiredDescriptionDefault
session_idYesThe session id returned by spicrawl_session_create or spicrawl_session_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
retiredNoWhy the session was retired; absent while usable.
warningNoThe API's reminder that this state dies with the session.
created_atNoCreation time, RFC 3339 UTC.
expires_atNoCurrent expiry, RFC 3339 UTC.
session_idYesThe session id.
fingerprintNoStored browser fingerprint; null when none.
usage_countNo
last_used_atNoLast use, RFC 3339 UTC; null if never used.
domain_scoresNoPer-domain standing, keyed by domain. Omitted when empty.
schema_versionNo
session_contextNoCookies and storage: secrets. Pass it as `session_context` to spicrawl_session_create to clone the session.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive). The description adds genuinely new behavioral context: the result contains live login cookies and other secrets and must not be echoed to the user or logs unless asked. That is important disclosure beyond the structured fields, though the failure mode for a non-LIVE session is left unstated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three compact sentences: payload contents first, downstream cloning use second, sensitivity warning last. Every sentence carries distinct information and nothing is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need only light description, and the description still characterizes them (cookies, fingerprint, domain scores). The secrets-handling warning closes the main risk gap; only edge behavior for expired/non-live sessions is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

One parameter with 100% schema description coverage, and the schema already explains that session_id comes from spicrawl_session_create or spicrawl_session_list. The description adds no format, validity, or lifecycle detail beyond what the schema provides, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Return the full stored state of a LIVE session') and enumerates the payload (cookies, localStorage, credentials, fingerprint, domain scores), which is far richer than a tautology. It does not explicitly contrast itself with the near-named sibling spicrawl_session_get, so an agent must infer that this returns raw stored state rather than session metadata.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete downstream use case — passing the session_context object to spicrawl_session_create to clone the session — which tells the agent why and when to call it. It offers no explicit 'when not to use' or comparison against session_get/session_list, so the guidance is contextual rather than exclusionary.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_createCreate a browsing sessionAInspect

Create a session: a persistent browser identity (cookie jar, localStorage, a fixed rendering tier and, by default, a sticky exit IP) that survives across scrapes. Use it for multi-step flows — log in, then fetch pages behind the login — by passing the returned id as session_id to every spicrawl_scrape call that should share the state. The rendering tier is fixed for the session's lifetime and every scrape on the session runs on it. Leave engine out and one is chosen for you. Release the session with spicrawl_session_release when done. Returns the session object (id, engine, status, sticky_key, proxy, expires_at).

ParametersJSON Schema
NameRequiredDescriptionDefault
engineNoRendering tier fixed for the session's lifetime. Optional: leave it out and one is chosen for you. `fetch` is plain HTTP; `js` is JavaScript rendering and refuses `actions`, `block_resources`, screenshots and PDFs; `browser` is a full browser.
rotate_ipNoComing soon: not available yet, do not send. Keep the cookie jar stable but rotate the managed-pool exit IP per request. Cannot combine with sticky_key.
sticky_keyNoComing soon: not available yet, do not send. Pin the managed-pool exit IP to this key; sessions sharing a key share an exit. Cannot combine with rotate_ip.
fingerprintNoComing soon: not available yet, do not send. Browser fingerprint pin, stored verbatim. Needs a full-browser session (`engine: "browser"`).
region_poolNoComing soon: not available yet, do not send. Managed-pool region to draw the exit from.
ttl_secondsNoIdle lifetime in seconds (min 30, default 1800; the organization's maximum is enforced, not clamped).
domain_scoresNoSeed per-domain error scores (usually copied from a spicrawl_session_context dump).
premium_proxyNoComing soon: not available yet, do not send. Use a residential managed-pool exit, exactly as on spicrawl_scrape.
proxy_countryNoComing soon: not available yet, do not send. ISO-3166 alpha-2 exit country; requires premium_proxy.
session_contextNoSeed the session's state (cookies, storage) — e.g. the `session_context` object from spicrawl_session_context, to clone a session.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesSession id; pass it as `session_id` to spicrawl_scrape.
leaseNoPresent only while a scrape holds the session; release and delete are refused until it lapses unless `force`.
proxyNoThe session's exit intent (never an IP or endpoint).
engineNoRendering tier the session is bound to: fetch, js or browser.
statusNoactive, released or expired; derived from the clock, and only `active` sessions can be used.
contextNoSize summary of the stored context (counts only, no secrets); null once the session is not active.
retiredNoWhy the session was taken out of service (expired, usage_exhausted, explicit); absent while usable.
warningsNoDecisions made on your behalf at creation, e.g. a TTL reduced to the organization maximum.
created_atNoCreation time, RFC 3339 UTC.
expires_atNoCurrent expiry (slides on use), RFC 3339 UTC.
project_idNoProject the session belongs to.
sticky_keyNoExit pin in use (empty when rotate_ip was set).
fingerprintNoBrowser fingerprint, as stored.
released_atNoRelease time, on released sessions, RFC 3339 UTC.
usage_countNoSuccessful scrapes that used the session.
last_used_atNoLast successful scrape, RFC 3339 UTC; null if never used.
domain_scoresNoPer-domain standing, keyed by domain. Omitted when empty.
hard_expires_atNoFixed ceiling the expiry never passes, RFC 3339 UTC.

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are sparse (readOnlyHint=false, openWorldHint=false, idempotentHint=false, destructiveHint=false). The description adds that the rendering tier is fixed for the session's lifetime, that sessions survive across scrapes, and that they must be released with spicrawl_session_release, which is meaningful for resource management. However, it doesn't disclose default TTL or expiration behavior beyond what the schema already shows.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense paragraph with no wasted sentences and is front-loaded with the essential definition. It is slightly long, but every clause (persistent identity, multi-step flow, engine fixed, release step) earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that an output schema exists, the description doesn't need to enumerate return values, though it helpfully mentions key fields. It covers the critical lifecycle (create → use via session_id → release), which is exactly what an agent needs for this stateful resource.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents every parameter thoroughly. The description adds a small amount of guidance (leave `engine` out and one is chosen) but not enough to go beyond the baseline 3 when the schema does all the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Create) and resource (a session/browsing identity), and elaborates exactly what a session is: a persistent state container (cookie jar, localStorage, fixed rendering tier, sticky exit IP). It distinguishes itself from siblings by naming spicrawl_session_release and spicrawl_scrape and describing their relationship.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when to use it (multi-step flows—log in then fetch pages behind the login) and how to use it with a sibling (pass the returned `id` as `session_id` to spicrawl_scrape calls that should share state). It also names a related lifecycle tool, spicrawl_session_release, for cleanup.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_deleteDelete a sessionA
DestructiveIdempotent
Inspect

Permanently delete a session and its stored state, including the record itself (prefer spicrawl_session_release if you only want to stop using it). Idempotent: deleting an already-deleted session succeeds. Refused while a scrape holds the session's lease unless force is true.

ParametersJSON Schema
NameRequiredDescriptionDefault
forceNoBreak an active lease (a scrape currently using the session). Only set when you know it is safe.
session_idYesThe session id returned by spicrawl_session_create or spicrawl_session_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
deletedYesAlways true on success (deleting an already-deleted session succeeds).
session_idYesThe id that was deleted.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already flag destructive=true and idempotent=true, but the description adds real behavioral context beyond them: what gets destroyed (the record and stored state), that repeat deletes succeed, and that deletion is refused while a scrape holds the lease unless force is set.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences with zero waste; the destructive scope is front-loaded, followed by the alternative and then edge-case behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema covering return values and annotations covering safety, the description supplies everything else an agent needs: destruction scope, idempotency, lease precondition, and escape hatch.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters are already documented in the schema. The description reinforces the force semantics ('unless force is true') but adds no syntax or format detail beyond what the schema provides; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Permanently delete a session and its stored state') and explicitly scopes it against the sibling spicrawl_session_release, so an agent can differentiate without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Names the alternative explicitly ('prefer spicrawl_session_release if you only want to stop using it') and states the selection condition, plus the refusal precondition involving an active lease. When-to-use and when-not-to-use are both covered.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_getGet a sessionA
Read-onlyIdempotent
Inspect

Return one session's metadata: status, engine (its rendering tier), proxy/exit, usage count, expiry, lease holder, and a summary of its stored context. Never returns the credentials themselves (use spicrawl_session_context for that).

ParametersJSON Schema
NameRequiredDescriptionDefault
session_idYesThe session id returned by spicrawl_session_create or spicrawl_session_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesSession id; pass it as `session_id` to spicrawl_scrape.
leaseNoPresent only while a scrape holds the session; release and delete are refused until it lapses unless `force`.
proxyNoThe session's exit intent (never an IP or endpoint).
engineNoRendering tier the session is bound to: fetch, js or browser.
statusNoactive, released or expired; derived from the clock, and only `active` sessions can be used.
contextNoSize summary of the stored context (counts only, no secrets); null once the session is not active.
retiredNoWhy the session was taken out of service (expired, usage_exhausted, explicit); absent while usable.
warningsNoDecisions made on your behalf at creation, e.g. a TTL reduced to the organization maximum.
created_atNoCreation time, RFC 3339 UTC.
expires_atNoCurrent expiry (slides on use), RFC 3339 UTC.
project_idNoProject the session belongs to.
sticky_keyNoExit pin in use (empty when rotate_ip was set).
fingerprintNoBrowser fingerprint, as stored.
released_atNoRelease time, on released sessions, RFC 3339 UTC.
usage_countNoSuccessful scrapes that used the session.
last_used_atNoLast successful scrape, RFC 3339 UTC; null if never used.
domain_scoresNoPer-domain standing, keyed by domain. Omitted when empty.
hard_expires_atNoFixed ceiling the expiry never passes, RFC 3339 UTC.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, and openWorldHint=false, so the safety profile is covered. The description adds valuable non-obvious behavior: it never returns credentials and points to the sibling tool that does, plus it lists the metadata fields exposed. It does not cover error behavior or lease semantics in depth.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tightly written sentences with no wasted words. The returned fields are front-loaded, and the credential limitation and alternative are appended as the key exception, making the structure easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter read tool with an output schema and rich annotations, the description is complete enough. It clarifies the return contents and the credential boundary, and the output schema handles the detailed response structure, leaving no critical gap for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the session_id parameter is already fully documented in the schema, including where it comes from. The description adds no additional syntax, format, or constraint details for the parameter, so the baseline score of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb (Return) and resource (one session's metadata) and enumerates the exact fields returned. It also explicitly distinguishes itself from the sibling spicrawl_session_context by noting it never returns credentials, so an agent can tell the two apart without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a clear context for use (fetch a session's metadata) and an explicit alternative for the credential case ('use spicrawl_session_context for that'). However, it does not state when to prefer this over spicrawl_session_list or describe prerequisites for calling it, so it falls short of full when/when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_listList sessionsA
Read-onlyIdempotent
Inspect

List this API key's project's sessions, newest first. Returns { sessions: [...], next_cursor? }; pass next_cursor back as cursor for the next page. Never includes the stored credentials.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoPage size (default 50, max 200).
cursorNo`next_cursor` from a previous page, verbatim.
engineNoOnly sessions on this rendering tier.
statusNoOnly sessions in this state.

Output Schema

ParametersJSON Schema
NameRequiredDescription
sessionsYesSessions on this page, newest first; may be short or empty under a `status` filter.
next_cursorNoPass back as `cursor` for the next page; absent on the last page.

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, so the bar is lower, and the description still adds a security-relevant trait: "Never includes the stored credentials." It also discloses the return envelope shape and that next_cursor is the pagination handle, going beyond structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with scope and ordering, then return shape and pagination, then the credential caveat. No filler and every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be fully explained, and the description covers scope, ordering, pagination, and the credential exclusion. It stops short of noting how the engine/status filters interact, but those are documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so all four params are self-documented, making 3 the baseline. The description's note that next_cursor is passed back as cursor largely restates the cursor param's own schema text, adding little beyond it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource+scope ("List this API key's project's sessions") and adds ordering semantics ("newest first"). An agent can tell this is a paginated list apart from the singular session_get / session_context siblings, though it doesn't explicitly name an alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the verb (list = enumerate sessions) and it explains the pagination loop, but there is no explicit when-to-use/when-not guidance or routing against siblings like requests_list or session_get. Adequate but with clear gaps.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_session_releaseRelease a sessionA
DestructiveIdempotent
Inspect

End a session when you are done with it: frees its exit IP and browser state; it can no longer be used with spicrawl_scrape. Its stored cookies and storage are purged and cannot be recovered; start a new session and log in again to continue. The record stays visible (status released) so its history can still be inspected. Refused while a scrape holds the session's lease unless force is true. Returns the session object.

ParametersJSON Schema
NameRequiredDescriptionDefault
forceNoBreak an active lease (a scrape currently using the session). Only set when you know it is safe.
session_idYesThe session id returned by spicrawl_session_create or spicrawl_session_list.

Output Schema

ParametersJSON Schema
NameRequiredDescription
idYesSession id; pass it as `session_id` to spicrawl_scrape.
leaseNoPresent only while a scrape holds the session; release and delete are refused until it lapses unless `force`.
proxyNoThe session's exit intent (never an IP or endpoint).
engineNoRendering tier the session is bound to: fetch, js or browser.
statusNoactive, released or expired; derived from the clock, and only `active` sessions can be used.
contextNoSize summary of the stored context (counts only, no secrets); null once the session is not active.
retiredNoWhy the session was taken out of service (expired, usage_exhausted, explicit); absent while usable.
warningsNoDecisions made on your behalf at creation, e.g. a TTL reduced to the organization maximum.
created_atNoCreation time, RFC 3339 UTC.
expires_atNoCurrent expiry (slides on use), RFC 3339 UTC.
project_idNoProject the session belongs to.
sticky_keyNoExit pin in use (empty when rotate_ip was set).
fingerprintNoBrowser fingerprint, as stored.
released_atNoRelease time, on released sessions, RFC 3339 UTC.
usage_countNoSuccessful scrapes that used the session.
last_used_atNoLast successful scrape, RFC 3339 UTC; null if never used.
domain_scoresNoPer-domain standing, keyed by domain. Omitted when empty.
hard_expires_atNoFixed ceiling the expiry never passes, RFC 3339 UTC.

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only say destructive/idempotent/not-read-only; the description goes well beyond them by disclosing exactly what is destroyed (exit IP, browser state, cookies and storage, irrecoverably), what survives (the record, status `released`, history inspectable), the lease-refusal behavior, and the `force` override. This is the richest part of the definition.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four tight clauses in two sentences, front-loaded with the action and its consequences, then the refusal rule and return. No filler and no repetition of the title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive, idempotent session-ending tool with an output schema already present, the description covers everything an agent needs: scope, irreversibility, downstream effects on spicrawl_scrape, retained record, and the lease conflict path. Return-value explanation is correctly omitted since the output schema carries it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds the missing trigger condition for `force` — refusal happens while a scrape holds the session's lease — which the schema only describes as 'break an active lease'. That contextual 'when is force needed' detail is genuine added value over the field-level text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb + resource ('End a session') with the consequences spelled out: frees exit IP and browser state, purges cookies/storage, record remains as status `released`. This distinguishes it from the sibling spicrawl_session_delete, which would imply record removal, without the agent having to open either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States the trigger clearly ('when you are done with it'), names the blocked follow-on ('can no longer be used with spicrawl_scrape'), and gives the recovery path ('start a new session and log in again'). It also documents the refusal condition and the escape hatch (`force` true). It never explicitly routes the agent to spicrawl_session_delete as the alternative, which keeps it just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_usageUsage breakdownA
Read-onlyIdempotent
Inspect

Billing-grade usage for your org over a date window, from the daily rollup (usage_daily), grouped by day, project, engine, feature or key. group_by=key returns per API key (key_id, name, prefix, project_id) counters per metric per day, from usage_key_daily; keyless (dashboard) usage is not in it, and a project-scoped key sees only its own project's keys. Use it to answer 'how many requests/credits did I use, and where'. The window defaults to the last 30 days including today; to is EXCLUSIVE (a single day 2026-08-01 is from=2026-08-01, to=2026-08-02) and the window may not exceed 400 days. The response carries stale_seconds (age of the newest rollup row; null when the window is empty). Requires the key to have the read scope.

ParametersJSON Schema
NameRequiredDescriptionDefault
toNoEXCLUSIVE end date, YYYY-MM-DD (UTC). Default: tomorrow, so today is included.
fromNoInclusive start date, YYYY-MM-DD (UTC). Default: 29 days before today.
metricsNoMetrics to return. Omit for all of them.
group_byNoHow to group the rows. Default 'day'.
project_idNoRestrict to one project id (must be a valid Spicrawl identifier).

Output Schema

ParametersJSON Schema
NameRequiredDescription
keysNogroup_by=key only: one entry per API key, with its counters per UTC day.
groupsNoThe buckets, sorted by key. Empty (or missing for group_by=key) does not mean no usage: read totals.
periodNoThe window, as a half-open range of UTC days.
totalsNoBilled quantities for the whole window. Authoritative; use these, not the sum of groups.
group_byNoThe dimension used: day, project, engine, feature or key.
warningsNoCaveats the numbers would otherwise hide.
usage_as_ofNoWhen the newest rollup row in the window was written; null when the window is empty.
unattributedNoThe part of each total the groups do not explain.
stale_secondsNoAge of usage_as_of in seconds; null when the window is empty. A large value means the rollup is behind.

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover only the safety profile (readOnly/idempotent/non-destructive), and the description adds substantial context beyond them: the default 30-day window, the EXCLUSIVE end date with a worked example, a hard 400-day window cap, the stale_seconds freshness field, the requirement for the `read` scope, and the blind spot for keyless usage. That is a rich, non-obvious behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single dense paragraph rather than a bulleted spec, which hurts scannability slightly, but the most important facts (what it returns, what it excludes, then window/date semantics, then auth) are front-loaded and each sentence carries non-redundant information. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the description rightly does not enumerate return fields; it still flags the one response property worth knowing (stale_seconds). Combined with window bounds, exclusivity, grouping caveats and the required scope, an agent has everything needed to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is already 100%, so the baseline is 3, but the description goes further by explaining semantics the schema only gestures at: `to` is exclusive with a single-day example, the default window is last 30 days including today, and group_by=key changes both the source table and the returned counter shape. Minor gaps remain around how metrics interact with grouping, but the added meaning is real.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb+resource: billing-grade org usage over a date window, drawn from the usage_daily rollup and groupable by day/project/engine/feature/key. It even names the alternate source table (usage_key_daily) for key grouping. It never names its obvious siblings (spicrawl_usage_summary, spicrawl_usage_reconciliation), so the agent gets no explicit differentiation within the usage family.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a concrete usage condition: "Use it to answer 'how many requests/credits did I use, and where'", and supplies negative scoping for group_by=key (keyless dashboard usage is absent; a project-scoped key only sees its own project). It stops short of stating when to prefer this over spicrawl_usage_summary or spicrawl_usage_reconciliation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_usage_reconciliationUsage reconciliationA
Read-onlyIdempotent
Inspect

Drift report for ONE day: re-derives usage from that day's request log and compares it with the billed daily rollup, so you can see whether the rollup missed or double-counted anything. Defaults to yesterday (UTC); today is still accumulating and would always show drift. Requires the read scope.

ParametersJSON Schema
NameRequiredDescriptionDefault
dayNoThe UTC day to reconcile, YYYY-MM-DD. Default: yesterday.

Output Schema

ParametersJSON Schema
NameRequiredDescription
dayNoThe UTC day reconciled.
driftNoDisagreements; empty means no drift.
warningsNoCaveats the numbers would otherwise hide.
repairableNoAlways false; nothing is repaired.
interpretationNoWhat the drift can and cannot prove. Read it before acting on drift.
request_log_rowsNoRows in the request log for the day; 0 means there was nothing to compare against.

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true and destructiveHint=false, so safety is covered. The description adds genuinely non-derivable context: the required `read` scope and the reason for the yesterday default (today's rollup is still accumulating), which is exactly the kind of behavioral detail annotations cannot carry.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler. The core behavior is front-loaded and the operational caveats (default, today's instability, required scope) follow in a single compact clause.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need no explanation, and the description covers scope, defaulting and auth. The only gap is operational cost: re-deriving a full day from the request log may be expensive or slow, and nothing here warns about that.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single `day` parameter is fully documented there, so the baseline is 3. The description goes slightly beyond by justifying the default (yesterday, because today always shows drift), which helps an agent decide whether to pass the parameter at all.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb and resource ('re-derives usage from that day's request log and compares it with the billed daily rollup') and scopes it to ONE day, which separates it in an agent's mind from the raw-usage siblings spicrawl_usage and spicrawl_usage_summary. The 'drift report' framing makes the output intent unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States when the tool applies (a settled UTC day) and effectively excludes one case: 'today is still accumulating and would always show drift.' It never names the sibling alternatives (spicrawl_usage, spicrawl_usage_summary) that an agent might reach for instead, so routing is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

spicrawl_usage_summaryUsage summaryA
Read-onlyIdempotent
Inspect

A compact current-period summary of your org's usage (totals as of now, from the daily rollup). No parameters. Use for a quick 'how much have I used so far'; use spicrawl_usage for a windowed breakdown. Requires the read scope.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
planNo
periodNoThe window, as a half-open range of UTC days.
creditsNoMetered usage against the plan's included credits. Micro-credits: 1 credit = 1000000.
metricsNoEvery billed metric for the period.
warningsNoCaveats the numbers would otherwise hide.
allowanceNoThe monthly allowance the API enforces. Omitted when it cannot be read (warnings says so).
usage_as_ofNoWhen the newest rollup row in the window was written; null when the window is empty.
unattributedNoMetric name -> quantity. credits are micro-credits (1 credit = 1000000), engine_ms milliseconds, bytes_* bytes, the rest counts.
stale_secondsNoAge of usage_as_of in seconds; null when the window is empty. A large value means the rollup is behind.
period_elapsed_daysNoWhole days of the period elapsed; use it to pace consumption.

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, non-destructive, and closed-world behavior. The description adds two things annotations cannot: the auth requirement ('Requires the `read` scope') and the data-freshness caveat implied by 'from the daily rollup' (totals are rolled-up, not real-time). Missing only a note on lag magnitude or pagination, so not a full 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with what the tool returns, followed by the routing guidance and the auth prerequisite. No filler; every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need no prose. With annotations covering safety and the description covering scope, freshness source, alternative routing, and auth scope, an agent has everything needed to call this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Zero parameters, so the baseline is 4. The description explicitly confirms 'No parameters', which matches the empty schema and prevents an agent from hunting for filter options.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource+scope: a current-period, org-level usage summary sourced from the daily rollup. It explicitly distinguishes itself from the sibling spicrawl_usage by naming it and the axis of difference (compact current totals vs. windowed breakdown).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit when-to-use ('quick how much have I used so far') and names the alternative tool plus the condition that selects it ('use spicrawl_usage for a windowed breakdown'). No inference required.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 25 tool updatesv0.1.8
    • First observedspicrawl_batch_add_items
    • First observedspicrawl_batch_cancel
    • First observedspicrawl_batch_close
    • First observedspicrawl_batch_list
    • First observedspicrawl_batch_results
    • First observedspicrawl_batch_retry
    • First observedspicrawl_batch_status
    • First observedspicrawl_batch_submit
    • First observedspicrawl_batch_task_content
    • First observedspicrawl_browser_connect_url
    • First observedspicrawl_docs_index
    • First observedspicrawl_docs_read
    • First observedspicrawl_docs_search
    • First observedspicrawl_request_get
    • First observedspicrawl_requests_list
    • First observedspicrawl_scrape
    • First observedspicrawl_session_context
    • First observedspicrawl_session_create
    • First observedspicrawl_session_delete
    • First observedspicrawl_session_get
    • First observedspicrawl_session_list
    • First observedspicrawl_session_release
    • First observedspicrawl_usage
    • First observedspicrawl_usage_reconciliation
    • First observedspicrawl_usage_summary

TDQS

A4.2/5.0

Scored across 25 tools

Disambiguation4/5

Most tools target clearly distinct resources and actions, and the descriptions explicitly disambiguate close pairs (usage vs usage_summary, session_get vs session_context, session_release vs session_delete, batch_status vs batch_list). The only mild overlaps are batch_results vs batch_task_content (page vs single item) and requests_list vs request_get, but the descriptions make the intended use clear.

Naming Consistency5/5

Every tool follows a uniform `spicrawl_` prefix with snake_case `resource_action` naming (batch_submit, session_create, usage, docs_read). The only minor deviation is the plural `spicrawl_requests_list` against the singular `spicrawl_request_get`, which is idiomatic rather than inconsistent.

Tool Count4/5

25 tools is at the heavy end, but the server covers several genuine families (scrape, batch lifecycle, session lifecycle, usage/billing, request logs, docs), so most tools earn their place. One tool (browser_connect_url) is a not-yet-functional stub that adds little weight.

Completeness4/5

Batch and session management have full lifecycles (submit/status/results/content/list/add/close/cancel/retry and create/list/get/context/release/delete), plus scrape, usage, request logs and docs search/read/index. Gaps are minor or announced: browser_connect_url is unavailable ('coming soon') and ai_extract/autoparse is not yet implemented.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables agents to scrape, crawl, map, search, and extract web pages as clean markdown or structured JSON directly through MCP tools.
    AGPL 3.0
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables MCP-compatible AI agents to search, scrape, crawl, map, and interact with live web pages, returning clean, structured, agent-ready content. It also supports recurring page monitoring, autonomous deep research, and credit-usage checks.
    102,401 npm
    1
    MIT
  • F
    license
    Not graded
    quality
    A
    maintenance
    Enables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.
    -