Skip to main content
Glama
spidra-mcp-server

spidra-mcp-server

Official
by spidra-io

spidra_crawl

Read-only

Crawl a website from a starting URL and follow links to find pages matching your instruction, optionally extracting structured data from each page; returns a job ID immediately.

Instructions

Start at one URL and follow links to the kind of pages you describe, optionally extracting data from each. Returns a "jobId" straight away.

Use it when you need many pages of one kind and don't know their URLs (every blog post, every product in a category, every page of docs). Don't use it when you know the URLs (spidra_batch_scrape) or only need a site's list of URLs (spidra_map, much cheaper). Then: check it with spidra_get_job (type "crawl") every 10 to 15 seconds until it's completed, failed or cancelled. Don't start it again while it runs. Stop a mistake with spidra_cancel_job.

  • "crawlInstruction" says which pages to return, e.g. "Individual book pages" or "Blog posts, not tag pages".

  • "transformInstruction" or "schema" extracts data from each page with AI. Leave both out for each page as Markdown, with no AI cost.

  • "maxPages" counts only pages returned (default 5). Pages read just to find them are free. Plan limits per crawl: Free 100, Starter 1,000, Builder 5,000, Pro 10,000.

  • The crawl stays under the start URL's path unless "crawlEntireDomain" is true.

  • A page you expected is missing? spidra_crawl_report says why each link was skipped.

Cost: 1 credit per page returned, plus AI credits with a transformInstruction or schema. A crawl that returns nothing is free.

Example: {"baseUrl": "https://books.toscrape.com", "crawlInstruction": "Individual book pages", "transformInstruction": "Title and price", "maxPages": 10}

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "baseUrl": "<baseUrl>",
      "crawlInstruction": "<crawlInstruction>"
    },
    "name": "spidra_crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
schemaNoA JSON Schema for the answer. You get exactly these fields and types, with null for anything the page doesn't show. Define every field you want: an object with no properties comes back empty. The root must be an object.
baseUrlYesWhere the crawl starts
cookiesNoCookies from the user's own signed-in browser session, for pages behind a login, e.g. "session=abc; token=xyz". Never stored.
maxDepthNoHow many links deep to go from the start URL. 0 is the start URL only.
maxPagesNoMost pages to return (default 5). Each costs 1 credit.
useProxyNoStealth Mode: browse through residential proxies, for sites that block or vary by country. Billed by bandwidth, not credits.
useSitemapNoUse the site's sitemap to find pages (default true)
excludePathsNoNever follow these URL paths, e.g. ["/tag/*"]
includePathsNoOnly follow URL paths matching these patterns, e.g. ["/blog/*"]
proxyCountryNoWith useProxy, the country to browse from: a two-letter code such as "us", "gb" or "de", or "global", "eu" or "asia"
respectRobotsNoFollow the site's robots.txt rules (default false)
allowSubdomainsNoAlso follow links to subdomains
crawlInstructionYesWhich pages to return, in plain words
crawlEntireDomainNoGo beyond the start URL's path to the whole site
ignoreQueryParamsNoTreat URLs that differ only by query string as one
transformInstructionNoWhat to extract from each page. Leave out for Markdown (no AI cost).

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed22 schema fields changedv0.7.0
    • addedInput schema / properties / allowSubdomains / description
      Added value: +"Also follow links to subdomains"
    • changedInput schema / properties / baseUrl / description
      Previous value: -"Starting URL for the crawl"New value: +"Where the crawl starts"
    • addedInput schema / properties / cookies / description
      Added value: +"Cookies from the user's own signed-in browser session, for pages behind a login, e.g. \"session=abc; token=xyz\". Never stored."
    • addedInput schema / properties / crawlEntireDomain / description
      Added value: +"Go beyond the start URL's path to the whole site"
    • changedInput schema / properties / crawlInstruction / description
      Previous value: -"Which links to follow, in plain language"New value: +"Which pages to return, in plain words"
    • changedInput schema / properties / excludePaths / description
      Previous value: -"URL path patterns to skip, e.g. [\"/tag/*\"]"New value: +"Never follow these URL paths, e.g. [\"/tag/*\"]"
    • addedInput schema / properties / ignoreQueryParams / description
      Added value: +"Treat URLs that differ only by query string as one"
    • changedInput schema / properties / includePaths / description
      Previous value: -"URL path patterns to include, e.g. [\"/blog/*\"]"New value: +"Only follow URL paths matching these patterns, e.g. [\"/blog/*\"]"
    • changedInput schema / properties / maxDepth / description
      Previous value: -"Max link depth from the base URL. 0 = base URL only."New value: +"How many links deep to go from the start URL. 0 is the start URL only."
    • addedInput schema / properties / maxDepth / maximum
      Added value: +9007199254740991
    • addedInput schema / properties / maxDepth / minimum
      Added value: +0
    • changedInput schema / properties / maxDepth / type
      Previous value: -"number"New value: +"integer"
    • changedInput schema / properties / maxPages / description
      Previous value: -"Max pages to crawl (default 5). Keep small — each page costs credits."New value: +"Most pages to return (default 5). Each costs 1 credit."
    • changedInput schema / properties / maxPages / maximum
      Previous value: -50New value: +10000
    • changedInput schema / properties / maxPages / type
      Previous value: -"number"New value: +"integer"
    • changedInput schema / properties / proxyCountry / description
      Previous value: -"Two-letter country code for the proxy, e.g. \"us\", \"de\", \"jp\", or \"eu\"/\"global\""New value: +"With useProxy, the country to browse from: a two-letter code such as \"us\", \"gb\" or \"de\", or \"global\", \"eu\" or \"asia\""
    • addedInput schema / properties / respectRobots
      Added value: +{
      +  "description": "Follow the site's robots.txt rules (default false)",
      +  "type": "boolean"
      +}
    • changedInput schema / properties / schema / additionalProperties
      Previous value: -falseNew value: +{}
    • changedInput schema / properties / schema / description
      Previous value: -"JSON Schema enforcing the exact output shape. Define EVERY field you want extracted — an untyped object with no properties comes back empty. Missing fields return null instead of hallucinated values."New value: +"A JSON Schema for the answer. You get exactly these fields and types, with null for anything the page doesn't show. Define every field you want: an object with no properties comes back empty. The root must be an object."
    • changedInput schema / properties / transformInstruction / description
      Previous value: -"What to extract from each page. Omit for raw markdown (no AI cost)."New value: +"What to extract from each page. Leave out for Markdown (no AI cost)."
    • changedInput schema / properties / useProxy / description
      Previous value: -"Route through a residential proxy (for blocked/geo-restricted sites)"New value: +"Stealth Mode: browse through residential proxies, for sites that block or vary by country. Billed by bandwidth, not credits."
    • addedInput schema / properties / useSitemap
      Added value: +{
      +  "description": "Use the site's sitemap to find pages (default true)",
      +  "type": "boolean"
      +}
  2. First observedv0.2.2

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover readOnly/openWorld; the description adds the async job model (returns jobId immediately, must be polled), concurrency caution, path-scoping default, sitemap/robots defaults, plan page limits, and a full cost model including that discovery pages and empty crawls are free. That is substantial context beyond structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Bold section labels, bullets and a worked example make it scannable and front-loaded with the decision criteria. It is on the long side and repeats a couple of schema facts (maxPages default 5, cost per page) that already live in the parameter descriptions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 16-parameter async tool with no output schema, the description supplies everything an agent needs: what is returned (jobId), how to follow up, how to abort, cost implications, and where to diagnose missing pages (spidra_crawl_report). No material gap remains.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning: maxPages counts only returned pages while traversal pages are free, per-tier limits, and concrete examples for crawlInstruction and the transformInstruction/schema tradeoff (leave both out for free Markdown). It does not explain the path-pattern or proxy parameters, which the schema already handles.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a precise verb chain (start at one URL, follow links, optionally extract) and states the immediate return value ("jobId"). It also names the sibling tools it is not (spidra_batch_scrape, spidra_map), so an agent can route correctly without reading any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit "Use it when" / "Don't use it when" sections name the alternatives and the deciding condition (unknown URLs vs known URLs vs URL list only). It goes further with a "Then" workflow: poll spidra_get_job every 10–15s, don't restart while running, cancel via spidra_cancel_job.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.