spidra_crawl
Crawl websites from a single starting URL by following links per your instructions. Extract structured data from discovered pages; returns a job ID for async status checks.
Instructions
Crawl a website starting from one URL: Spidra discovers pages by following links according to your plain-language instruction, and optionally extracts structured data from every page. Returns IMMEDIATELY with a jobId — it does not wait.
Best for: extracting from many pages when you do NOT know their URLs upfront (docs sites, blogs, product catalogs). Not for: URLs you already know (use spidra_scrape or spidra_batch_scrape — cheaper and faster). Workflow: call this, then poll spidra_check_crawl_status with the jobId every 10-15 seconds until terminal. Do NOT resubmit while a crawl is pending. Cancel a mistake with spidra_cancel_crawl.
Behavior notes:
"crawlInstruction" controls which links are followed (e.g. "Follow blog post links only, skip tag pages").
"transformInstruction" controls what is extracted per page; omit it (and schema) for raw markdown with no AI token cost.
Keep "maxPages" small (default 5, max 50) — every page costs credits.
Usage example:
{
"name": "spidra_crawl",
"arguments": {
"baseUrl": "https://example.com/blog",
"crawlInstruction": "Follow blog post links only, skip tag and category pages",
"transformInstruction": "Extract the title, author, and publish date",
"maxPages": 10
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| schema | No | JSON Schema enforcing the exact output shape. Define EVERY field you want extracted — an untyped object with no properties comes back empty. Missing fields return null instead of hallucinated values. | |
| baseUrl | Yes | Starting URL for the crawl | |
| cookies | No | ||
| maxDepth | No | Max link depth from the base URL. 0 = base URL only. | |
| maxPages | No | Max pages to crawl (default 5). Keep small — each page costs credits. | |
| useProxy | No | Route through a residential proxy (for blocked/geo-restricted sites) | |
| excludePaths | No | URL path patterns to skip, e.g. ["/tag/*"] | |
| includePaths | No | URL path patterns to include, e.g. ["/blog/*"] | |
| proxyCountry | No | Two-letter country code for the proxy, e.g. "us", "de", "jp", or "eu"/"global" | |
| allowSubdomains | No | ||
| crawlInstruction | Yes | Which links to follow, in plain language | |
| crawlEntireDomain | No | ||
| ignoreQueryParams | No | ||
| transformInstruction | No | What to extract from each page. Omit for raw markdown (no AI cost). |