spidra_crawl
Crawl a website from a starting URL and follow links to find pages matching your instruction, optionally extracting structured data from each page; returns a job ID immediately.
Instructions
Start at one URL and follow links to the kind of pages you describe, optionally extracting data from each. Returns a "jobId" straight away.
Use it when you need many pages of one kind and don't know their URLs (every blog post, every product in a category, every page of docs). Don't use it when you know the URLs (spidra_batch_scrape) or only need a site's list of URLs (spidra_map, much cheaper). Then: check it with spidra_get_job (type "crawl") every 10 to 15 seconds until it's completed, failed or cancelled. Don't start it again while it runs. Stop a mistake with spidra_cancel_job.
"crawlInstruction" says which pages to return, e.g. "Individual book pages" or "Blog posts, not tag pages".
"transformInstruction" or "schema" extracts data from each page with AI. Leave both out for each page as Markdown, with no AI cost.
"maxPages" counts only pages returned (default 5). Pages read just to find them are free. Plan limits per crawl: Free 100, Starter 1,000, Builder 5,000, Pro 10,000.
The crawl stays under the start URL's path unless "crawlEntireDomain" is true.
A page you expected is missing? spidra_crawl_report says why each link was skipped.
Cost: 1 credit per page returned, plus AI credits with a transformInstruction or schema. A crawl that returns nothing is free.
Example: {"baseUrl": "https://books.toscrape.com", "crawlInstruction": "Individual book pages", "transformInstruction": "Title and price", "maxPages": 10}
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"baseUrl": "<baseUrl>",
"crawlInstruction": "<crawlInstruction>"
},
"name": "spidra_crawl"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| schema | No | A JSON Schema for the answer. You get exactly these fields and types, with null for anything the page doesn't show. Define every field you want: an object with no properties comes back empty. The root must be an object. | |
| baseUrl | Yes | Where the crawl starts | |
| cookies | No | Cookies from the user's own signed-in browser session, for pages behind a login, e.g. "session=abc; token=xyz". Never stored. | |
| maxDepth | No | How many links deep to go from the start URL. 0 is the start URL only. | |
| maxPages | No | Most pages to return (default 5). Each costs 1 credit. | |
| useProxy | No | Stealth Mode: browse through residential proxies, for sites that block or vary by country. Billed by bandwidth, not credits. | |
| useSitemap | No | Use the site's sitemap to find pages (default true) | |
| excludePaths | No | Never follow these URL paths, e.g. ["/tag/*"] | |
| includePaths | No | Only follow URL paths matching these patterns, e.g. ["/blog/*"] | |
| proxyCountry | No | With useProxy, the country to browse from: a two-letter code such as "us", "gb" or "de", or "global", "eu" or "asia" | |
| respectRobots | No | Follow the site's robots.txt rules (default false) | |
| allowSubdomains | No | Also follow links to subdomains | |
| crawlInstruction | Yes | Which pages to return, in plain words | |
| crawlEntireDomain | No | Go beyond the start URL's path to the whole site | |
| ignoreQueryParams | No | Treat URLs that differ only by query string as one | |
| transformInstruction | No | What to extract from each page. Leave out for Markdown (no AI cost). |