Run a crawl recipe
crawl_recipeRun a recipe of browser steps against a page: fill a form, click through, paginate, and extract fields. Use crawl instead when you only need a page's HTML.
blocks run in order. The first must be {"type": "start", "details": {"source": "<url>"}}; only its url is used, and no later block may be a start block. Then any of: extract (details: selector, name, property -- e.g. textContent, href, innerHTML), click (selector), input (selector, text), remove (selector), scroll-to-element (selector), wait (seconds), paginate (selector to click, startBlock: the id of the block to rewind to; both blocks need an id), screenshot, save-html, jsonschema, datalayer, extract-variable, add-field.
A recipe of only the start block answers with the page's HTML in content (markdown with output, which is refused for anything longer); anything longer answers with the extracted result in data. expectedOutput "item" folds everything into one object, "list" keeps a row per entry. Spends a flat number of credits. If status is "queued" or "running", call crawl_poll with the job_id to collect it.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"blocks": "<blocks>"
},
"name": "crawl_recipe"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | Render the page in a browser so javascript runs. Turn off for static pages to pay less. | |
| city | No | The city inside that country to appear to browse from, as the pool spells it: letters, digits, hyphens or underscores, e.g. berlin or new_york. Requires country. Doubles the price of the crawl. | |
| name | No | A label for the run, seen only in the crawler's logs. | |
| touch | No | Report the screen as touch capable. Requires viewport. | |
| blocks | Yes | The steps, in the order they run. The first is the start block naming the url. | |
| images | No | Download the images on the page. | |
| mobile | No | Tell the page it is on a phone. Requires viewport; pair it with user_agent for sites that sniff it. | |
| output | No | How the page comes back in `content`: "html" verbatim, or "markdown" -- smaller and easier to read. Free either way. | html |
| adblock | No | Drop ads and trackers before the page loads. Free either way. | |
| country | No | The country to appear to browse from, as a two-letter ISO 3166-1 code such as es, us or de. Free. Required to use city. | |
| premium | No | Fetch through the premium proxy pool instead of the standard one. Costs more; use it for targets that refuse ordinary requests. | |
| viewport | No | The window size as WIDTHxHEIGHT, e.g. 390x844. Required for device_scale_factor, mobile, landscape and touch. | |
| landscape | No | Report the screen as landscape. Requires viewport. | |
| user_agent | No | The user agent to send instead of the crawler's own. | |
| wait_until | No | When the page counts as loaded. Use domcontentloaded or load for pages that keep a connection open. | networkidle2 |
| main_content | No | With `output` "markdown" only: convert just the page's main block, without navigation, header, footer, sidebars or cookie banners. Falls back to the whole page when no main block with real text is found. | |
| cookies_banner | No | How to answer cookie banners: accept opts in, reject opts out where the banner allows it, none leaves them on the page. | accept |
| expectedOutput | No | "item" folds the output into one object; "list" keeps a row per entry, for recipes that paginate or extract many rows. | item |
| device_scale_factor | No | The device pixel ratio, e.g. 3 for a recent phone. Requires viewport. |