Scrape pages by URL
scrape_urlsFetch pages whose URLs you already have and return their content. No project is created. Use crawl_site when the URLs must be found by following links, and map_site to list a site's URLs without fetching. list_pages with url reads a page a project already stored, for free. Inputs: urls come from the request or from map_site. formats adds bodies, e.g. 'markdown,text'. config overrides fetch settings, e.g. {"render_js": "always"}; get_project with no project_id lists the keys. full=true reads one URL in full and runs config.actions first, e.g. [{"type": "click", "selector": "#load-more", "repeat": "until_gone"}] or [{"type": "scroll", "repeat": 5}] (each scroll waits for the page to grow); a step's unknown key is refused. formats="markdown,screenshot" also takes a picture of the page; for one URL it is given as screenshotUrl. Cost: 1 credit for a plain fetch, 4 with a browser, +1 for a screenshot. A page the site refuses is reported as blocked and is free. A short page is rendered once to see what its scripts add (example.com adds five languages), and kept at 4 when they add text; config {"render_js": "never"} reads only what the server sent, at 1. Lists: a list runs as one batch. If it outlasts the connection's time limit, a job comes back for get_job, and sending the same list again within ten minutes returns that batch, marked reused with its ageSeconds; after that, or with force_refresh=true, the URLs are read again. Needs write access; a read-only connection is refused with code 'read_only'.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"urls": "<urls>"
},
"name": "scrape_urls"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| full | No | true reads the one URL in full -- raw html, head and extracted fields, images, the engine used -- running config's browser actions first. Default false. | |
| urls | Yes | Absolute http(s) URLs, 1 to 500; exactly one when full is true. | |
| config | No | Fetch settings to override, as {key: value}; omit for the defaults. | |
| formats | No | Bodies per page, comma-separated from markdown (default), text, cleanHtml and rawHtml, plus screenshot for a picture of the page (+1 credit; one URL gives it as screenshotUrl). With full=true, config's formats list chooses them instead. | markdown |
| force_refresh | No | true reads the URLs again now, even when the same call was made in the last ten minutes (which otherwise returns that read, marked reused). Default false. | |
| parse_documents | No | Read PDF (a scanned one by OCR) and Word (.docx) URLs as text (default true). A parse_documents key in config wins. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| job | No | While a batch is still going: {kind: 'batch', id} for get_job and cancel_job. | |
| code | No | Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found. | |
| note | No | What was left out and how to get it. | |
| page | No | full=true only: the page in full -- markdown, html, head {title, description, canonical, h1}, fields, images, engine, pageStatus -- with each body capped at 12,000 characters. | |
| rest | No | A list past its index: {count, byStatus, notOk} for the urls not shown. | |
| error | No | Only when the call failed: what went wrong, in words an assistant can act on. | |
| index | No | A list only: every url as {url, title, words, status}, so none is lost when excerpts are cut. A url that did not come back ok also has reason and errorCode, e.g. status 'unreachable' with errorCode DNS_FAILED is a name that does not resolve ('missing' is a 404). | |
| pages | No | The pages read. A list: up to 50, each {url, title, words, status, links (a count), markdown excerpt}, the excerpts sharing 60,000 characters. One URL: that page in full, each body asked for capped at 12,000 characters. | |
| detail | No | Only on a failure: the API's own wording, when the error was rephrased. | |
| notice | No | Only when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer. | |
| reused | No | true when this is the read an identical call made in the last ten minutes, not a new one; force_refresh=true reads again. | |
| status | No | 'done' once read; 'running' with a `job` while a batch goes on; with full=true, 'blocked' when the site refused and 'error' when the read failed. On an error, the HTTP status code. | |
| notices | No | With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself. | |
| ageSeconds | No | With reused: how many seconds ago that read finished. |