Crawl a whole site once
crawl_siteCrawl a site once from a start URL, following links and the sitemap, and return the pages read. No project is set up. Use it for a site or section whose page URLs you do not know; for known URLs use scrape_urls. To watch the site afterwards, pass the crawl_id to create_project. Inputs: include_paths=['/blog/*'] reads the blog index and everything under it; paths match the URL path, never the host. Excluded pages do not count toward limit. limit=200 with max_depth=3 suits one section. config takes the keys get_project lists when called with no project_id. Cost: 1 to 4 credits per page, by the engine that read it; refused pages are free. Timing: it waits for the crawl, minutes for a large limit. If it outlasts the connection's time limit, a job comes back for get_job, and calling again returns the same crawl. The crawl is kept for a day. Needs write access; a read-only connection is refused with code 'read_only'.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl_site"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The absolute http(s) URL to start from, e.g. the home page. | |
| limit | No | The most pages to read, 1 to 5,000 (default 50); the plan's page cap also applies. | |
| config | No | Fetch settings to override, as {key: value}; omit for the defaults. | |
| max_depth | No | Link hops to follow from the start URL (0 = that page only; default 3). | |
| exclude_paths | No | Globs over the URL path to skip, e.g. ['/tag/*', '*.pdf']; an exclude wins over an include. | |
| include_paths | No | Globs over the URL path to keep; a bare '/blog/' matches that one page only. Omit for the whole site. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| job | No | While it is still going: {kind: 'crawl', id} for get_job and cancel_job. | |
| url | No | The start URL. | |
| code | No | Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found. | |
| note | No | What was cut, and how to read a page in full. | |
| stop | No | Why the crawl ended early, e.g. its page limit; empty otherwise. | |
| error | No | Only when the call failed: what went wrong, in words an assistant can act on. | |
| index | No | Every page read (up to 500) as {url, title, words, status}. | |
| pages | No | Up to 50 pages, each {url, title, words, status, links (a count), markdown excerpt}. | |
| counts | No | Pages by outcome, e.g. {ok, blocked, error}. | |
| cursor | No | Present when more pages exist: pass it to get_job for the next window. | |
| detail | No | Only on a failure: the API's own wording, when the error was rephrased. | |
| notice | No | Only when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer. | |
| status | No | 'done', 'cancelled' or 'error' once finished, 'running' with a `job` while it goes on. On an error, the HTTP status code. | |
| notices | No | With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself. | |
| crawl_id | No | This crawl's id: for get_job, cancel_job, and create_project(crawl_id=...) to keep it. |