web_crawl
Crawl one site from a start URL and return each page as deduplicated Markdown while obeying robots.txt. Use when page URLs are not known in advance.
Instructions
Crawl one site from a start URL and return every page as markdown, deduplicated, robots.txt obeyed. Use when the pages are not known in advance; when they are, web_fetch_many is cheaper, and web_map lists a site's URLs for a few hundred tokens before deciding to crawl. max_pages (20), max_depth (2), include (URL substring), query (rank by relevance, stop when saturated), strategy=dfs for a manual or a paginated listing, schema or tables for rows from every page. For many rows pass out_file (.csv, .json, .jsonl): a path comes back instead of the pages. Returns the pages and a crawl_id; pass it back as crawl_id to resume an interrupted crawl. Links to a page's own site are paths, to be joined to that page's url.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL. Crawl stays on this domain. | |
| also | No | More sites to crawl alongside `url`, under the same page and token budget. Each domain gets an equal share of the pages, so one large site cannot spend the whole run. | |
| mode | No | Leave unset: auto learns the tier per domain. | |
| dedup | No | Skip near-duplicate pages, reporting what they duplicate instead of their content. Default true. | |
| query | No | Rank pages by relevance to these words, keep only the relevant blocks, and stop when new pages add nothing (see `stop_when_saturated`). | |
| output | No | Output shape: pages (default), llms.txt, llms-full.txt. | |
| robots | No | robots.txt policy: obey (default for crawling), warn, ignore. | |
| schema | No | Rows instead of prose from every page, as on web_fetch: "auto" or your own selectors. With `out_file`, one row per item, each carrying its page's `url`. | |
| scroll | No | Scroll every page until it stops growing before reading it (`"auto"` or a round count), for sites whose listings load as you scroll. Implies browser tiers; slower. | |
| tables | No | Every page's data tables as typed rows, as on web_fetch. | |
| include | No | Only follow URLs containing this substring (e.g. "/docs/"). | |
| timeout | No | Timeout in ms per page. Default 45000. | |
| crawl_id | No | Resume an interrupted crawl by the `crawl_id` it returned: queue, pages already fetched and parameters all come back. web_status lists the ones with work left. | |
| out_file | No | Write the pages to this file instead of returning them, as CSV, JSON or JSON Lines depending on the extension. What comes back is a path and a count. | |
| strategy | No | Ordering: best_first (default when a query is given), bfs, or dfs to follow one branch to its end before starting the next — what a manual or a paginated listing wants. | |
| max_depth | No | Link depth from the start URL. Default 2. | |
| max_pages | No | Maximum pages to fetch. Default 20, max 200. | |
| extraction | No | markdown (default), text, or html. | |
| max_duration_ms | No | Give up after this long regardless. Default 120000. | |
| max_tokens_total | No | Whole-crawl token cap. Default 60000. | |
| since_last_crawl | No | Seed from the site's sitemap and fetch only what its `lastmod` says has changed since this machine last read it. A page with no date is always fetched: silence is not "unchanged". | |
| max_chars_per_page | No | Per-page content cap in chars. Default 8000. | |
| stop_when_saturated | No | Stop once the crawl stops learning anything new about `query`. Default true when a query is given. |