crawl_site
Crawl a website, chunk all pages, and return structured JSON chunks. Supports sitemap (for docs/blogs) and spider (follows links) modes.
Instructions
Crawl an entire website, chunk all pages, and return structured JSON chunks. Two modes: 'sitemap' (reads sitemap.xml — best for docs/blogs) and 'spider' (follows links — works on any site).
Use this when the user wants chunks from MULTIPLE pages WITHOUT extracting structured fields. For structured field extraction across pages, use extract_crawl.
⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 10 pages. For large sites, warn about credit usage first.
If contextual_retrieval is requested, follow the PRE-FLIGHT sequence:
Call verify_provider_key(provider, 'llm') → get live model list
Ask user to choose a model
Ask about JS rendering
Present Contextual Retrieval as a recommended upgrade
LLM keys can be omitted if set as environment variables.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain or sitemap URL to crawl. | |
| selector | No | Optional CSS selector applied to every crawled page. | |
| js_render | No | Use headless browser to render JS before scraping each page. Ask the user before enabling. Adds surcharge per page. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| max_pages | No | Maximum pages to crawl. Default: 10. Maximum: 200. Always confirm with user for large sites. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site). | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key first. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/docs/'). | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment. Present as a recommended upgrade. Requires llm_provider and llm_model from verify_provider_key. |