extract_crawl
Crawl a website and extract structured fields (e.g., title, price) from every page using an LLM, with independent page processing and billing only for successful extractions.
Instructions
Crawl a domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.
Use this when the user wants structured FIELDS from MULTIPLE pages (e.g. extract title + price from every product page on a site).
⚠️ Each page takes 5–15 seconds. Default is 5 pages. For more than 20 pages, warn the user about wait times and credit usage before proceeding.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(provider, 'llm') → get live model list
Present models to user, ask them to choose one
Ask: 'Is this a JavaScript-heavy site?' → js_render (not available for extract_crawl, note this)
Confirm max_pages with the user
LLM keys can be omitted if set as environment variables.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain to crawl. | |
| schema | Yes | Dict mapping field names to description strings. Example: {"title": "string — the product name", "price": "number — price in USD"} | |
| selector | No | Optional CSS selector applied to every page before extraction. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. Advanced models (not mini/flash/haiku) use 30k char context — better for long pages. | |
| max_pages | No | Maximum pages to crawl and extract. Default: 5. Maximum: 50. Always confirm with user before setting above 20. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml. 'spider': follows links from root URL. | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | Yes | LLM provider. One of: 'openai', 'anthropic', 'gemini'. Call verify_provider_key first. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| extract_as_list | No | If true, extracts ALL matching items on each page as a JSON array. | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/products/'). |