crawl
Crawl a website from a seed URL to retrieve content from many pages. Index documentation, wikis, and multi-page references efficiently.
Instructions
Crawl a site from a seed URL and return content from many pages. Use for indexing docs, wikis, multi-page references. Built for offline reuse: every page lands in the local cache.
Key parameters:
strategy: "bfs" (default) | "dfs" | "sitemap" (fastest for doc sites) | "map" (URL-only discovery).
max_depth: link-following depth (default 2).
max_pages: page cap (default 20).
include_patterns / exclude_patterns: regex filters on URLs.
max_tokens_out / include_full_markdown / citation_format: budget + shape controls.
Returns pages[] with title, evidence, depth. Content is deduplicated across pages (anchor-fragment aware). All pages are cached for later cache / find_similar queries.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Seed URL to start crawling from | |
| strategy | No | Crawl strategy: bfs (breadth-first), dfs (depth-first), sitemap (use sitemap.xml), map (URL-only discovery — returns list of URLs without content, faster than full crawl) | |
| use_auth | No | Use stored auth credentials (default: false) | |
| max_depth | No | Maximum link depth from seed (default: 2) | |
| max_pages | No | Maximum pages to crawl (default: 20) | |
| extract_links | No | Return link graph between pages (default: false) | |
| max_tokens_out | No | Token-budget cap on total output. Uses cl100k-base BPE; non-OpenAI tokenizer counts may drift ~5-15%. When both max_tokens_out and max_chars are set, max_tokens_out wins. | |
| citation_format | No | Citation rendering style. 'numbered' (default) inline [N] markers; 'json' returns a citations[] array; 'anthropic_tags' wraps sources in <source id='...'> tags. | |
| max_total_chars | No | Max total chars across all pages (default: 100000) | |
| exclude_patterns | No | URL regex blacklist — skip matching URLs | |
| include_patterns | No | URL regex whitelist — only crawl matching URLs | |
| include_full_markdown | No | Include full markdown body in the response. Default false on multi-result tools (returns evidence excerpts only); set true to restore. |