crawl_project
Crawl a project's internal pages using breadth-first link discovery, bounded by configurable page and depth limits. Returns discovered URLs and uncrawled page count, with optional meta data extraction and Markdown storage.
Instructions
Discover a project's pages by breadth-first crawling internal links from its base URL, bounded by max_pages/max_depth (overridable per call). Discovery is deterministic (links sorted before the cap) and sitemap-seeded when a sitemap.xml/robots.txt Sitemap is present, so the same site yields the same page subset every run. Returns the discovered URLs plus pages_not_crawled[] (≤100, else a count) so a hit cap is never silent. Runs in the project's authenticated context. Set meta=true to also get each page's title + meta description (captured during the crawl, so behind-login/JS pages work), and save_md=true to save every crawled page as readable Markdown to data/fetches// (returns pages[] with saved_path + saved_dir). Discovery (optionally + capture) — use test_project to crawl and run full audits.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| meta | No | Also return each crawled page's title + meta description in pages[] (default: false). | |
| project | Yes | Project name | |
| save_md | No | Save each crawled page as readable Markdown to data/fetches/<project>/ (default: false). Returns saved_dir + per-page saved_path. | |
| save_dir | No | Directory to save Markdown into (implies save_md=true). Defaults to data/fetches/<project>/. | |
| max_depth | No | Override max depth for this crawl | |
| max_pages | No | Override max pages for this crawl. 0 = the WHOLE site (unbounded, stops at a safety ceiling of 2000 and flags ceiling_hit). | |
| use_sitemap | No | Seed discovery from sitemap.xml / robots.txt Sitemap: lines when present (default: true). Set false for pure link-crawl. |