trawl_crawl_web
Crawl a website and save its pages as Markdown for a client, with options to control crawl scope and maximum page count.
Instructions
Crawl a website and save pages as Markdown for a client.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| max_pages | No | Max pages to crawl (default: 100) | |
| pool_name | Yes | Pool name | |
| stay_in_path | No | Only crawl under starting URL path (default: true) |