Crawl a set of pages under an allowlist
domscout_crawl_siteCrawl a website from a seed URL to extract page content as Markdown, respecting robots.txt and configurable depth, page limits, and URL filters.
Instructions
Costs 1 credit per page crawled — submitting the crawl itself is free. Returns a job ID immediately — poll it with domscout_get_job. Always respects robots.txt (this cannot be disabled), caps at 500 pages and depth 5, and requires an explicit HTTPS allowedOrigins list. Requires a Business plan or above.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| seedUrl | Yes | Where to start. Must be https. | |
| maxDepth | No | ||
| maxPages | No | ||
| allowedOrigins | Yes | Bare https origins the crawl may visit, e.g. "https://example.com". Must include the seed origin. | |
| excludePatterns | No | ||
| extractMarkdown | No | Capture each page as Markdown. | |
| includePatterns | No |