crawl
Fetch multiple pages as markdown from a starting URL. Use include and exclude patterns to target specific links, ideal for catalog ingestion or building a site-wide RAG corpus from many pages at once.
Instructions
Bulk-fetch markdown across a site. Use this when an agent needs the contents of many pages at once (catalog ingestion, site-wide RAG corpus). Pair with includePatterns / excludePatterns to scope which URLs are fetched. Returns an array of pages each with markdown and per-page status.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL. | |
| depth | No | Link-hops from the starting URL to follow. | |
| country | No | Two-letter proxy egress country code. | us |
| maxPages | No | Hard cap on pages fetched. Defaults to 10. | |
| sessionId | No | Optional saved-browser-session ID for login-protected sites. | |
| useBrowser | No | Render each page in a headless browser (for SPAs). | |
| sessionName | No | Optional saved-browser-session name. | |
| excludePatterns | No | Glob/regex patterns. URLs matching any pattern are skipped. | |
| includePatterns | No | Glob/regex patterns. Only URLs matching at least one pattern are fetched. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| pages | Yes | ||
| durationMs | Yes | ||
| totalPages | Yes | ||
| completedPages | Yes |