crawl
Find pages on a site relevant to a query by following links from a start URL; returns matching URLs with snippets and respects robots.txt.
Instructions
Crawl a site from a start page, following the links most relevant to query first. Returns the best pages as URLs with one-line snippets (not full text); fetch them for details. Respects robots.txt.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page to start from. | |
| query | Yes | What you are looking for. Links are followed in order of relevance to it. | |
| keywords | No | Exact terms or synonyms that should match, to sharpen the query. | |
| max_depth | No | Maximum link hops from the start page (default 3). | |
| max_pages | No | Stop after this many pages (default 30, max 200). | |
| max_results | No | Maximum pages to return (default 10). | |
| other_sites | No | Also follow links to other sites (default false). |