spidra_crawl_report
Explain a finished crawl to see how instructions were interpreted, how many pages were read, and which links were skipped by reason. Use it to debug missing pages and adjust crawl settings.
Instructions
Explain a finished crawl: how it read the instruction, how many pages it read only to find the wanted ones (free), and every link it didn't follow, counted by reason. Pass "reason" to list the URLs for one reason, each with the page it was found on.
Use it when a page you expected is missing, then adjust crawlInstruction, includePaths or maxPages and crawl again. Reasons: not_relevant, outside_domain, include_pattern, exclude_pattern, file_type, other_version_or_language, duplicate_page, redirected_off_site, robots_txt, action_link. The report is empty while the crawl runs.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| jobId | Yes | The crawl's jobId | |
| limit | No | Skipped URLs to return (default 200) | |
| offset | No | Skipped URLs to skip, for paging | |
| reason | No | List skipped URLs for this reason only, e.g. "not_relevant" |