crawler_path
Read-onlyIdempotent
Where each crawler's path into a domain dies, as an observed data flow: edge decision, robots.txt as a file, robots.txt rules, then the page, JSON-LD, sitemap, llms.txt and entitymap.json it reached. Give agent (e.g. ClaudeBot, GPTBot, googlebot) for one identity's path and a one-line answer; omit it for every identity plus the stores and breaks. Uses the latest record on file, or scans first when there is none (fresh=true forces a scan). The answer-engine output is drawn but never measured.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| agent | No | Identity id or label, e.g. claudebot, GPTBot, Googlebot | |
| fresh | No | Scan now instead of using the record on file | |
| domain | Yes |