Observed crawler path on a site
crawler_pathWhere each crawler's path into a domain dies, as an observed data flow: edge decision, robots.txt as a file, robots.txt rules, then the page, JSON-LD, sitemap, llms.txt and entitymap.json it reached. Give agent (e.g. ClaudeBot, GPTBot, googlebot) for one identity's path and a one-line answer; omit it for every identity plus the stores and breaks. Uses the latest record on file, or scans first when there is none (fresh=true forces a scan). The answer-engine output is drawn but never measured.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"domain": "<domain>"
},
"name": "crawler_path"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| agent | No | Identity id or label, e.g. claudebot, GPTBot, Googlebot | |
| fresh | No | Scan now instead of using the record on file | |
| domain | Yes |