check_robots
Evaluate robots.txt content offline to determine whether specified AI crawlers can access a URL path, using default or custom user-agent tokens.
Instructions
Evaluate robots.txt text (no network) for AI crawlers.
Args: robots_txt: full robots.txt content. path: URL path to test, e.g. "/" or "/blog/post". user_agents: crawler tokens to test; defaults to the built-in AI crawler list.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"robots_txt": "<robots_txt>"
},
"name": "check_robots"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | / | |
| robots_txt | Yes | ||
| user_agents | No |