check_robots
Evaluate robots.txt content offline to determine whether specified AI crawlers can access a URL path, using default or custom user-agent tokens.
Instructions
Evaluate robots.txt text (no network) for AI crawlers.
Args: robots_txt: full robots.txt content. path: URL path to test, e.g. "/" or "/blog/post". user_agents: crawler tokens to test; defaults to the built-in AI crawler list.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | / | |
| robots_txt | Yes | ||
| user_agents | No |