Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.
Scored across 3 tools
Each tool has a distinct role: access_check_url performs the live check, access_explain_verdicts is static interpretive documentation, and access_list_agents enumerates crawler tokens. They complement rather than overlap, though the explain/check pairing is somewhat meta and could be misselected by an agent expecting a fresh verdict.
All three names use a clean, consistent 'access_' prefix followed by a verb_noun structure (check_url, explain_verdicts, list_agents). The pattern is fully predictable across the set.
Three tools is slightly lean but well-matched to a narrow, focused purpose (verdict lookup for agent access). Each tool earns its place, though explain_verdicts is close to being reference material that could live in the description.
The domain—determining and interpreting whether an agent may fetch a URL—is well covered with checking, verdict interpretation, and agent-token listing. Minor gaps like batch URL checks or a directory of known charged sites exist but are not essential to the core workflow.
Given a URL, say whether an AI agent may fetch it, on what terms, and at what price. Reads robots.txt for a generic agent and for every AI user-agent the operators document, the Content-signal and Content-Usage directives, any RSL licence the origin links, and the live response to an honestly-identified request.
Verdicts: allowed (it served us), allowed_if_identified (the written policy permits
this but bot management refused an anonymous request — signing as a named crawler is the
documented route), charged (there is a price, quoted where the rail will show one),
blocked (the site disallows the agents that would want this, with nothing to buy), and
unknown (the origin did not answer, or the URL is not there).
Never evades: no CAPTCHA solving, no proxies, no spoofed user-agents. Refusals come back as findings with the evidence attached.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute http(s) URL of the page to check, e.g. https://example.com/article | |
| skipWellKnown | No | Skip the five well-known probes; faster, and they are almost always absent | |
| includeEvidence | No | Include every request made, with status and headers (default false: it is large) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond readOnlyHint/openWorldHint/idempotentHint by disclosing how the check is performed (robots.txt, Content-signal/Content-Usage, RSL licences, a live honestly-identified request) and the operating stance: no CAPTCHA solving, no proxies, no spoofed user-agents. It also states that refusals surface as findings with evidence rather than errors, which is exactly the behavioral context an agent needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads the one-line purpose, then mechanism, then the full verdict space, then the non-evasion guarantee. The long parenthetical verdict definitions are justified because no output schema exists — each sentence carries information the caller cannot get elsewhere.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description must convey the result space itself, and it enumerates all five verdicts with their meanings and the conditions under which each arises. Combined with the annotations covering the safety profile and 100% parameter coverage, an agent has everything needed to invoke and interpret this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters (url, skipWellKnown, includeEvidence) are already documented in the schema. The description adds no syntax or trade-off detail for skipWellKnown or includeEvidence (e.g. what 'five well-known probes' means for verdict confidence), so it does not exceed the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a precise verb+resource: 'Given a URL, say whether an AI agent may fetch it, on what terms, and at what price.' The enumerated verdict vocabulary ('allowed', 'charged', 'blocked', 'unknown') makes the scope unmistakable. It never names the sibling tools (access_explain_verdicts, access_list_agents), so the differentiation is only implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: the reader infers this is the pre-fetch permission check, and 'Refusals come back as findings' hints at how to interpret output. There is no explicit when-to-use, when-not-to-use, or pointer to access_explain_verdicts when the caller needs the reasoning behind a verdict.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the five verdicts this index can return, what each one means for an agent that has just been refused, and what the agent's next move is for each. Costs nothing and makes no request; call it once to interpret access_check_url results.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and closed-world. The description adds value beyond them by stating 'Costs nothing and makes no request' and describing the return content (five verdicts, meanings, next moves), which tells the agent this is a free static lookup with no side effects or latency. It does not elaborate on caching or exact output format, keeping it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with zero filler; the core purpose is front-loaded and the cost/latency note follows immediately. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, no-output-schema explainer, the description gives everything an agent needs: what it returns, when to call it, and that it is free and side-effect-free. No return schema exists, but the description adequately characterizes the output ('five verdicts, what each means, next move').
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4 per the rubric. No parameter semantics are needed, and the description does not need to compensate for any schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('Return') and resource ('the five verdicts this index can return'), and distinguishes itself from siblings by being the explainer for access_check_url results. An agent can immediately tell this is a static reference tool, not another access check.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use it ('call it once to interpret access_check_url results') and frames the audience ('an agent that has just been refused'). It effectively names the related sibling access_check_url but does not state when not to use it or any exclusion; still, the guidance is clear enough for correct invocation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the AI crawler tokens this index reads robots.txt rules for — only tokens their operators document — plus the widely-written tokens that no operator sends, which make a robots rule unenforceable. Useful when writing or auditing a robots.txt.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint=false and idempotentHint, so the safety profile is covered; the description goes beyond them by disclosing the composition of the result set (operator-documented tokens plus widely-written tokens no operator sends, which make rules unenforceable). It does not state whether the list is exhaustive/static or how results are ordered, which would be useful for a snapshot-style tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no filler: the return scope leads, the practical use case follows. The em-dash aside about unenforceable rules is dense but load-bearing, not padding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description carries the burden of explaining what comes back, and it does describe the semantic content of the set well. It does not describe the shape of each entry (bare token string vs. token plus operator/provenance), which is the one thing an agent would still have to discover by calling it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so per the rubric the baseline is 4 and there is nothing for the description to disambiguate. It correctly avoids inventing parameter talk, though it also cannot add value here.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('Return') and a tightly scoped resource ('the AI crawler tokens this index reads robots.txt rules for'), and it further specifies the two categories returned, which is unusually precise. It does not name or contrast with the siblings access_check_url or access_explain_verdicts, but the list-vs-check-vs-explain distinction is self-evident from the resource itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Useful when writing or auditing a robots.txt' gives a concrete, actionable context of use rather than vague hand-waving. It stops short of stating when not to use it or pointing to a sibling (e.g. 'for a single URL's verdict, use access_check_url'), so it is clear-but-unbounded rather than exhaustive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
access_check_urlaccess_explain_verdictsaccess_list_agentsAsk before you fetch: will this domain serve your crawler, refuse it, or charge it?
Free public web freshness and response-metadata checks for AI agents.
Scan any URL for AI agent readability — Vercel Spec, llmstxt.org, and agent-protocol manifests.
Fetch any URL as clean Markdown or metadata, and buy digital goods via x402 — for AI agents.
