TPC Hosting Free Tools

robots.txt tester

robots_txt_test

Read-onlyIdempotent

Fetches a site's robots.txt and tests whether a given crawler may fetch a given path, following RFC 9309. Also lists the Sitemap lines and parser warnings.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "robots_txt_test"
  }
}

Input Schema

Name	Required	Description
`url`	Yes	A website address such as https://example.com or just example.com. Only public sites on the standard ports can be checked.
`path`	No	Path (and query) to test, starting with /. Defaults to /.
`user_agent`	No	Crawler token such as Googlebot, Bingbot, GPTBot or ClaudeBot. Defaults to Googlebot.

Schema Changelog

Changes observed during successful MCP inspections.

First observedNov 11, 2026

TDQS

A3.8/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, openWorld and non-destructive, so the safety profile is covered. The description adds value beyond that by disclosing the evaluation standard (RFC 9309) and the extra outputs returned (Sitemap lines and parser warnings) — important because no output schema exists. It stops short of saying whether the raw robots.txt body is returned or how fetch failures/timeouts are reported.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, no filler. The primary operation is front-loaded and the secondary outputs are appended after, which matches how an agent scans a definition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description usefully enumerates what comes back (permission verdict, Sitemap lines, parser warnings) and cites the governing spec. What remains unstated — raw content inclusion, failure behavior for missing/unreachable robots.txt — is minor for a read-only diagnostic tool whose parameters are fully documented.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with per-parameter descriptions covering the URL format, the path requirement, and user-agent examples, so the baseline is 3. The description confirms the semantic relationship between the three parameters (crawler vs path vs site) but adds no format or edge-case detail beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ("Fetches a site's robots.txt") and adds a concrete operation ("tests whether a given crawler may fetch a given path") plus a secondary return (Sitemap lines, parser warnings). The crawler-permission framing separates it from generic fetch/status siblings like url_status_check or sitemap_check, though no sibling is named explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the purpose: an agent can infer this is for checking crawl permissions for a specific bot/path. However, there is no explicit when-to-use vs when-not, no mention of prerequisites, and no pointer to sitemap_check despite the overlap in Sitemap reporting. The RFC 9309 reference hints at standards-compliant evaluation but is not framed as guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All TPC Hosting Free Tools tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?