robots-txt-checker
Server Details
Checks if a URL path is allowed by robots.txt. x402 payment required (testnet USDC).
- Status
- Healthy
- Uptime
- 89.9% over 47 days
- Last Tested
- Transport
- Streamable HTTP · MCP 2025-11-25
- URL
TDQS
Scored across 1 tool
With only one tool exposed, there is no possibility of confusing it with another operation. The purpose is singular and clearly stated.
There is only one tool name, so there is no mixed convention or irregular verb pattern to flag. The name is descriptive and matches the server's purpose.
A single tool is minimal but appropriate for a narrowly scoped robots.txt checker. It is slightly below the typical 3-15 range, but the server is not over-scoped.
The server's entire purpose is to check a URL path against robots.txt, and the single tool fully covers that action. No additional operations are needed for this domain.
Available Tools
1 toolrobots_txt_checkerrobots_txt_checkerAInspect
Checks if a URL path is allowed by robots.txt. x402 payment required (Base mainnet USDC).
| Name | Required | Description | Default |
|---|---|---|---|
| checks | Yes | JSON-encoded strings, each an object like {"url":...,"path":...,"userAgent":...} | |
| payment | No | x402 payment payload (base64 JSON), signed EIP-712 exact/transferWithAuthorization. Omit on the first call — the tool will return payment requirements (accepts[]) as an error result. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries the behavioral burden. It discloses the charging mechanism, the required payment network, and the two-step flow where the first call returns accepts[] as an error result. This is valuable beyond the schema, though it does not detail the success response shape or robots.txt matching semantics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences with no filler. The core purpose is front-loaded, and the essential payment constraint follows immediately. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The input contract and payment handshake are covered by the schema and description together. However, with no output schema and no annotations, the description does not explain what a successful result looks like or how robots.txt matching rules are applied. This leaves some ambiguity for an agent planning to parse the tool's response.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters. The description adds the payment requirement context but does not need to redefine the parameters themselves. The baseline of 3 applies because the description neither helps nor hurts parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('checks') and names the exact resource ('robots.txt') and the question it answers ('is a URL path allowed'). It is clear and unambiguous, though there are no sibling tools to differentiate from, so it does not quite reach the top tier.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides concrete invocation guidance: x402 payment is required on Base mainnet USDC, and the caller must omit the payment parameter on the first call to receive payment requirements as an error result. It does not describe alternative tools, but no siblings exist.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
- First observed
robots_txt_checker
Related MCP Connectors
Test any x402 paid URL without paying: 402, PAYMENT-REQUIRED, USDC, amounts, payTo, EIP-712, CORS.
Whether an AI agent may fetch a URL, on what terms, and at what price. Free, no key, no signup.
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Pay-per-call website safety and URL checks for AI agents. USDC on Base via x402, no API keys.
261
Related MCP Servers
- FlicenseNot gradedqualityBmaintenanceEnables per-request paid AI analysis of public URLs, condition verification, and AI consultation via x402 micropayments on Base, with no account or subscription.6 npm-
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to autonomously pay for metered web APIs on a per-call basis, settling x402 v2 USDC micropayments via EIP-3009 authorization on Base Sepolia without human involvement. Spending is bounded by configurable per-call caps and host restrictions, and exposes tools to fetch paywalled URLs and inspect the agent wallet's address and balance.MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI assistants to call pay-per-call x402 endpoints on Base mainnet using the caller's own USDC wallet, covering single and batch token risk checks, the latest BTC radar signal, project info, and a free health probe. Payments are EIP-3009 signed with a configurable per-call spending cap, and any quote with an unexpected payee, network, or asset is refused before payment.MIT
- AlicenseNot gradedqualityCmaintenanceChecks token contract safety for honeypot, tax, proxy, blacklist, ownership risks, and returns a risk score, enabling rug-pull protection for agents via pay-per-call x402 micropayments.MIT
Glama MCP Gateway
Add one secure layer between your agents and this server.