FreeTier Agent API

robots-check

robots_check

Read-only

3 free MCP calls per day (all tools together), then $0.003 USDC via x402 or card credits (paid in-band: params._meta["x402/payment"]). Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps. When to use: Use robots-check before crawling a URL with your own user-agent.

Input Schema

Name	Required	Description	Default
`ua`	No	user-agent token, default *
`url`	Yes	absolute http(s) URL

Output Schema

Name	Required	Description	Default
`url`	No
`allowed`	No
`sources`	No
`sitemaps`	No
`user_agent`	No
`crawl_delay`	No
`matched_rule`	No
`robots_status`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 26, 2026

changedOutput schema / description

Previous value: -"Inferred from a real response captured on 2026-10-24; extra fields may appear, null-valued fields are untyped."New value: +"Inferred from a real response captured on 2026-10-26; extra fields may appear, null-valued fields are untyped."

Changed1 schema field changedOct 24, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "description": "Inferred from a real response captured on 2026-10-24; extra fields may appear, null-valued fields are untyped.",
+  "properties": {
+    "allowed": {
+      "type": [
+        "boolean",
+        "null"
+      ]
+    },
+    "crawl_delay": {},
+    "matched_rule": {},
+    "robots_status": {
+      "type": [
+        "number",
+        "null"
+      ]
+    },
+    "sitemaps": {
+      "type": [
+        "array",
+        "null"
+      ]
+    },
+    "sources": {
+      "items": {
+        "properties": {
+          "name": {
+            "type": [
+              "string",
+              "null"
+            ]
+          },
+          "url": {
+            "type": [
+              "string",
+              "null"
+            ]
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "url": {
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "user_agent": {
+      "type": [
+        "string",
+        "null"
+      ]
+    }
+  },
+  "type": "object"
+}

First observedOct 18, 2026

TDQS

A3.9/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly and openWorld, but the description adds material context they do not: a free-tier quota of 3 calls per day across all tools, and the paid fallback via x402 or card credits injected through params._meta. The return value fields are also disclosed, though rate-limit behavior after payment and failure modes for unreachable robots.txt are not.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every sentence carries useful information: pricing, capability, return values, and usage condition. The commercial preamble is front-loaded ahead of the purpose, which slightly delays the core statement but is justified for cost-aware agents.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With annotations covering safety and an output schema covering return shape, the remaining gaps are small. An agent knows the cost model, the inputs, the outputs, and when to invoke it; only edge-case behavior such as missing or unreachable robots.txt is left unstated.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with both url and ua documented in the schema, so the schema carries the parameter burden. The description only restates the user-agent concept ('for a given user-agent') without adding format or default guidance.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb and resource: checks whether a URL may be crawled per robots.txt for a given user-agent, and lists the return fields (allowed, matched rule, crawl-delay, sitemaps). It is clearly distinct from URL-fetching siblings like page_check or link_check, though it does not say so explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use it: before crawling a URL with your own user-agent. It gives clear context but no exclusions or named alternatives for cases where robots enforcement is not needed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All FreeTier Agent API tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?