@readfetch/mcp-x402-proxy

readfetch_map

Read-only

Map a site's crawlable URLs without crawling it by parsing robots.txt, harvesting sitemap and sitemap-index URLs, detecting RSS/Atom/JSON feeds, and counting homepage links to plan scraping or crawling.

Instructions

Discover a site's crawlable surface without crawling it: parse robots.txt allow/disallow rules, collect URLs from sitemap.xml and sitemap indexes, find RSS, Atom and JSON feeds, and count internal and external links on the homepage. Use the returned URLs to plan a /v1/scrape or /v1/crawl.

Input Schema

Name	Required	Description
`url`	Yes	Public http(s) URL of the site to map.
`limit`	No	Maximum number of URLs to return (1-1000).
`max_bytes`	No	Abort if any single document exceeds this many bytes.
`timeout_s`	No	Total budget for the whole call in seconds; every sitemap fetch and the browser fallback share it. 95 s is the edge ceiling (Cloudflare closes at 100 s).

Output Schema

Name	Required	Description	Default
`url`	No
`host`	No
`urls`	No
`feeds`	No
`links`	No
`robots`	No
`status`	No
`sitemaps`	No
`url_count`	No
`status_code`	No
`total_found`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedv0.1.13Nov 1, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "feeds": {
+      "items": {
+        "format": "uri",
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "host": {
+      "type": "string"
+    },
+    "links": {
+      "properties": {
+        "external": {
+          "type": "integer"
+        },
+        "internal": {
+          "type": "integer"
+        }
+      },
+      "type": "object"
+    },
+    "robots": {
+      "properties": {
+        "allow": {
+          "items": {},
+          "type": "array"
+        },
+        "crawl_delay": {
+          "type": "null"
+        },
+        "disallow": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "sitemaps": {
+          "items": {
+            "format": "uri",
+            "type": "string"
+          },
+          "type": "array"
+        }
+      },
+      "type": "object"
+    },
+    "sitemaps": {
+      "items": {
+        "format": "uri",
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "status": {
+      "type": "string"
+    },
+    "status_code": {
+      "type": "integer"
+    },
+    "total_found": {
+      "type": "integer"
+    },
+    "url": {
+      "format": "uri",
+      "type": "string"
+    },
+    "url_count": {
+      "type": "integer"
+    },
+    "urls": {
+      "items": {
+        "format": "uri",
+        "type": "string"
+      },
+      "type": "array"
+    }
+  },
+  "type": "object"
+}

First observedv0.1.0Nov 1, 2026

TDQS

A4.2/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=true, and the description adds real value on top: it emphasizes no crawling occurs and enumerates exactly which documents are fetched, so the agent knows the cost profile. It omits failure behavior (site with no robots.txt or sitemap) and the browser-fallback cost noted only in the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the core capability and the 'without crawling it' qualifier, then the follow-up action. No filler or restatement of the tool name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and annotations cover the safety profile. The description covers what is collected and why, leaving only edge-case behavior (empty robots.txt, partial failures) unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so url, limit, max_bytes, and timeout_s are already fully documented with ranges and defaults. The description adds no parameter-level detail such as how limit interacts with the four discovery sources, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('discover') and resource ('a site's crawlable surface') and then enumerates the concrete sources parsed: robots.txt, sitemap.xml/indexes, RSS/Atom/JSON feeds, and homepage link counts. It is clearly differentiated from readfetch_crawl and readfetch_scrape by the phrase 'without crawling it'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The closing sentence explicitly routes the agent: 'Use the returned URLs to plan a /v1/scrape or /v1/crawl,' which positions this as the pre-flight step before those siblings. It stops short of stating when-not-to-use (e.g., a single known URL should go straight to readfetch_fetch), so it is not a full 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server