Hosted Octocrawl: scrape public web pages into Markdown and map a site's URLs, each result with an Evidence Record (final URL, HTTP status, robots.txt decision, content hashes). Blocked or empty pages come back with the reason. No account needed: 20 pages a day per address without a key.
Scored across 2 tools
The two tools have clearly separate purposes: map discovers URLs without reading pages, while scrape reads exactly one URL as Markdown. Both descriptions explicitly state what they are not for, making misselection unlikely.
Both names are short imperative verbs in a consistent lowercase style, but they do not follow a shared verb_noun pattern. The naming is readable and predictable enough for an agent.
Two tools are slightly thin under the usual 3-15 range, but each earns its place for this focused crawler: discovery and single-page reading are distinct core operations. The surface is well-scoped rather than padded.
The pair covers URL discovery and one-page extraction, with robots and sitemap behavior handled in map. A batch-read or multi-URL scrape operation is absent, but agents can work around it by looping scrape calls.
List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.
Inputs: url (required), mode, debug, limit, +11 more. Returns: id, links, counts, status, +3 more
Read one URL as Markdown with its Evidence Record (final URL, fetch time, HTTP status, robots.txt decision, hashes), over HTTP first and in the browser when the page needs it. Not for: many URLs or a whole site section. Compact by default; debug adds the full audit. warnings name what the content cannot vouch for (client_rendered_suspected, robots_overridden), and agentHints the next step.
Inputs: url (required), mode, debug, mobile, +15 more. Returns: status, finalUrl, markdown, readiness, +8 more
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Add one secure layer between your agents and this server.