Octocrawl

Map a site

map

Read-onlyIdempotent

List a site's URLs by gathering start-page links and declared sitemap entries without reading page bodies, within scope, robots.txt, and timeout limits.

Instructions

List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.

Input Schema

Name	Required	Description
`url`	Yes	http(s) URL of the start page
`mode`	No	The declared identity robots.txt, the page and the sitemaps are read under. authed is not offered: a map reads public sitemaps and one public page.
`debug`	No	Return the full map response instead of the compact one.
`limit`	No	Links returned at most. Default 5000; a hosted server takes up to 5000. Reaching it is status completed with stoppedBy limit.
`search`	No	Keep only the URLs in which every word (at most 10) appears, case-insensitively, in the decoded URL or its title. A filter, not a ranking: the order stays the discovery order, and limit counts the matches.
`sitemap`	No	include (default): the start page's links and the sitemaps. skip: no sitemap is read. only: no page is read; the links are the sitemap entries in their listed order (the start URL only when a sitemap lists it).
`timeout`	No	Milliseconds for the whole map. Default 60000; a hosted server takes up to 60000.
`integration`	No	Your workflow label (printable, no spaces); recorded, not sent to sites.
`excludePaths`	No	Pathname regexes that leave a URL out; they win over includePaths.
`includePaths`	No	Pathname regexes a URL must match (as on crawl).
`regexOnFullURL`	No	Match includePaths and excludePaths against the canonical URL instead of its pathname. Default false.
`ignoreRobotsTxt`	No	Also return the URLs robots.txt disallows or whose robots.txt could not be read, each with that verdict (robots disallowed or unreachable), and read the start page and sitemaps past it. robots.txt is still read and recorded. Default false. A local server only; a hosted one refuses it.
`crawlEntireDomain`	No	Admit URLs anywhere on the start host, not only in the start URL's path subtree. Default false.
`includeSubdomains`	No	Admit every host under the start URL's apex (the host with one leading www. removed; no public-suffix list). Default false. Each new host's robots.txt is read, for at most 20 hosts.
`ignoreQueryParameters`	No	Fold URLs that differ only in their query string into the first one seen, returned without its query; each merge is counted (refused.collapsed, with samples under debug). Default false.
`deduplicateSimilarURLs`	No	Fold /a and /a/, / and /index.html, www and apex, http and https into one URL. Default true.

Output Schema

Name	Required	Description
`id`	Yes	The map's record id.
`links`	Yes
`counts`	No
`status`	Yes
`warning`	No	The warnings' messages, joined.
`stoppedBy`	Yes
`agentHints`	No

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.7.0Oct 18, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations (readOnly/openWorld/idempotent/non-destructive) by disclosing deadline semantics (partial results, stoppedBy timeout), exactly what is folded or counted, robots.txt handling, and the compact-vs-debug output difference. This is the kind of behavioral context annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and tightly packed with no filler sentences; every clause carries a distinct fact. The multi-clause sentences in the middle are dense enough to demand a re-read, which keeps it just short of a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 16-parameter tool with an output schema already covering the return shape, the description supplies the missing operational context: scope rules, deadline behavior, filtering, and output verbosity. An agent has everything needed to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all 16 parameters and the baseline is 3. The description reinforces the behaviorally significant ones (ignoreRobotsTxt, debug/compact default, the deadline interaction with timeout), adding coupling context rather than mere restatement.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('List a site's URLs') and immediately bounds scope: start page links plus declared sitemap entries, read without fetching page bodies. It also excludes itself from sibling tasks ('Not for: reading pages'), so an agent can separate it from scrape/crawl without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear exclusion ('Not for: reading pages, as one page body is read at most') and states conditional behavior (robots.txt applies unless ignoreRobotsTxt is set), which guides tool selection. It stops short of naming the alternative sibling tools (scrape/crawl) explicitly, leaving that routing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.