Octocrawl

Map a site

map

Read-onlyIdempotent

List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "map"
  }
}

Input Schema

Name	Required	Description
`url`	Yes	http(s) URL of the start page
`mode`	No	The declared identity robots.txt, the page and the sitemaps are read under. authed is not offered: a map reads public sitemaps and one public page.
`debug`	No	Return the full map response instead of the compact one.
`limit`	No	Links returned at most. Default 5000; a hosted server takes up to 5000. Reaching it is status completed with stoppedBy limit.
`search`	No	Keep only the URLs in which every word (at most 10) appears, case-insensitively, in the decoded URL or its title. A filter, not a ranking: the order stays the discovery order, and limit counts the matches.
`sitemap`	No	include (default): the start page's links and the sitemaps. skip: no sitemap is read. only: no page is read; the links are the sitemap entries in their listed order (the start URL only when a sitemap lists it).
`timeout`	No	Milliseconds for the whole map. Default 60000; a hosted server takes up to 60000.
`integration`	No	Your workflow label; recorded, not sent to sites.
`excludePaths`	No	Pathname regexes that leave a URL out; they win over includePaths.
`includePaths`	No	Pathname regexes a URL must match.
`regexOnFullURL`	No	Match includePaths and excludePaths against the canonical URL instead of its pathname. Default false.
`crawlEntireDomain`	No	Admit URLs anywhere on the start host, not only in the start URL's path subtree. Default false.
`includeSubdomains`	No	Admit every host under the start URL's apex (the host with one leading www. removed; no public-suffix list). Default false. Each new host's robots.txt is read, for at most 20 hosts.
`ignoreQueryParameters`	No	Fold URLs that differ only in their query string into the first one seen, returned without its query; each merge is counted (refused.collapsed, with samples under debug). Default false.
`deduplicateSimilarURLs`	No	Fold /a and /a/, / and /index.html, www and apex, http and https into one URL. Default true.

Output Schema

Name	Required	Description
`id`	Yes	The map's record id.
`links`	Yes
`counts`	No	returned, refused, and each refusal reason above zero, by name.
`status`	Yes
`warning`	No	The warnings' messages, joined.
`stoppedBy`	Yes
`agentHints`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedNov 4, 2026

addedOutput schema / properties / counts / additionalProperties
```
Added value: +{
+  "type": "integer"
+}
```

addedOutput schema / properties / counts / description

Added value: +"returned, refused, and each refusal reason above zero, by name."

changedOutput schema / properties / links / items / properties / robots / description

Previous value: -"The link's robots.txt verdict: every link with debug=true; in the compact answer only on a link robots.txt keeps out, returned under ignoreRobotsTxt."New value: +"The robots.txt verdict; in the compact answer only on a link it keeps out (ignoreRobotsTxt)."

Changed1 schema field changedNov 2, 2026

changedInput schema / properties / integration / description

Previous value: -"Your workflow label (printable, no spaces); recorded, not sent to sites."New value: +"Your workflow label; recorded, not sent to sites."

Changed1 schema field changedOct 18, 2026

changedInput schema / properties / includePaths / description

Previous value: -"Pathname regexes a URL must match (as on crawl)."New value: +"Pathname regexes a URL must match."

Changed2 schema fields changedOct 17, 2026

changedInput schema / properties / integration / description

Previous value: -"Your own label for the integration or workflow this request belongs to (1 to 100 printable characters, no spaces). Stored in Octocrawl's records (the scrape record, the task status), never sent to the target."New value: +"Your workflow label (printable, no spaces); recorded, not sent to sites."

changedOutput schema / properties / id / description

Previous value: -"The map's record id (GET /v1/maps/:id on the REST API)."New value: +"The map's record id."

Changed3 schema fields changedOct 17, 2026

removedInput schema / properties / ignoreRobotsTxt

Removed value: -{
-  "description": "Also return the URLs robots.txt disallows or whose robots.txt could not be read, each with that verdict (robots disallowed or unreachable), and read the start page and sitemaps past it. robots.txt is still read and recorded. Default false. A local server only; a hosted one refuses it.",
-  "type": "boolean"
-}

changedInput schema / properties / limit / maximum
```
Previous value: -100000New value: +5000
```
changedInput schema / properties / timeout / maximum
```
Previous value: -300000New value: +60000
```

First observedOct 15, 2026

TDQS

A4.1/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the read-only, open-world, idempotent annotations, the description discloses robots.txt handling, scope folding (host, www twin, subtree, asset exclusion), counting of omitted URLs, deadline behavior with partial results and stoppedBy timeout, and compact-vs-debug output. This is rich behavioral context that annotations alone do not carry.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but front-loads the purpose and key constraints, with no filler. Some long semicolon-heavy sentences could be clearer, but it is appropriately sized for a 15-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity, output schema, and rich annotations, the description covers the essential behavior: scope, sitemap handling, deadlines, and debug output. The nonexistent ignoreRobotsTxt reference is a notable blemish, but otherwise an agent has what it needs to call the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents all 15 parameters. However, the description references an 'ignoreRobotsTxt' setting that is not in the input schema, which actively misleads an agent trying to use that parameter; otherwise most parameter semantics are already in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: lists a site's URLs via the start page's HTTP links plus declared sitemap entries, without reading pages. It explicitly distinguishes scope and output from page-reading tools with 'Not for: reading pages' and 'No page title is fetched', so an agent can tell it apart from crawl/scrape siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides a clear when-not ('Not for: reading pages') and describes the sitemap modes that select different behaviors. It does not name sibling tools like crawl or scrape as the alternative, so the routing guidance is strong but not fully explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Octocrawl tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?