Octocrawl

Map a site

map

Read-onlyIdempotent

List a site's URLs without reading each page: the start page's links (read over HTTP alone) and the entries of the sitemaps the site declares, within one deadline. URLs stay in scope (the start host and its www twin, the start path's subtree, assets left out, similar URLs folded) and robots.txt applies unless ignoreRobotsTxt is set; what was left out is counted. No page title is fetched. At the deadline the answer is what was found (partial, stoppedBy timeout). Compact by default; debug adds each link's evidence and the sources read. Not for: reading pages, as one page body is read at most.

Input Schema

Name	Required	Description
`url`	Yes	http(s) URL of the start page
`mode`	No	The declared identity robots.txt, the page and the sitemaps are read under. authed is not offered: a map reads public sitemaps and one public page.
`debug`	No	Return the full map response instead of the compact one.
`limit`	No	Links returned at most. Default 5000; a hosted server takes up to 5000. Reaching it is status completed with stoppedBy limit.
`search`	No	Keep only the URLs in which every word (at most 10) appears, case-insensitively, in the decoded URL or its title. A filter, not a ranking: the order stays the discovery order, and limit counts the matches.
`sitemap`	No	include (default): the start page's links and the sitemaps. skip: no sitemap is read. only: no page is read; the links are the sitemap entries in their listed order (the start URL only when a sitemap lists it).
`timeout`	No	Milliseconds for the whole map. Default 60000; a hosted server takes up to 60000.
`integration`	No	Your workflow label (printable, no spaces); recorded, not sent to sites.
`excludePaths`	No	Pathname regexes that leave a URL out; they win over includePaths.
`includePaths`	No	Pathname regexes a URL must match.
`regexOnFullURL`	No	Match includePaths and excludePaths against the canonical URL instead of its pathname. Default false.
`crawlEntireDomain`	No	Admit URLs anywhere on the start host, not only in the start URL's path subtree. Default false.
`includeSubdomains`	No	Admit every host under the start URL's apex (the host with one leading www. removed; no public-suffix list). Default false. Each new host's robots.txt is read, for at most 20 hosts.
`ignoreQueryParameters`	No	Fold URLs that differ only in their query string into the first one seen, returned without its query; each merge is counted (refused.collapsed, with samples under debug). Default false.
`deduplicateSimilarURLs`	No	Fold /a and /a/, / and /index.html, www and apex, http and https into one URL. Default true.

Output Schema

Name	Required	Description
`id`	Yes	The map's record id.
`links`	Yes
`counts`	No
`status`	Yes
`warning`	No	The warnings' messages, joined.
`stoppedBy`	Yes
`agentHints`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed6 schema fields changedOct 18, 2026

removedInput schema / properties / ignoreRobotsTxt

Removed value: -{
-  "description": "Also return the URLs robots.txt disallows or whose robots.txt could not be read, each with that verdict (robots disallowed or unreachable), and read the start page and sitemaps past it. robots.txt is still read and recorded. Default false. A local server only; a hosted one refuses it.",
-  "type": "boolean"
-}

changedInput schema / properties / includePaths / description

Previous value: -"Pathname regexes a URL must match (as on crawl)."New value: +"Pathname regexes a URL must match."

changedInput schema / properties / integration / description

Previous value: -"Your own label for the integration or workflow this request belongs to (1 to 100 printable characters, no spaces). Stored in Octocrawl's records (the scrape record, the task status), never sent to the target."New value: +"Your workflow label (printable, no spaces); recorded, not sent to sites."

changedInput schema / properties / limit / maximum
```
Previous value: -100000New value: +5000
```
changedInput schema / properties / timeout / maximum
```
Previous value: -300000New value: +60000
```

changedOutput schema / properties / id / description

Previous value: -"The map's record id (GET /v1/maps/:id on the REST API)."New value: +"The map's record id."

First observedOct 9, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, and the description adds substantial behavior beyond them: robots.txt compliance and its override, URL scope rules (host, www twin, subtree, assets excluded, folding), the deadline/partial-result contract (stoppedBy timeout), no title fetching, and compact-vs-debug output modes. This is rich, non-obvious context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core purpose and scope lead the paragraph and every clause carries information about behavior, but it is extremely dense and parenthetical, which taxes readability. Efficient rather than padded, though a structural break would help.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter, open-world crawler with an output schema already present, the description covers everything an agent needs to call correctly: scope, robots handling, deadline semantics, partial-result behavior, and output verbosity. Return-value detail is properly left to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description reinforces and clarifies semantics: it defines what mode governs (the declared identity), what happens at limit/timeout (status completed, stoppedBy), and folding/counting behavior (refused.collapsed, deduplicateSimilarURLs). It adds value beyond the schema text rather than merely restating it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource combo up front: 'List a site's URLs without reading each page.' It also scopes exactly what is listed (start page's links over HTTP plus declared sitemap entries) and names the boundary against its sibling with 'Not for: reading pages.' An agent can distinguish this from scrape without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives a clear negative use case ('Not for: reading pages, as one page body is read at most') which directly routes to scrape, plus scope conditions (robots.txt applies unless ignoreRobotsTxt is set). It lacks an explicit 'use this when you need X' positive trigger, but the exclusion is strong and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Octocrawl tools Inspect

Try in Browser

Context Every Agent Can ShareAgents share what they learn, hand off work, and keep it when you switch models. Start with 5 GB free.fulcradynamics.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.