Octocrawl

Start a crawl

crawl

Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery. On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.

Input Schema

Name	Required	Description
`url`	Yes
`mode`	No	The declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode.
`mobile`	No
`verify`	No	Same as scrape.
`formats`	No	What to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). On an account's credits a browser page costs 5; with fewer left, a screenshot is refused.
`headers`	No
`sitemap`	No	include (default): also queue the site's sitemap URLs. skip: no sitemap. only: the start URL and the sitemap entries.
`timeout`	No
`waitFor`	No
`blockAds`	No
`fastMode`	No
`maxDepth`	No	Links followed from the start URL at most. Omitted or null: no limit.
`maxPages`	No	Pages fetched at most. Omitted: 100. null: the server's default (none on a local server; hosted servers cap it).
`maxCredits`	No	The most credits this job may spend; it stops when the next page would pass it.
`excludeTags`	No
`includeTags`	No
`integration`	No	Your workflow label (printable, no spaces); recorded, not sent to sites.
`excludePaths`	No	Pathname regexes that skip a discovered link; they win over includePaths.
`includeLinks`	No
`includePaths`	No	Pathname regexes a discovered link must match; the start URL is always fetched.
`maxFileBytes`	No
`idempotencyKey`	No	Makes a retry safe: the same key and arguments return the first taskId. Kept 24 hours.
`maxConcurrency`	No	Pages fetched at once, at most (only lowers the worker count).
`regexOnFullURL`	No	Match includePaths and excludePaths against each link's canonical URL instead of its pathname. Default false.
`allowSubdomains`	No	Follow links to every host under the start URL's apex. Default false.
`onlyMainContent`	No
`crawlEntireDomain`	No	Follow links anywhere on the start host, not only in the start path's subtree. Default false.
`allowExternalLinks`	No	Follow links to any host, each with its own robots.txt; maxDepth and maxPages bound the walk. Default false. Not with allowlistedDomains.
`allowlistedDomains`	No
`removeBase64Images`	No
`ignoreQueryParameters`	No	Treat URLs that differ only in their query as one page. Default false.
`deduplicateSimilarURLs`	No	Treat /a and /a/, www and apex, http and https as one page. Default true.

Output Schema

Name	Required	Description	Default
`taskId`	Yes
`replayed`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedOct 28, 2026
- addedInput schema / properties / maxConcurrency / maximum
```
Added value: +2
```

Changed1 schema field changedOct 21, 2026

addedInput schema / properties / maxCredits

Added value: +{
+  "description": "The most credits this job may spend; it stops when the next page would pass it.",
+  "minimum": 1,
+  "type": "integer"
+}

Changed4 schema fields changedOct 20, 2026

changedInput schema / properties / formats / description

Previous value: -"Same as scrape."New value: +"What to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). On an account's credits a browser page costs 5; with fewer left, a screenshot is refused."

addedInput schema / properties / formats / items

Added value: +{
+  "anyOf": [
+    {
+      "enum": [
+        "markdown",
+        "links",
+        "json",
+        "html",
+        "rawHtml",
+        "images",
+        "tables",
+        "screenshot",
+        "screenshot@fullPage"
+      ],
+      "type": "string"
+    },
+    {
+      "additionalProperties": false,
+      "properties": {
+        "modelFallback": {
+          "type": "boolean"
+        },
+        "prompt": {
+          "maxLength": 4000,
+          "type": "string"
+        },
+        "schema": {
+          "type": "object"
+        },
+        "type": {
+          "const": "json"
+        }
+      },
+      "required": [
+        "type",
+        "schema"
+      ],
+      "type": "object"
+    },
+    {
+      "additionalProperties": false,
+      "properties": {
+        "selectors": {
+          "items": {
+            "additionalProperties": false,
+            "properties": {
+              "attribute": {
+                "maxLength": 100,
+                "minLength": 1,
+                "pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$",
+                "type": "string"
+              },
+              "selector": {
+                "maxLength": 200,
+                "minLength": 1,
+                "type": "string"
+              }
+            },
+            "required": [
+              "selector",
+              "attribute"
+            ],
+            "type": "object"
+          },
+          "maxItems": 50,
+          "minItems": 1,
+          "type": "array"
+        },
+        "type": {
+          "const": "attributes"
+        }
+      },
+      "required": [
+        "type",
+        "selectors"
+      ],
+      "type": "object"
+    },
+    {
+      "additionalProperties": false,
+      "properties": {
+        "fields": {
+          "items": {
+            "additionalProperties": false,
+            "properties": {
+              "attribute": {
+                "pattern": "^[A-Za-z_][A-Za-z0-9_:.-]*$",
+                "type": "string"
+              },
+              "name": {
+                "maxLength": 64,
+                "minLength": 1,
+                "type": "string"
+              },
+              "selector": {
+                "maxLength": 200,
+                "minLength": 1,
+                "type": "string"
+              }
+            },
+            "required": [
+              "name"
+            ],
+            "type": "object"
+          },
+          "maxItems": 50,
+          "minItems": 1,
+          "type": "array"
+        },
+        "itemSelector": {
+          "maxLength": 200,
+          "minLength": 1,
+          "type": "string"
+        },
+        "type": {
+          "const": "list"
+        }
+      },
+      "required": [
+        "type"
+      ],
+      "type": "object"
+    },
+    {
+      "additionalProperties": false,
+      "properties": {
+        "fullPage": {
+          "type": "boolean"
+        },
+        "quality": {
+          "maximum": 100,
+          "minimum": 1,
+          "type": "integer"
+        },
+        "type": {
+          "const": "screenshot"
+        },
+        "viewport": {
+          "additionalProperties": false,
+          "properties": {
+            "height": {
+              "maximum": 1080,
+              "minimum": 240,
+              "type": "integer"
+            },
+            "width": {
+              "maximum": 1920,
+              "minimum": 320,
+              "type": "integer"
+            }
+          },
+          "required": [
+            "width",
+            "height"
+          ],
+          "type": "object"
+        }
+      },
+      "required": [
+        "type"
+      ],
+      "type": "object"
+    }
+  ]
+}

changedInput schema / properties / maxFileBytes / maximum
```
Previous value: -524288000New value: +5242880
```

changedInput schema / properties / mode / description

Previous value: -"standard (default) or research; for signed-in pages send a batch in mode authed."New value: +"The declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode."

AddedOct 20, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only cover safety and openness; the description adds substantive behavior they cannot carry: slice limits (500 pages per crawl, 1,000 URLs per batch), per-page credit cost against maxCredits, 7-day result retention, and that collapsed/refused links are still counted in discovery. That is exactly the async-job context an agent needs before starting a paid job.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the crawl scope and return value, then limits and costs — good ordering with almost no filler. Some sentences pack several distinct facts ('Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery') and would read better split.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 32-parameter asynchronous job tool, it covers the return shape, the follow-up tools, cost and cap behavior, dedup semantics and retention. The output schema carries the result structure, so the description only needs to explain the task lifecycle, which it does.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 32 params at 56% schema coverage, the description earns credit for stating the maxPages default of 100, the maxDepth link-traversal meaning, and for delegating the page-level options to scrape's documented parameters rather than restating them. It leaves the traversal-policy cluster (allowExternalLinks vs allowlistedDomains, crawlEntireDomain, allowSubdomains) to the schema, which is a real but moderate gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource plus crawl scope: 'follow links in the start path's subtree on its host and www twin'. That scope statement alone separates it from scrape (single page) and map (URL discovery) without needing their schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It lays out the full pipeline — this returns { taskId }, then wait_crawl, then get_crawl_pages — which tells the agent exactly what to call next. It stops short of an explicit exclusion ('use scrape for a single page, map for URL lists only'), so the when-not case is inferred rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All Octocrawl tools Inspect

Try in Browser

Context Every Agent Can ShareAgents share what they learn, hand off work, and keep it when you switch models. Start with 5 GB free.fulcradynamics.comAd

Glama MCP Gateway

Add one secure layer between your agents and this server.