lilbee

Overview Schema Source Related Servers Discussions Admin

crawl

Start a non-blocking site scan that follows links by depth, then poll crawl_status(task_id) for progress and results.

Instructions

Start a non-blocking crawl; poll via crawl_status(task_id). depth=0 (default) = single URL, N = follow links N levels, null = whole site. render_mode: "http"/"browser".

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes
`depth`	No
`max_pages`	No
`render_mode`	No	How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.
`include_subdomains`	No

Output Schema

TableJSON Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedv1.0.16Oct 20, 2026

removedInput schema / $defs

Removed value: -{
-  "CrawlRenderMode": {
-    "description": "How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.",
-    "enum": [
-      "http",
-      "browser"
-    ],
-    "title": "CrawlRenderMode",
-    "type": "string"
-  }
-}

removedInput schema / properties / render_mode / $ref
```
Removed value: -"#/$defs/CrawlRenderMode"
```

addedInput schema / properties / render_mode / description

Added value: +"How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS."

addedInput schema / properties / render_mode / enum
```
Added value: +[
+  "http",
+  "browser"
+]
```
addedInput schema / properties / render_mode / type
```
Added value: +"string"
```

Changed14 schema fields changedv1.0.2Jul 22, 2026
- removedInput schema / properties / depth / anyOf
```
Removed value: -[
-  {
-    "type": "integer"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / depth / default
```
Removed value: -null
```
- removedInput schema / properties / depth / title
```
Removed value: -"Depth"
```
- addedInput schema / properties / depth / type
```
Added value: +"integer"
```
- addedInput schema / properties / include_subdomains
```
Added value: +{
+  "type": "boolean"
+}
```
- removedInput schema / properties / max_pages / anyOf
```
Removed value: -[
-  {
-    "type": "integer"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / max_pages / default
```
Removed value: -null
```
- removedInput schema / properties / max_pages / title
```
Removed value: -"Max Pages"
```
- addedInput schema / properties / max_pages / type
```
Added value: +"integer"
```
- addedInput schema / properties / render_mode / $ref
```
Added value: +"#/$defs/CrawlRenderMode"
```
- removedInput schema / properties / render_mode / anyOf
```
Removed value: -[
-  {
-    "$ref": "#/$defs/CrawlRenderMode"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / render_mode / default
```
Removed value: -null
```
- removedInput schema / properties / url / title
```
Removed value: -"Url"
```
- removedInput schema / title
```
Removed value: -"crawlArguments"
```

Changed2 schema fields changedv1.0.1Jun 17, 2026

addedInput schema / $defs

Added value: +{
+  "CrawlRenderMode": {
+    "description": "How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.",
+    "enum": [
+      "http",
+      "browser"
+    ],
+    "title": "CrawlRenderMode",
+    "type": "string"
+  }
+}

addedInput schema / properties / render_mode

Added value: +{
+  "anyOf": [
+    {
+      "$ref": "#/$defs/CrawlRenderMode"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}

First observedv1.0.0Jun 15, 2026

TDQS

B3.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it does disclose the key behavioral trait: the crawl is non-blocking and results must be polled via crawl_status. However, it omits other important behaviors such as default max_pages, whether include_subdomains affects scope, rate limiting, or any permission requirements.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but front-loaded: the non-blocking nature and polling path come first, followed by parameter shorthand. Every clause carries information, though the backtick-heavy telegraphic style trades some readability for compactness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Since an output schema exists, return-value detail is unnecessary, and the async polling model is covered. The gap is on the input side: two of five parameters have no semantics anywhere, and there is no indication of crawl scope limits or failure behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 20%, so the description must compensate. It usefully explains depth values including the null = whole-site case, which contradicts the schema's integer type and would otherwise be undiscoverable. But max_pages and include_subdomains remain undocumented in both the schema and the description, and the render_mode explanation merely repeats the schema text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start a non-blocking crawl') and the non-blocking qualifier immediately sets it apart from the synchronous siblings. It also names the follow-up tool crawl_status(task_id), making the crawl lifecycle obvious. It never says what a crawl actually does (presumably fetch linked pages from a site), leaving the resource slightly opaque.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a follow-up protocol ('poll via crawl_status(task_id)') rather than when-to-use guidance, and it encodes depth semantics (0 = single URL, N = N levels) that help choose parameters. There is no explicit statement of when to prefer this over siblings like search or catalog_browse, and no exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.