lilbee

Overview Schema Source Related Servers Discussions Admin

crawl

Start a non-blocking website scan with configurable depth and HTTP or browser rendering, then poll its status by task ID. Supports optional login credentials for the seed host.

Instructions

Start a non-blocking crawl; poll via crawl_status(task_id). depth=0 (default) = single URL, N = follow links N levels, null = whole site. render_mode: "http"/"browser". authorization (a whole Authorization header value) or username and password, and cookie (a Cookie header value), log in to the seed host only.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl"
  }
}

Input Schema

Name	Required	Description
`url`	Yes
`depth`	No
`cookie`	No
`password`	No
`username`	No
`max_pages`	No
`render_mode`	No	How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.
`authorization`	No
`include_subdomains`	No

Output Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

Changed4 schema fields changedv1.0.17Oct 29, 2026
- addedInput schema / properties / authorization
```
Added value: +{
+  "type": "string"
+}
```
- addedInput schema / properties / cookie
```
Added value: +{
+  "type": "string"
+}
```
- addedInput schema / properties / password
```
Added value: +{
+  "type": "string"
+}
```
- addedInput schema / properties / username
```
Added value: +{
+  "type": "string"
+}
```

Changed5 schema fields changedv1.0.16Oct 20, 2026

removedInput schema / $defs

Removed value: -{
-  "CrawlRenderMode": {
-    "description": "How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.",
-    "enum": [
-      "http",
-      "browser"
-    ],
-    "title": "CrawlRenderMode",
-    "type": "string"
-  }
-}

removedInput schema / properties / render_mode / $ref
```
Removed value: -"#/$defs/CrawlRenderMode"
```

addedInput schema / properties / render_mode / description

Added value: +"How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS."

addedInput schema / properties / render_mode / enum
```
Added value: +[
+  "http",
+  "browser"
+]
```
addedInput schema / properties / render_mode / type
```
Added value: +"string"
```

Changed14 schema fields changedv1.0.2Jul 22, 2026
- removedInput schema / properties / depth / anyOf
```
Removed value: -[
-  {
-    "type": "integer"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / depth / default
```
Removed value: -null
```
- removedInput schema / properties / depth / title
```
Removed value: -"Depth"
```
- addedInput schema / properties / depth / type
```
Added value: +"integer"
```
- addedInput schema / properties / include_subdomains
```
Added value: +{
+  "type": "boolean"
+}
```
- removedInput schema / properties / max_pages / anyOf
```
Removed value: -[
-  {
-    "type": "integer"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / max_pages / default
```
Removed value: -null
```
- removedInput schema / properties / max_pages / title
```
Removed value: -"Max Pages"
```
- addedInput schema / properties / max_pages / type
```
Added value: +"integer"
```
- addedInput schema / properties / render_mode / $ref
```
Added value: +"#/$defs/CrawlRenderMode"
```
- removedInput schema / properties / render_mode / anyOf
```
Removed value: -[
-  {
-    "$ref": "#/$defs/CrawlRenderMode"
-  },
-  {
-    "type": "null"
-  }
-]
```
- removedInput schema / properties / render_mode / default
```
Removed value: -null
```
- removedInput schema / properties / url / title
```
Removed value: -"Url"
```
- removedInput schema / title
```
Removed value: -"crawlArguments"
```

Changed2 schema fields changedv1.0.1Jun 17, 2026

addedInput schema / $defs

Added value: +{
+  "CrawlRenderMode": {
+    "description": "How a crawl fetches pages. ``http`` uses no browser; ``browser`` runs Chromium with JS.",
+    "enum": [
+      "http",
+      "browser"
+    ],
+    "title": "CrawlRenderMode",
+    "type": "string"
+  }
+}

addedInput schema / properties / render_mode

Added value: +{
+  "anyOf": [
+    {
+      "$ref": "#/$defs/CrawlRenderMode"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null
+}

First observedv1.0.0Jun 15, 2026

TDQS

A3.8/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It does disclose the key behavioral traits: the call is non-blocking, auth/cookie credentials are applied to the seed host only, and render_mode selects a JS-capable browser. It says nothing about pagination caps (max_pages), subdomain inclusion, rate limiting, or what the task returns, leaving real gaps for a mutation-like long-running operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Every clause earns its place: the operation and polling sibling come first, then per-parameter semantics packed into compact terse fragments. No redundancy with the schema beyond a brief restatement of the render_mode enum, which is acceptable framing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Return values need not be explained since an output schema exists, and the async lifecycle plus auth scoping are covered. However, for a 9-parameter tool with 11% schema coverage, two parameters (max_pages, include_subdomains) are invisible to the agent and the credential-submission risk is only lightly touched, so the definition is adequate but incomplete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 11%, so the description must compensate, and it does for most parameters: depth's three-way semantics (0 / N / null), render_mode's http-vs-browser meaning, and the authorization/username+password/cookie login trio with the seed-host scoping caveat. It leaves max_pages and include_subdomains completely undocumented, which keeps it out of the top band.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb and resource ('Start a non-blocking crawl') and immediately names the sibling to poll with (crawl_status), so the agent can separate it from crawl_status/crawl_cancel without opening a schema. It falls short of 5 only because the crawl's scope (site vs single page) is only implied through the depth legend.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent to crawl_status(task_id) for progress, which is the main decision point after invocation, and the depth legend tells the agent which mode to pick for a single URL vs whole site. There is no guidance on when not to crawl (e.g., when 'search' or 'list_documents' would be cheaper), so no exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.