WebLens

crawl_site

Crawl a site and get clean markdown for every page, in one call. Same-host BFS with depth and page-budget limits, robots.txt honoured by default. $0.0015 per requested page (1-25). Price: $0.0015 per requested page (1-25); an x402 payment is at least $0.004

Input Schema

Name	Required	Description
`url`	Yes	The URL to process
`limit`	No	Page budget — you are charged per requested page
`exclude`	No	Skip URLs whose path+query contains one of these
`include`	No	Only crawl URLs whose path+query contains one of these
`timeout`	No	Request timeout in milliseconds
`maxChars`	No	Per-page content character cap
`maxDepth`	No	Link depth from the start URL (0 = start page only)
`respectRobots`	No	Honour robots.txt (default true; disable only for sites you control)

Schema Changelog

Changes observed during successful MCP inspections.

Changed29 schema fields changedOct 13, 2026
- addedInput schema / properties / exclude / default
```
Added value: +[]
```
- changedInput schema / properties / exclude / description
```
Previous value: -"Skip URLs whose path+query contains one of these substrings"New value: +"Skip URLs whose path+query contains one of these"
```
- addedInput schema / properties / exclude / items / maxLength
```
Added value: +200
```
- addedInput schema / properties / exclude / items / minLength
```
Added value: +1
```
- addedInput schema / properties / exclude / maxItems
```
Added value: +20
```
- addedInput schema / properties / include / default
```
Added value: +[]
```
- changedInput schema / properties / include / description
```
Previous value: -"Only crawl URLs whose path+query contains one of these substrings"New value: +"Only crawl URLs whose path+query contains one of these"
```
- addedInput schema / properties / include / items / maxLength
```
Added value: +200
```
- addedInput schema / properties / include / items / minLength
```
Added value: +1
```
- addedInput schema / properties / include / maxItems
```
Added value: +20
```
- addedInput schema / properties / limit / default
```
Added value: +10
```
- changedInput schema / properties / limit / description
```
Previous value: -"Page budget (1-25, default: 10) — charged per requested page"New value: +"Page budget — you are charged per requested page"
```
- addedInput schema / properties / limit / example
```
Added value: +10
```
- addedInput schema / properties / limit / maximum
```
Added value: +25
```
- addedInput schema / properties / limit / minimum
```
Added value: +1
```
- addedInput schema / properties / maxChars / default
```
Added value: +8000
```
- changedInput schema / properties / maxChars / description
```
Previous value: -"Per-page content character cap (default 8000)"New value: +"Per-page content character cap"
```
- addedInput schema / properties / maxDepth / default
```
Added value: +2
```
- changedInput schema / properties / maxDepth / description
```
Previous value: -"Link depth from the start URL (0-3, default: 2)"New value: +"Link depth from the start URL (0 = start page only)"
```
- addedInput schema / properties / maxDepth / example
```
Added value: +2
```
- addedInput schema / properties / maxDepth / maximum
```
Added value: +3
```
- addedInput schema / properties / maxDepth / minimum
```
Added value: +0
```
- addedInput schema / properties / respectRobots / default
```
Added value: +true
```
- changedInput schema / properties / respectRobots / description
```
Previous value: -"Honour robots.txt (default: true; disable only for sites you control)"New value: +"Honour robots.txt (default true; disable only for sites you control)"
```
- addedInput schema / properties / timeout / default
```
Added value: +10000
```
- changedInput schema / properties / timeout / description
```
Previous value: -"Per-page timeout in ms (default 10000)"New value: +"Request timeout in milliseconds"
```
- changedInput schema / properties / url / description
```
Previous value: -"Start URL — the crawl stays on this host"New value: +"The URL to process"
```
- addedInput schema / properties / url / example
```
Added value: +"https://example.com"
```
- addedInput schema / properties / url / format
```
Added value: +"uri"
```
First observedAug 4, 2026

TDQS

A3.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations the description carries the full burden and does well: it discloses the traversal strategy (same-host BFS), hard limits (depth, page budget), the robots.txt policy and its override, and a per-page cost plus an x402 payment floor. It stops short of describing error/timeout handling or what happens when the budget is exhausted.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The opening is front-loaded and tight, but the pricing sentence is duplicated almost verbatim ('$0.0015 per requested page (1-25). Price: $0.0015 per requested page (1-25)...'), which wastes space and reads as boilerplate concatenation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter mutation-free crawl tool with no annotations and no output schema, the description covers traversal semantics, cost, and robots policy well. Return shape is briefly covered ('clean markdown for every page') but pagination, partial-failure behavior, and auth mechanics around x402 are left unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter including limit, maxDepth, include/exclude and respectRobots is already documented in the schema. The description only restates the per-page price and the 1-25 range, adding no syntax or behavioral nuance beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (crawl) and resource (site) plus the concrete payoff ('clean markdown for every page, in one call'), so the bulk-crawl intent is unmistakable. It never names the obvious siblings (fetch_webpage for a single page, map_site for URL discovery), so the differentiation is left implicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the scope ('every page, in one call', same-host BFS) and the depth/page-budget limits, which tell an agent this is the batch option rather than a single fetch. There is no explicit when-to-use, when-not-to-use, or named alternative, so the agent must infer the routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All WebLens tools Inspect

Try in Browser

Give agents eyes on the webHosted screenshot MCP. Send a URL, get a pixel-perfect image. $0.002 per grab. Works with Claude, Cursor, and Codex.grabbit.liveAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

crawl_site

Input Schema

Schema Changelog

TDQS

Glama MCP Gateway

Resources