Skip to main content
Glama

oassis — web for agents

web_crawl

Follows a site's links and reads every page. Returns a jobId; poll it with web_crawl_status. Charged up front for the pages it is allowed to read (limit), and the pages it never reads are refunded. Use web_map first if you only need the urls.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesWhere to start.
limitNoPages it may read (default 25, max 200).
maxAgeNoAccept an answer up to this many milliseconds old. A cache hit costs $0.0002 instead of the format price. Leave it out to force a fresh render.
formatsNoOutputs you want in the same response. `controls` is the map of what can be clicked; `elements` needs `selectors`; `json` needs `json.prompt`.
maxDepthNoHow far to follow links (default 2, max 5).
excludePathsNo
includePathsNo
includeSubdomainsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden and does well: it discloses the async job pattern (returns jobId, poll with web_crawl_status) and the unusual billing model (charged up front for `limit` pages, unread pages refunded). No auth requirements or rate limits are mentioned, but the async and cost behavior are the non-obvious traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, all front-loaded and earned: purpose first, then the async contract and cost model, then the routing alternative. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an async, no-output-schema, no-annotation crawl tool, the description covers the critical facts an agent needs: what it does, how to retrieve results, cost behavior, and when to prefer web_map. Missing only the semantics of the path-filter parameters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 63% with 8 parameters. The description reinforces the billing meaning of `limit` (charged for the pages it is allowed to read), adding value beyond the schema's 'Pages it may read'. But excludePaths, includePaths, and includeSubdomains receive no explanation in either place, and the `formats`/`magAge` semantics live only in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: follows a site's links and reads every page. It explicitly separates itself from the sibling web_map by noting to use web_map first if only urls are needed, so an agent can distinguish them without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear alternative condition (use web_map if you only need urls) and explains the follow-up workflow (poll with web_crawl_status). It does not, however, address when to choose this over web_scrape or how it relates to the batch tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources