Skip to main content
Glama

robotbase-dev

Web Crawl Async

web_crawl_async
Read-onlyIdempotent

Submit a same-host crawl as an asynchronous job and return a job_id immediately. When to use: you need more than a couple of pages and cannot hold a long HTTP request; submit here, then poll web_crawl_job until status is done, failed or cancelled. Do not use: for one page (web_fetch) or when you need the result in this call (web_crawl). Returns: {ok, job_id, status, url, depth, max_pages, max_chars, idempotent, poll_with}. Pass idempotency_key to retry safely (24h window returns the same job). Metered: shares the web free bucket (100 calls/day per IP), then 3 credits per submit over x402.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesStart URL (same host is crawled).
depthNoLink depth 1-2. Default 2.
max_charsNoPer-page content cap 200-4000. Default 1000.
max_pagesNoMax pages 1-20. Default 20.
idempotency_keyNoOptional; retries within 24h return the same job.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnly, idempotent, openWorld, non-destructive), yet the description still adds substantial context beyond them: the metering model (100 calls/day free bucket per IP, then 3 credits per submit over x402), the 24h idempotency-key retry window, and the exact return shape. This is exactly the kind of operational context annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but well-structured, with a clear purpose sentence followed by labeled when-to-use, when-not-to-use, returns, and metering sections. No sentence is filler; even the metering note is decision-relevant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With only 5 params at full schema coverage, no output schema, and rich annotations, the description is fully sufficient: it tells the agent what happens, how to poll, how to retry idempotently, and what it costs. Nothing needed to invoke correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so ranges and defaults for depth, max_chars, and max_pages are already documented in the schema. The description's note on idempotency_key ('retry safely, 24h window returns same job') largely repeats the schema text, adding little new semantics. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Submit a same-host crawl as an asynchronous job') plus the immediate return of job_id. It clearly distinguishes itself from web_crawl (synchronous), web_fetch (single page), and web_crawl_job (polling) by name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly gives when-to-use (more than a couple of pages, can't hold a long request, then poll web_crawl_job until done/failed/cancelled) and when-not-to-use (one page → web_fetch; need result in this call → web_crawl). The alternatives are named with the condition that selects each.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.