Skip to main content
Glama

crawl_site

Start a background crawl of a configured documentation site to update its local corpus. It stages changes and only replaces the corpus on success, preserving the previous version if it fails.

Instructions

Start a background crawl for a configured site. Returns immediately with a job ID. The crawl builds in a staging area and replaces the stored corpus only when it completes, so a cancelled or failed crawl leaves the previous corpus intact.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
site_keyYesSite key from config file (e.g., 'langchain_py', 'rust_docs')
incrementalNoEnable incremental mode (skip unchanged pages)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: it discloses that execution is asynchronous (returns a job ID immediately), that writes go to a staging area and replace the corpus atomically, and that cancelled/failed crawls leave the prior corpus intact. That last point meaningfully explains why destructiveHint=false despite this being a mutating crawl.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, zero filler. The core action is front-loaded, and the async return and atomicity guarantee follow in logical order.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description compensates by stating the return value (job ID) and explains the lifecycle/failure semantics. An agent has everything needed to invoke this correctly and understand the consequences.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so site_key and incremental are already documented in the schema. The description adds no extra semantics about these parameters, which is the expected baseline when the schema carries the full burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start a background crawl for a configured site') and adds the async contract ('Returns immediately with a job ID'). This cleanly separates it from siblings like get_job_status and cancel_crawl without needing to name them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The 'background crawl' framing implies this is the kicking-off action, but there is no explicit when-to-use guidance or mention of alternatives such as diff_crawl or list_sites. Usage is inferable from the verb rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.