Skip to main content
Glama

crawl_site

Start a background crawl for a configured documentation site by site key. Returns a job ID immediately, so agents can proceed while content refreshes for offline reads.

Instructions

Start a background crawl for a configured site. Returns immediately with a job ID.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
site_keyYesSite key from config file (e.g., 'langchain_py', 'rust_docs')
incrementalNoEnable incremental mode (skip unchanged pages)

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. First observed

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds key behavioral context beyond annotations: the crawl runs in the background and the call returns immediately with a job ID, implying the actual work happens asynchronously. This helps the agent know to poll get_job_status later. It also clarifies that the site must already be configured, setting a prerequisite not visible from annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no filler: the first states the action and scope, the second states the return behavior. Information is front-loaded and every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple 2-parameter tool with no output schema, the description is sufficient: it explains the start action, background execution, and the immediate return of a job ID. It could mention how to monitor progress, but the existence of a get_job_status sibling makes that omission minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both site_key and incremental are already fully documented. The description does not add any additional meaning for these parameters, so the baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a clear action: start a background crawl for a configured site, and explicitly notes it returns a job ID immediately. This distinguishes it from sibling tools like cancel_crawl or get_job_status, which operate on existing crawls.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it—when you want to kick off a crawl of a configured site—but does not explicitly discuss alternatives or exclusion conditions. An agent must infer that cancel_crawl or diff_crawl serve different purposes, so some routing burden remains on the agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Sriram-PR/doc-scraper'

If you have feedback or need assistance with the MCP directory API, please join our Discord server