Skip to main content
Glama

crawl

Find pages on a site relevant to a query by following links from a start URL; returns matching URLs with snippets and respects robots.txt.

Instructions

Crawl a site from a start page, following the links most relevant to query first. Returns the best pages as URLs with one-line snippets (not full text); fetch them for details. Respects robots.txt.

Input Schema

NameRequiredDescriptionDefault
urlYesPage to start from.
queryYesWhat you are looking for. Links are followed in order of relevance to it.
keywordsNoExact terms or synonyms that should match, to sharpen the query.
max_depthNoMaximum link hops from the start page (default 3).
max_pagesNoStop after this many pages (default 30, max 200).
max_resultsNoMaximum pages to return (default 10).
other_sitesNoAlso follow links to other sites (default false).

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the load and does well: it discloses the return shape (URLs plus one-line snippets, explicitly not full text) and notes robots.txt compliance. It is silent on cost/latency and what happens on blocked or failed fetches, which are the remaining behavioral gaps.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences, each earning its place: purpose, return shape, and the follow-up action. The most decision-relevant information (what you get back and where to go next) is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter tool with no output schema, the description covers the essential unknowns: result granularity and the sibling handoff. Minor gaps remain around crawl duration, failure modes, and the distinction from `links`, but nothing that would cause a misinvocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and every parameter including defaults and the max_pages cap is documented in the schema. The description only reinforces `query` as the relevance signal; it adds no additional meaning about keywords, depth, or other_sites beyond what the schema states.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Crawl a site from a start page') plus the ordering principle (links most relevant to query first), and explicitly separates itself from the `fetch` sibling by saying results are snippets, not full text.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear two-step workflow: crawl to discover candidate pages, then `fetch` them for details. It does not address when to prefer the `links` sibling or when a plain search would be better, so it stops short of full when/when-not coverage.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools