Skip to main content
Glama
blakemartz

private-search-4-agents

by blakemartz

Crawl site

crawl_site

Map a website into a URL/title/snippet list and cache pages for later fetching; use it to discover and scope site content before extracting pages locally.

Instructions

Map a site (sitemap, else link-following if enabled) into a URL/title/snippet list; pages are cached for fetch_url. A start URL with a path stays under it; include_path '/' crawls the whole site.

Input Schema

NameRequiredDescriptionDefault
urlYesSite root or docs URL
max_pagesNoPage cap (default 20)
exclude_pathNoSkip URLs under this path prefix
include_pathNoOnly URLs under this path prefix; replaces the default scope
same_domain_onlyNoStay on the start domain (default true)

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv4.0.0

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the burden and does disclose the discovery strategy, the return shape (URL/title/snippet list), a side effect (caching pages for fetch_url), and scoping rules by path. It omits failure behavior, robots/politeness handling, and whether the crawl is bounded by anything other than max_pages.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences with no filler, front-loading the core behavior and return shape before the scoping rules. Every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter tool with no output schema, no annotations, and rich schema coverage, the description covers purpose, output contents, caching side effect, and scope defaults. It falls short on crawl limits, failure modes, and the ambiguous 'if enabled' condition.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline would be 3, but the description adds real semantics: a start URL with a path constrains the crawl to that subtree, and include_path '/' widens it to the whole site. That explains the interaction between url and include_path/default scope in a way the schema fields alone only hint at.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource: it maps a site into a URL/title/snippet list, and it names the resolution strategy (sitemap first, else link-following). It also distinguishes itself from the fetch_url sibling by stating that crawled pages are cached for later fetch_url calls.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The crawl-then-fetch_url workflow is implied by 'pages are cached for fetch_url', which is useful context, but there is no explicit statement of when to crawl versus when to use search or fetch_url directly, and no prerequisites or exclusions. 'else link-following if enabled' also raises a condition (a setting being enabled) that the agent cannot verify from the schema.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools