Skip to main content
Glama
Agency MCP Server

Agency MCP Server

Site crawl

crawl_site
Read-only

Crawl a client site from an allowlisted start URL, respecting robots.txt, to audit broken links, redirects, duplicate titles and descriptions, and orphan pages.

Instructions

Rate-limited crawl from a start URL inside the client's allowlist, obeying robots.txt. Reports broken links, redirects, duplicate titles and descriptions, and orphan pages (sitemap URLs no crawled page links to).

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "client": "<client>",
      "url": "<url>"
    },
    "name": "crawl_site"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesAbsolute URL inside the client's allowlist.
clientYesClient id from the server's config (dental, cafe, plumber).
maxPagesNoDefault 50; capped by the server's MAX_CRAWL_PAGES.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnly/openWorld/non-destructive; the description adds genuinely new behavior: rate limiting, robots.txt obedience, allowlist restriction, and the specific report contents. It omits crawl duration, depth behavior, and how results are returned, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two dense sentences with zero filler; the constraint set and the reported findings are ordered before any detail that could be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description must carry the return-value burden, and it does by enumerating broken links, redirects, duplicate titles/descriptions, and orphan pages. Missing only operational detail such as crawl scale or result delivery, which matters mildly for an open-world crawl.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and all three parameters are documented there, including the MAX_CRAWL_PAGES cap. The description only restates that the start url must be inside the allowlist, adding little beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Rate-limited crawl from a start URL") plus the scope constraint (allowlist, robots.txt) and the exact findings produced. An agent can distinguish it from audit_page or check_crawler_access without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The allowlist and robots.txt constraints imply when the tool is applicable, but no explicit when-to-use/when-not guidance or named alternatives (e.g. single-page audit vs. full crawl) are given. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.