Skip to main content
Glama

Crawl and audit a public site

edgeo_crawl_site
Read-only

Crawl a bounded sample of same-host HTML pages from a start URL and sitemap, respecting robots.txt and crawl-delay, to audit public site content without rendering JavaScript or leaving the host.

Instructions

Run a bounded same-host HTML crawl seeded from the start URL and same-host sitemap URLs. Defaults to 50 attempted URLs and accepts up to 200. Preserves trailing slashes when requesting URLs, reads robots.txt first, applies its rules and crawl-delay, separates non-HTML sitemap resources, and never renders JavaScript or follows links to other hosts. Reports known queued URL counts, not sitewide coverage.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "edgeo_crawl_site"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesPublic HTTP or HTTPS start URL.
delayMsNo
maxPagesNoMaximum URL fetch attempts, including non-HTML sitemap resources and URL variants that redirect to an already seen page.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.1.4
    • changedInput schema / properties / maxPages / default
      Previous value: -20New value: +50
    • changedInput schema / properties / maxPages / maximum
      Previous value: -50New value: +200
  2. Changed1 schema field changedv0.1.2
    • addedInput schema / properties / maxPages / description
      Added value: +"Maximum URL fetch attempts, including non-HTML sitemap resources and URL variants that redirect to an already seen page."
  3. First observedv0.1.0

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds substantial behavior beyond the read-only and open-world annotations: default and maximum URL counts, robots.txt handling and crawl delay, trailing-slash preservation, separation of non-HTML sitemap resources, no JavaScript rendering, no cross-host link following, and a caveat that reported counts are queued URLs rather than sitewide coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded and contains no filler, but it is one dense multi-clause paragraph that could be easier to scan if split. Every clause still conveys relevant behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only open-world crawl tool with no output schema, the description covers scope, limits, robots handling, and result caveats well. It is slightly incomplete on what the reported output actually contains beyond queued URL counts, but the core behavior is clear.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67%, so the schema already documents the URL and maxPages parameters well. The description repeats the 50/200 defaults and adds useful interpretation of maxPages as attempted URLs, but it does not explain delayMs semantics or its interaction with robots crawl-delay. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific action and scope: a bounded same-host HTML crawl seeded from a start URL and sitemap URLs. It is clearly distinct from sibling tools like audit_page and analyze_visibility.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the described crawl scope, but the description never explicitly states when to choose this tool over edgeo_audit_page or other siblings. It also does not name exclusions or prerequisites beyond the public HTTP/HTTPS requirement in the schema.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.