Skip to main content
Glama
web-insights-mcp
by nqcthedev

Crawl website

crawl_site
Read-onlyIdempotent

Map a website by discovering its pages and returning each URL, HTTP status, and title while honoring robots.txt; use it to find broken pages, list blog posts, or pick URLs to fetch.

Instructions

Discover the pages of a website and return each page's URL, HTTP status and . Reads sitemap.xml (and sitemap indexes) when present, otherwise follows same-origin links breadth-first. Always honours robots.txt (disallowed URLs are listed under skipped) and Crawl-delay. Use it to map a site, find broken pages, list blog posts, or pick URLs to read with fetch_page. Page bodies are not returned — call fetch_page for content.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl_site"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesStart URL. Only pages on the same origin (scheme + host + port) are crawled.
strategyNo"auto" uses sitemap.xml when available and falls back to following links; "sitemap" or "links" force one method.auto
max_depthNoLink depth from the start URL (only used by the "links" strategy).
max_pagesNoMaximum pages to fetch (hard cap 200).
path_prefixNoOnly crawl URLs whose path starts with this, e.g. "/blog/".

Output Schema

NameRequiredDescriptionDefault
pagesYes
statsYes
robotsYes
sitemapNo
skippedYes
startUrlYes
strategyYes
truncatedYes
durationMsYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, non-destructive, open-world behavior, and the description adds genuinely new operational context: sitemap-first with breadth-first fallback, robots.txt enforcement with disallowed URLs surfaced under `skipped`, and Crawl-delay compliance. It also sets expectations about what is NOT returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four compact sentences, front-loaded with the outcome, then method, then constraints, then the hand-off to fetch_page. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 5 parameters, full schema coverage, an output schema, and rich annotations, the description covers everything an agent needs: scope, method, politeness constraints, and the correct follow-up tool. Nothing material is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so url, strategy, max_depth, max_pages and path_prefix are already documented in the schema. The description reinforces the same-origin constraint and the sitemap/link strategies but does not add syntax or format detail beyond the schema, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a concrete verb+resource ('Discover the pages of a website') and names the exact return payload ('URL, HTTP status and <title>'). It also explicitly distinguishes itself from the fetch_page sibling by noting that page bodies are not returned.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use cases (map a site, find broken pages, list blog posts) and routes the agent to the alternative tool when content is needed ('call fetch_page for content'). The boundary between crawling and fetching is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.