Skip to main content
Glama
Gi0tto

sluicer

by Gi0tto
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

Crawl a site

crawl_site
Read-onlyIdempotent

Crawl a website from a starting URL, follow its internal links, and get concise summaries of every page while respecting robots.txt and optional limits.

Instructions

Crawl a site from url, following its links, and summarise every page.

url: an http(s) address where the crawl starts; only links on its site are followed. max_pages: the most pages taken, 1 to 25. max_depth: the most links from url, 0 to 3; 0 reads url alone. respect_tdm: give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary. include: text an address must contain for its link to be followed (any one of them); plain text, not a pattern. exclude: text that stops a link being followed when its address contains it.

Returns {"ok", "url", "pages", "stopped"}. Pages come breadth first, each {"ok", "url", "depth", "found_on", "landed", "fetch", "canonical", "summary", "sources", "types", "links"} -- the summary and the types declared, not the records; call extract_declared on a page for those -- or, when it has nothing, {"ok": false, "error"} with the page's reason. stopped is "done", "max_pages" (links were left unfollowed) or "time_budget" (a minute passed). One request at a time, a second apart or the site's Crawl-delay, robots.txt obeyed; a page asked again after a request that may succeed later says so in "retries". ok is false only when no page could be read, and error then says why. Past 75,000 bytes the heaviest summary answers of any page go first, named in that page's summary_left_out, then the last pages, counted in pages_left_out.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesan http(s) address where the crawl starts; only links on its site are followed.
excludeNotext that stops a link being followed when its address contains it.
includeNotext an address must contain for its link to be followed (any one of them); plain text, not a pattern.
max_depthNothe most links from url, 0 to 3; 0 reads url alone.
max_pagesNothe most pages taken, 1 to 25.
respect_tdmNogive a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
okYes
urlNo
errorNo
pagesNo
stoppedNo
pages_left_outNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed7 schema fields changedv0.9.0
    • addedOutput schema / $defs / CrawledPage / properties / retries
      Added value: +{
+  "items": {
+    "$ref": "#/$defs/RetryAnswer"
+  },
+  "title": "Retries",
+  "type": "array"
+}
    • addedOutput schema / $defs / CrawledPage / properties / summary_left_out
      Added value: +{
+  "items": {
+    "type": "string"
+  },
+  "title": "Summary Left Out",
+  "type": "array"
+}
    • changedOutput schema / $defs / ErrorDetail / description
      Previous value: -"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. The others need something\nto change first -- an install, an input, or the caller's mind about a site\nthat said no."New value: +"Why a tool could not answer.\n\n``retryable`` is true only for ``fetch_failed``, and on a crawled page for\n``rate_limited``: the same call may work later. Not every ``fetch_failed``\nis: a redirect loop, or an encoding this install cannot read, would be\nmet again. The others need something to change first -- an install, an\ninput, or the caller's mind about a site that said no."
    • changedOutput schema / $defs / ErrorDetail / properties / code / enum
      Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved"
+]
    • changedOutput schema / $defs / PageError / properties / code / enum
      Previous value: -[
-  "missing_extra",
-  "refused_by_robots",
-  "refused_address",
-  "fetch_failed",
-  "too_large",
-  "bad_input",
-  "tdm_reserved",
-  "redirected_off_site",
-  "crawl_delay_too_long",
-  "rate_limited"
-]New value: +[
+  "missing_extra",
+  "refused_by_robots",
+  "refused_by_site",
+  "payment_required",
+  "refused_address",
+  "fetch_failed",
+  "too_large",
+  "bad_input",
+  "tdm_reserved",
+  "redirected_off_site",
+  "crawl_delay_too_long",
+  "rate_limited"
+]
    • addedOutput schema / $defs / RetryAnswer
      Added value: +{
+  "description": "One time a crawled page was asked again: what the request before came\nto, and the seconds from its end to this one's start.",
+  "properties": {
+    "after": {
+      "title": "After",
+      "type": "number"
+    },
+    "reason": {
+      "title": "Reason",
+      "type": "string"
+    }
+  },
+  "required": [
+    "reason",
+    "after"
+  ],
+  "title": "RetryAnswer",
+  "type": "object"
+}
    • addedOutput schema / properties / pages_left_out
      Added value: +{
+  "title": "Pages Left Out",
+  "type": "integer"
+}
  2. Changed6 schema fields changedv0.7.0
    • addedInput schema / properties / exclude / description
      Added value: +"text that stops a link being followed when its address contains it."
    • addedInput schema / properties / include / description
      Added value: +"text an address must contain for its link to be followed (any one of them); plain text, not a pattern."
    • addedInput schema / properties / max_depth / description
      Added value: +"the most links from url, 0 to 3; 0 reads url alone."
    • addedInput schema / properties / max_pages / description
      Added value: +"the most pages taken, 1 to 25."
    • addedInput schema / properties / respect_tdm / description
      Added value: +"give a page whose site reserves its text and data mining rights (TDMRep) as a tdm_reserved error, never its summary."
    • addedInput schema / properties / url / description
      Added value: +"an http(s) address where the crawl starts; only links on its site are followed."
  3. First observedv0.1.0

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, non-destructive behavior, and the description substantially expands on this with robots.txt compliance, rate limiting, one-request-at-a-time, retries, TDM handling, breadth-first order, time budget, and size-limit behavior. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but every sentence adds necessary detail, organized as purpose, parameters, return shape, then operational constraints. The structure is front-loaded and scannable despite its density.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and six parameters, the description fully covers input, output shape, error cases, stopping conditions, rate limiting, robots.txt, and size limits. It even routes the agent to extract_declared for page records, so nothing needed to call the tool correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all six parameters. The description mostly restates the schema's parameter descriptions; it adds useful behavioral context around respect_tdm and include/exclude, but no major new parameter-level semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action ('Crawl a site from url, following its links, and summarise every page') with a clear resource and scope. This differentiates it from siblings like fetch_page (single page) and extract_declared (records extraction).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly conveys when crawling is appropriate and even points to extract_declared as the alternative for extracting declared records. It does not explicitly contrast with map_site or fetch_page, but the crawl/summarize purpose and max_depth=0 behavior provide clear context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools