Skip to main content
Glama

Crawl a site's design system

crawl_site

Crawl multiple same-origin pages to extract shared and unique design patterns. Consolidate the evidence into a site-wide design system and page inventory for replication.

Instructions

Use this when the goal spans several pages and you need shared versus page-specific design patterns. Avoid it when one page is the target; analyze or create a kit for that page instead. Merge design evidence from bounded same-origin pages into a site-wide system and page inventory. Cost: heavy. Session: loads several bounded same-origin pages. May write files when an output path is provided. Possible follow-ups if evidence is still missing: create_replication_kit.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesAbsolute http:// or https:// start URL; same-origin links are crawled from here.
maxDepthNoMaximum link depth from the start URL.
maxPagesNoMaximum pages to load, start page and failed loads included.
outputDirNoAbsolute directory: writes crawl/pages/<slug>/design.json + layout.json (+ viewport.jpg), crawl/site.json, and crawl/SITE.md.
timeoutMsNoNavigation timeout in milliseconds.
userAgentNoOptional browser user-agent override.
waitUntilNoPage lifecycle event to await before extraction.domcontentloaded
autoScrollNoScroll through the page before extraction to trigger lazy-rendered content.
maxScrollsNoMaximum viewport-sized scroll steps when autoScroll is enabled.
stripQueryNoTreat URLs differing only by query string as the same page.
viewportWidthNoBrowser viewport width in CSS pixels.
viewportHeightNoBrowser viewport height in CSS pixels.
excludePatternsNoGlob patterns for paths to skip. Exclude wins over include.
includePatternsNoGlob patterns ('*' wildcard) a path must match to be crawled. Empty means all paths.
maxLinksPerPageNoLinks harvested per page.
includeScreenshotsNoWith outputDir, save a viewport screenshot per page.
maxElementsPerPageNoDOM elements analyzed per page (lighter than analyze_site).
allowPrivateNetworkNoAllow localhost and private-network URLs. Enable only for trusted sites.

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
resultYes
guidanceYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed6 schema fields changedv0.10.1
    • addedOutput schema / properties / guidance
      Added value: +{
      +  "additionalProperties": false,
      +  "properties": {
      +    "cost": {
      +      "enum": [
      +        "light",
      +        "standard",
      +        "heavy"
      +      ],
      +      "type": "string"
      +    },
      +    "nextTools": {
      +      "description": "Candidate follow-ups; do not call them automatically or in parallel.",
      +      "items": {
      +        "enum": [
      +          "analyze_site",
      +          "analyze_html",
      +          "inspect_component",
      +          "capture_screenshot",
      +          "extract_assets",
      +          "extract_animations",
      +          "extract_layout",
      +          "clone_section",
      +          "capture_interactions",
      +          "extract_responsive",
      +          "compare_replica",
      +          "create_replication_kit",
      +          "capture_motion",
      +          "crawl_site"
      +        ],
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "sessionPolicy": {
      +      "enum": [
      +        "offline-document",
      +        "single-url",
      +        "multi-viewport",
      +        "dual-page",
      +        "site-crawl"
      +      ],
      +      "type": "string"
      +    },
      +    "summary": {
      +      "minLength": 1,
      +      "type": "string"
      +    },
      +    "tool": {
      +      "const": "crawl_site",
      +      "type": "string"
      +    },
      +    "warnings": {
      +      "description": "Non-fatal limitations or incomplete evidence from this run.",
      +      "items": {
      +        "type": "string"
      +      },
      +      "type": "array"
      +    },
      +    "writesFiles": {
      +      "type": "boolean"
      +    }
      +  },
      +  "required": [
      +    "tool",
      +    "summary",
      +    "nextTools",
      +    "warnings",
      +    "cost",
      +    "writesFiles",
      +    "sessionPolicy"
      +  ],
      +  "type": "object"
      +}
    • addedOutput schema / properties / result / additionalProperties
      Added value: +{}
    • addedOutput schema / properties / result / properties
      Added value: +{
      +  "designSystem": {
      +    "additionalProperties": {},
      +    "properties": {},
      +    "type": "object"
      +  },
      +  "files": {
      +    "items": {
      +      "type": "string"
      +    },
      +    "type": "array"
      +  },
      +  "maxDepthReached": {
      +    "maximum": 9007199254740991,
      +    "minimum": 0,
      +    "type": "integer"
      +  },
      +  "metadata": {
      +    "additionalProperties": {},
      +    "properties": {},
      +    "type": "object"
      +  },
      +  "outputDir": {
      +    "type": "string"
      +  },
      +  "pages": {
      +    "items": {
      +      "additionalProperties": {},
      +      "properties": {},
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "pagesCrawled": {
      +    "maximum": 9007199254740991,
      +    "minimum": 0,
      +    "type": "integer"
      +  },
      +  "skipped": {
      +    "items": {
      +      "additionalProperties": {},
      +      "properties": {},
      +      "type": "object"
      +    },
      +    "type": "array"
      +  },
      +  "warnings": {
      +    "description": "Non-fatal limitations or incomplete evidence from this run.",
      +    "items": {
      +      "type": "string"
      +    },
      +    "type": "array"
      +  }
      +}
    • addedOutput schema / properties / result / required
      Added value: +[
      +  "metadata",
      +  "pagesCrawled",
      +  "maxDepthReached",
      +  "pages",
      +  "skipped",
      +  "designSystem",
      +  "files",
      +  "warnings"
      +]
    • addedOutput schema / properties / result / type
      Added value: +"object"
    • changedOutput schema / required
      Previous value: -[
      -  "result"
      -]New value: +[
      +  "result",
      +  "guidance"
      +]
  2. Addedv0.10.0

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnlyHint=false and destructiveHint=false. The description adds genuinely useful behavioral context: the operation is heavy, loads several bounded same-origin pages, may write files when an output path is provided, and might require follow-up tools. This goes beyond the annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four compact sentences, with the most decision-relevant guidance (when to use, when to avoid) front-loaded. The cost, session scope, side effects, and follow-up routing are each stated in a single phrase with no filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 18 parameters and an output schema, the description covers what an agent needs to decide and invoke correctly: use-case boundary, scope limits, write behavior, cost, and next steps. The heavy lifting on parameters is done by the fully documented schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema fully documents all 18 parameters. The description adds only broad context about bounded same-origin crawling and conditional file output, which slightly reinforces but does not materially extend the parameter documentation. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('crawl'), a clear resource ('a site's design system'), and the exact purpose: identifying shared versus page-specific design patterns across several pages. It also distinguishes itself from single-page tools by explicitly saying to avoid it when one page is the target.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use ('goal spans several pages'), when-not-to-use ('one page is the target'), and names alternatives ('analyze or create a kit for that page instead'). It even names create_replication_kit as a possible follow-up, which is strong routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.