Skip to main content
Glama

Crawl a whole site once

crawl_site
Idempotent

Crawl a site once from a start URL, following links and the sitemap, and return the pages read. No project is set up. Use it for a site or section whose page URLs you do not know; for known URLs use scrape_urls. To watch the site afterwards, pass the crawl_id to create_project. Inputs: include_paths=['/blog/*'] reads the blog index and everything under it; paths match the URL path, never the host. Excluded pages do not count toward limit. limit=200 with max_depth=3 suits one section. config takes the keys get_project lists when called with no project_id. Cost: 1 to 4 credits per page, by the engine that read it; refused pages are free. Timing: it waits for the crawl, minutes for a large limit. If it outlasts the connection's time limit, a job comes back for get_job, and calling again returns the same crawl. The crawl is kept for a day. Needs write access; a read-only connection is refused with code 'read_only'.

Input Schema

NameRequiredDescriptionDefault
urlYesThe absolute http(s) URL to start from, e.g. the home page.
limitNoThe most pages to read, 1 to 5,000 (default 50); the plan's page cap also applies.
configNoFetch settings to override, as {key: value}; omit for the defaults.
max_depthNoLink hops to follow from the start URL (0 = that page only; default 3).
exclude_pathsNoGlobs over the URL path to skip, e.g. ['/tag/*', '*.pdf']; an exclude wins over an include.
include_pathsNoGlobs over the URL path to keep; a bare '/blog/' matches that one page only. Omit for the whole site.

Output Schema

NameRequiredDescriptionDefault
jobNoWhile it is still going: {kind: 'crawl', id} for get_job and cancel_job.
urlNoThe start URL.
codeNoOnly on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.
noteNoWhat was cut, and how to read a page in full.
stopNoWhy the crawl ended early, e.g. its page limit; empty otherwise.
errorNoOnly when the call failed: what went wrong, in words an assistant can act on.
indexNoEvery page read (up to 500) as {url, title, words, status}.
pagesNoUp to 50 pages, each {url, title, words, status, links (a count), markdown excerpt}.
countsNoPages by outcome, e.g. {ok, blocked, error}.
cursorNoPresent when more pages exist: pass it to get_job for the next window.
detailNoOnly on a failure: the API's own wording, when the error was rephrased.
statusNo'done', 'cancelled' or 'error' once finished, 'running' with a `job` while it goes on. On an error, the HTTP status code.
crawl_idNoThis crawl's id: for get_job, cancel_job, and create_project(crawl_id=...) to keep it.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed4 schema fields changed
    • changedInput schema / properties / config / description
      Previous value: -"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."New value: +"Fetch settings to override, as {key: value}; omit for the defaults."
    • changedInput schema / properties / include_paths / description
      Previous value: -"Globs over the URL path to keep, e.g. ['/blog/*'] for a section (its index included); a bare '/blog/' matches only that one page. Omit for the whole site."New value: +"Globs over the URL path to keep; a bare '/blog/' matches that one page only. Omit for the whole site."
    • changedInput schema / properties / max_depth / description
      Previous value: -"How many links deep to follow from the start URL (0 = that page only; default 3)."New value: +"Link hops to follow from the start URL (0 = that page only; default 3)."
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "description": "A crawl: the pages read, as an index plus excerpts, or a job when it is still going.",
      +  "properties": {
      +    "code": {
      +      "description": "Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.",
      +      "title": "Code"
      +    },
      +    "counts": {
      +      "description": "Pages by outcome, e.g. {ok, blocked, error}.",
      +      "title": "Counts"
      +    },
      +    "crawl_id": {
      +      "description": "This crawl's id: for get_job, cancel_job, and create_project(crawl_id=...) to keep it.",
      +      "title": "Crawl Id"
      +    },
      +    "cursor": {
      +      "description": "Present when more pages exist: pass it to get_job for the next window.",
      +      "title": "Cursor"
      +    },
      +    "detail": {
      +      "description": "Only on a failure: the API's own wording, when the error was rephrased.",
      +      "title": "Detail"
      +    },
      +    "error": {
      +      "description": "Only when the call failed: what went wrong, in words an assistant can act on.",
      +      "title": "Error"
      +    },
      +    "index": {
      +      "description": "Every page read (up to 500) as {url, title, words, status}.",
      +      "title": "Index"
      +    },
      +    "job": {
      +      "description": "While it is still going: {kind: 'crawl', id} for get_job and cancel_job.",
      +      "title": "Job"
      +    },
      +    "note": {
      +      "description": "What was cut, and how to read a page in full.",
      +      "title": "Note"
      +    },
      +    "pages": {
      +      "description": "Up to 50 pages, each {url, title, words, status, links (a count), markdown excerpt}.",
      +      "title": "Pages"
      +    },
      +    "status": {
      +      "description": "'done', 'cancelled' or 'error' once finished, 'running' with a `job` while it goes on. On an error, the HTTP status code.",
      +      "title": "Status"
      +    },
      +    "stop": {
      +      "description": "Why the crawl ended early, e.g. its page limit; empty otherwise.",
      +      "title": "Stop"
      +    },
      +    "url": {
      +      "description": "The start URL.",
      +      "title": "Url"
      +    }
      +  },
      +  "title": "CrawlOut",
      +  "type": "object"
      +}
  2. Changed6 schema fields changed
    • addedInput schema / properties / config / description
      Added value: +"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."
    • addedInput schema / properties / exclude_paths / description
      Added value: +"Globs over the URL path to skip, e.g. ['/tag/*', '*.pdf']; an exclude wins over an include."
    • addedInput schema / properties / include_paths / description
      Added value: +"Globs over the URL path to keep, e.g. ['/blog/*'] for a section (its index included); a bare '/blog/' matches only that one page. Omit for the whole site."
    • addedInput schema / properties / limit / description
      Added value: +"The most pages to read, 1 to 5,000 (default 50); the plan's page cap also applies."
    • addedInput schema / properties / max_depth / description
      Added value: +"How many links deep to follow from the start URL (0 = that page only; default 3)."
    • addedInput schema / properties / url / description
      Added value: +"The absolute http(s) URL to start from, e.g. the home page."
  3. First observed

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial behavior beyond annotations: cost model (1-4 credits per page, refused pages free), timing (waits for the crawl, may hand back a job for get_job), crawl retention (kept for a day), and auth requirements (needs write access, read-only refused with 'read_only'). This is well past what readOnlyHint=false and idempotentHint=true convey. It stops short of a 5 only because the pagination/return-shape handling is left entirely to the output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Dense but deliberately organized into labeled sections (Inputs, Cost, Timing, access), with the highest-value scope statement front-loaded. Every sentence carries information, though the block is longer than strictly necessary for a 6-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-param, open-world, potentially long-running crawl, the description covers scope, alternatives, cost, timing/job fallback, retention, and auth failure mode. Since an output schema exists, it correctly omits return-value detail without leaving any pre-call decision unanswered.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, so the baseline is 3, but the description adds real semantics the schema lacks: paths match the URL path and never the host, an exclude wins over an include, and excluded pages do not count toward limit. It also gives concrete tuning advice (limit=200 with max_depth=3 for one section) and locates valid config keys via get_project.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Crawl a site once from a start URL'), plus the mechanism (following links and the sitemap) and the output ('return the pages read'). It distinguishes itself from scrape_urls and create_project by name, so an agent can pick it apart from siblings without opening schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use ('a site or section whose page URLs you do not know'), explicit alternative ('for known URLs use scrape_urls'), and a follow-on path ('to watch the site afterwards, pass the crawl_id to create_project'). The routing logic is fully spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.