Skip to main content
Glama

Crawl a whole site once

crawl_site
Idempotent

Crawl a site once from a start URL, following links and the sitemap, and return the pages read. No project is set up. Use it for a site or section whose page URLs you do not know; for known URLs use scrape_urls. To watch the site afterwards, pass the crawl_id to create_project. Inputs: include_paths=['/blog/*'] reads the blog index and everything under it; paths match the URL path, never the host. Excluded pages do not count toward limit. limit=200 with max_depth=3 suits one section. config takes the keys get_project lists when called with no project_id. Cost: 1 to 4 credits per page, by the engine that read it; refused pages are free. Timing: it waits for the crawl, minutes for a large limit. If it outlasts the connection's time limit, a job comes back for get_job, and calling again returns the same crawl. The crawl is kept for a day. Needs write access; a read-only connection is refused with code 'read_only'.

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "crawl_site"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesThe absolute http(s) URL to start from, e.g. the home page.
limitNoThe most pages to read, 1 to 5,000 (default 50); the plan's page cap also applies.
configNoFetch settings to override, as {key: value}; omit for the defaults.
max_depthNoLink hops to follow from the start URL (0 = that page only; default 3).
exclude_pathsNoGlobs over the URL path to skip, e.g. ['/tag/*', '*.pdf']; an exclude wins over an include.
include_pathsNoGlobs over the URL path to keep; a bare '/blog/' matches that one page only. Omit for the whole site.

Output Schema

NameRequiredDescriptionDefault
jobNoWhile it is still going: {kind: 'crawl', id} for get_job and cancel_job.
urlNoThe start URL.
codeNoOnly on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.
noteNoWhat was cut, and how to read a page in full.
stopNoWhy the crawl ended early, e.g. its page limit; empty otherwise.
errorNoOnly when the call failed: what went wrong, in words an assistant can act on.
indexNoEvery page read (up to 500) as {url, title, words, status}.
pagesNoUp to 50 pages, each {url, title, words, status, links (a count), markdown excerpt}.
countsNoPages by outcome, e.g. {ok, blocked, error}.
cursorNoPresent when more pages exist: pass it to get_job for the next window.
detailNoOnly on a failure: the API's own wording, when the error was rephrased.
noticeNoOnly when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer.
statusNo'done', 'cancelled' or 'error' once finished, 'running' with a `job` while it goes on. On an error, the HTTP status code.
noticesNoWith `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself.
crawl_idNoThis crawl's id: for get_job, cancel_job, and create_project(crawl_id=...) to keep it.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changed
    • changedOutput schema / properties / notices / description
      Previous value: -"With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, and `engine` on a search's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself."New value: +"With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, `engine` on a search's and `outlook` {tier, likely, why} on a page's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself."
  2. Changed2 schema fields changed
    • addedOutput schema / properties / notice
      Added value: +{
      +  "description": "Only when a limit, not the site, held pages back: one line saying what they needed and why (the plan, the stealth trial or the credits left). Tell the person; it comes first in the answer.",
      +  "title": "Notice"
      +}
    • addedOutput schema / properties / notices
      Added value: +{
      +  "description": "With `notice`: one per reason, {reason, pages, needs, examples, unlockedBy, upgradeUrl}, and `engine` on a search's; unlockedBy is the plan that reads them, null for credits or a cap the workspace set itself.",
      +  "title": "Notices"
      +}
  3. Changed4 schema fields changed
    • changedInput schema / properties / config / description
      Previous value: -"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."New value: +"Fetch settings to override, as {key: value}; omit for the defaults."
    • changedInput schema / properties / include_paths / description
      Previous value: -"Globs over the URL path to keep, e.g. ['/blog/*'] for a section (its index included); a bare '/blog/' matches only that one page. Omit for the whole site."New value: +"Globs over the URL path to keep; a bare '/blog/' matches that one page only. Omit for the whole site."
    • changedInput schema / properties / max_depth / description
      Previous value: -"How many links deep to follow from the start URL (0 = that page only; default 3)."New value: +"Link hops to follow from the start URL (0 = that page only; default 3)."
    • changedOutput schema / (root)
      Previous value: -nullNew value: +{
      +  "additionalProperties": true,
      +  "description": "A crawl: the pages read, as an index plus excerpts, or a job when it is still going.",
      +  "properties": {
      +    "code": {
      +      "description": "Only on a failure: a machine-readable reason, e.g. read_only, validation, plan_limit or not_found.",
      +      "title": "Code"
      +    },
      +    "counts": {
      +      "description": "Pages by outcome, e.g. {ok, blocked, error}.",
      +      "title": "Counts"
      +    },
      +    "crawl_id": {
      +      "description": "This crawl's id: for get_job, cancel_job, and create_project(crawl_id=...) to keep it.",
      +      "title": "Crawl Id"
      +    },
      +    "cursor": {
      +      "description": "Present when more pages exist: pass it to get_job for the next window.",
      +      "title": "Cursor"
      +    },
      +    "detail": {
      +      "description": "Only on a failure: the API's own wording, when the error was rephrased.",
      +      "title": "Detail"
      +    },
      +    "error": {
      +      "description": "Only when the call failed: what went wrong, in words an assistant can act on.",
      +      "title": "Error"
      +    },
      +    "index": {
      +      "description": "Every page read (up to 500) as {url, title, words, status}.",
      +      "title": "Index"
      +    },
      +    "job": {
      +      "description": "While it is still going: {kind: 'crawl', id} for get_job and cancel_job.",
      +      "title": "Job"
      +    },
      +    "note": {
      +      "description": "What was cut, and how to read a page in full.",
      +      "title": "Note"
      +    },
      +    "pages": {
      +      "description": "Up to 50 pages, each {url, title, words, status, links (a count), markdown excerpt}.",
      +      "title": "Pages"
      +    },
      +    "status": {
      +      "description": "'done', 'cancelled' or 'error' once finished, 'running' with a `job` while it goes on. On an error, the HTTP status code.",
      +      "title": "Status"
      +    },
      +    "stop": {
      +      "description": "Why the crawl ended early, e.g. its page limit; empty otherwise.",
      +      "title": "Stop"
      +    },
      +    "url": {
      +      "description": "The start URL.",
      +      "title": "Url"
      +    }
      +  },
      +  "title": "CrawlOut",
      +  "type": "object"
      +}
  4. Changed6 schema fields changed
    • addedInput schema / properties / config / description
      Added value: +"Optional fetch settings: any subset of the keys describe_project_config lists, e.g. {\"render_js\": \"always\", \"only_main_content\": true}. Omit for the defaults."
    • addedInput schema / properties / exclude_paths / description
      Added value: +"Globs over the URL path to skip, e.g. ['/tag/*', '*.pdf']; an exclude wins over an include."
    • addedInput schema / properties / include_paths / description
      Added value: +"Globs over the URL path to keep, e.g. ['/blog/*'] for a section (its index included); a bare '/blog/' matches only that one page. Omit for the whole site."
    • addedInput schema / properties / limit / description
      Added value: +"The most pages to read, 1 to 5,000 (default 50); the plan's page cap also applies."
    • addedInput schema / properties / max_depth / description
      Added value: +"How many links deep to follow from the start URL (0 = that page only; default 3)."
    • addedInput schema / properties / url / description
      Added value: +"The absolute http(s) URL to start from, e.g. the home page."
  5. First observed

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations it discloses credit cost (1-4 per page, refused pages free), blocking/timing behavior, the job-handoff fallback to get_job when the connection times out, idempotent re-call semantics, a one-day retention window, and the write-access requirement with the exact 'read_only' refusal code. All of this is consistent with readOnlyHint=false and idempotentHint=true.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action, then routing, then inputs, then labeled Cost/Timing/permission lines; each sentence carries distinct operational information an agent needs before spending credits or waiting on a long crawl.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a six-parameter, long-running, credit-consuming crawl with an output schema already covering return values, the description covers cost, timing, failure modes, permissions, and retention. Nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds real meaning: paths match the URL path and never the host, excluded pages do not count toward limit, limit=200 with max_depth=3 suits a single section, and config keys come from get_project with no project_id. That is genuine added semantics rather than restating field text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource ('Crawl a site once from a start URL, following links and the sitemap') and immediately states the distinguishing condition against a named sibling ('for known URLs use scrape_urls'). An agent can route to this tool versus scrape_urls or map_site without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an explicit when-to-use rule (unknown page URLs) and a when-not rule with the alternative (known URLs -> scrape_urls), plus a follow-up path (pass crawl_id to create_project to keep watching). Nothing about selection is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.