Skip to main content
Glama

Crawl a site into a dataset

writ_crawl_site
Destructive

COLLECTS a site or a section into a queryable, change-tracked dataset (one row per page). Scope: intent (a section, in plain words), rank_cap=N (the top N items of a listing), seed_urls (known pages), or the whole site capped by page_budget. Modes: markdown (default, no AI), extract_mode='schema' (uniform records), executor='ai' + extract_prompt (per-page understanding, 5x the page rate). A bounded crawl answers with its rows; an open one returns a crawl_id for writ_crawl_status wait=true. NOT FOR: reading a few pages now (writ_scrape). Every option: writ_help('writ_crawl_site').

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "writ_crawl_site"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
nameNo
waitNoHold until done and return the rows (default true when bounded).
limitNo
speedNo
deviceNo
intentNoThe section wanted, in plain words; the server derives paths and depth.
outputNo
max_ageNo
save_asNo
delay_msNo
executorNoregular (default, no AI) | ai (5x the page rate).
ocr_modeNo
rank_capNoTop N items of a listing (one page per agent, in parallel).
max_depthNo
seed_urlsNoExact pages to collect (no discovery).
persona_idNo
shard_sizeNo
page_budgetNoCap on pages fetched.
render_modeNo
same_domainNo
content_specNo
extract_modeNomarkdown (default) | schema | html.
exclude_pathsNo
include_pathsNo
preview_charsNo
extract_promptNoWith executor=ai: what to extract from each page.
extract_schemaNoWith extract_mode=schema: CSS {row_selector, fields: [{name, selector, attr?}]}, not a JSON Schema.
respect_robotsNo
timeout_secondsNo
use_residentialNo
allow_subdomainsNo
relevance_thresholdNo
residential_countryNo
max_concurrent_shardsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed24 schema fields changed
    • addedInput schema / properties / allow_subdomains
      Added value: +{
      +  "type": "boolean"
      +}
    • addedInput schema / properties / content_spec
      Added value: +{
      +  "type": "object"
      +}
    • addedInput schema / properties / delay_ms
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / device
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / exclude_paths
      Added value: +{
      +  "items": {
      +    "type": "string"
      +  },
      +  "type": "array"
      +}
    • addedInput schema / properties / extract_schema
      Added value: +{
      +  "description": "With extract_mode=schema: CSS {row_selector, fields: [{name, selector, attr?}]}, not a JSON Schema.",
      +  "type": "object"
      +}
    • addedInput schema / properties / include_paths
      Added value: +{
      +  "items": {
      +    "type": "string"
      +  },
      +  "type": "array"
      +}
    • addedInput schema / properties / limit
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / max_age
      Added value: +{
      +  "minimum": 0,
      +  "type": "integer"
      +}
    • addedInput schema / properties / max_concurrent_shards
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / max_depth
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / name
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / ocr_mode
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / output
      Added value: +{
      +  "type": "object"
      +}
    • addedInput schema / properties / preview_chars
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / relevance_threshold
      Added value: +{
      +  "maximum": 1,
      +  "minimum": 0,
      +  "type": "number"
      +}
    • addedInput schema / properties / render_mode
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / residential_country
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / respect_robots
      Added value: +{
      +  "type": "boolean"
      +}
    • addedInput schema / properties / same_domain
      Added value: +{
      +  "type": "boolean"
      +}
    • addedInput schema / properties / shard_size
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / speed
      Added value: +{
      +  "type": "string"
      +}
    • addedInput schema / properties / timeout_seconds
      Added value: +{
      +  "type": "integer"
      +}
    • addedInput schema / properties / use_residential
      Added value: +{
      +  "type": "boolean"
      +}
  2. Changed35 schema fields changed
    • removedInput schema / properties / allow_subdomains
      Removed value: -{
      -  "type": "boolean"
      -}
    • removedInput schema / properties / content_spec
      Removed value: -{
      -  "description": "Which elements of each page to keep: {preset: 'full'|'main', include_comments: bool, exclude_selectors: [css], include_selectors: [css], keep: {images: bool}}. 'main' = article body only; 'full' = the whole page including comment and discussion threads. Comments, replies and discussion survive only with 'full' plus include_comments; otherwise they are stripped out.",
      -  "type": "object"
      -}
    • removedInput schema / properties / delay_ms
      Removed value: -{
      -  "description": "Politeness delay between fetches per host (default 250).",
      -  "type": "integer"
      -}
    • removedInput schema / properties / device
      Removed value: -{
      -  "description": "A linked Writ desktop's agent_id (writ_devices) to act on. Omitted: the desktop this connection chose with writ_devices action='use', if any.",
      -  "type": "string"
      -}
    • removedInput schema / properties / exclude_paths
      Removed value: -{
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • changedInput schema / properties / executor / description
      Previous value: -"regular (default) = deterministic crawl, no AI. ai = a fleet of AI agents reads every page against `extract_prompt` and returns structured records, for data with no clean CSS selector. Bills at 5x the page rate."New value: +"regular (default, no AI) | ai (5x the page rate)."
    • changedInput schema / properties / extract_mode / description
      Previous value: -"markdown (default) | schema (uniform records via extract_schema) | html (each page's raw HTML, for selectors or embedded JSON)"New value: +"markdown (default) | schema | html."
    • changedInput schema / properties / extract_prompt / description
      Previous value: -"Required with executor=ai: what each agent extracts from each page, in plain language (e.g. 'the product name, price and SKU')."New value: +"With executor=ai: what to extract from each page."
    • removedInput schema / properties / extract_schema
      Removed value: -{
      -  "type": "object"
      -}
    • removedInput schema / properties / include_paths
      Removed value: -{
      -  "items": {
      -    "type": "string"
      -  },
      -  "type": "array"
      -}
    • changedInput schema / properties / intent / description
      Previous value: -"Plain-English goal. The server derives include/exclude paths and a depth from it against a sample of the site's real URLs, and ranks the frontier by relevance, so on an unfamiliar site it scopes better than guessed path regexes."New value: +"The section wanted, in plain words; the server derives paths and depth."
    • removedInput schema / properties / limit
      Removed value: -{
      -  "description": "Rows of collected data to return when wait=true (default 50).",
      -  "type": "integer"
      -}
    • removedInput schema / properties / max_age
      Removed value: -{
      -  "description": "Only meaningful with `save_as`: if that saved crawl already completed within this many seconds, return its collected data instead of crawling again. 0 always crawls.",
      -  "minimum": 0,
      -  "type": "integer"
      -}
    • removedInput schema / properties / max_concurrent_shards
      Removed value: -{
      -  "description": "Explicit parallel-shard cap; overrides the `speed` allocation.",
      -  "type": "integer"
      -}
    • removedInput schema / properties / max_depth
      Removed value: -{
      -  "type": "integer"
      -}
    • removedInput schema / properties / name
      Removed value: -{
      -  "type": "string"
      -}
    • removedInput schema / properties / ocr_mode
      Removed value: -{
      -  "description": "auto (default) | off | force",
      -  "type": "string"
      -}
    • removedInput schema / properties / output
      Removed value: -{
      -  "description": "Response shape, for an answer a program or an API consumes rather than a reader. {shape: 'envelope' (default: Writ's full answer, projected) | 'table' ({columns, rows, total}) | 'records' (bare list of records) | 'record' (the newest record alone: one entity, a usage meter, a dashboard), fields: ['used', 'percent_used as pct', 'items.0.price as first_price'] (ordered pick, renames, dotted paths; a missing path is null, so keys are stable), exclude: ['depth'], include_meta: false (page metadata content_kind/depth/thumbnails is stripped unless true), key: 'usage' (wrap)}. On writ_crawl_site with save_as it is saved as the API's default shape.",
      -  "properties": {
      -    "exclude": {
      -      "items": {
      -        "type": "string"
      -      },
      -      "type": "array"
      -    },
      -    "fields": {
      -      "items": {
      -        "type": "string"
      -      },
      -      "type": "array"
      -    },
      -    "include_meta": {
      -      "type": "boolean"
      -    },
      -    "key": {
      -      "type": "string"
      -    },
      -    "shape": {
      -      "enum": [
      -        "envelope",
      -        "table",
      -        "records",
      -        "record"
      -      ],
      -      "type": "string"
      -    }
      -  },
      -  "type": "object"
      -}
    • addedInput schema / properties / page_budget / description
      Added value: +"Cap on pages fetched."
    • removedInput schema / properties / persona_id / description
      Removed value: -"Saved identity to crawl as (writ_personas lists them), for pages behind a login. Every shard shares the persona's signed-in session and one sticky exit IP. 2FA is minted server-side. A desktop persona ('device:…') crawls on its own desktop, from that machine."
    • removedInput schema / properties / preview_chars
      Removed value: -{
      -  "description": "Cut each inline page's text cells to this many characters (default 12000; 0 = full pages). Cut rows list the fields under `_truncated`; a full page comes from writ_workflow_data(workflow_id=<data_workflow_id>, refs=['<run_id>:<record_index>']).",
      -  "type": "integer"
      -}
    • changedInput schema / properties / rank_cap / description
      Previous value: -"Top N items of a listing (a front page, search results, a category). The server reads the seed page's link order (which is the ranking), seeds exactly those N item pages, and pins the crawl to them, one page per agent, in parallel. Without it the same request is a breadth crawl that mostly collects nav and pagination and does not answer a top-N ask. include_paths, when given, is the item-link shape."New value: +"Top N items of a listing (one page per agent, in parallel)."
    • removedInput schema / properties / relevance_threshold
      Removed value: -{
      -  "description": "0-1. Every discovered page is scored against `intent` and skipped below the bar, so a broad crawl collects only what the goal needs (≈0.3 for 'the pricing and docs pages'). Unset for a whole-site sweep.",
      -  "maximum": 1,
      -  "minimum": 0,
      -  "type": "number"
      -}
    • removedInput schema / properties / render_mode
      Removed value: -{
      -  "description": "How each page is fetched, independent of `executor`, which decides who reads it. auto (default) = plain HTTP first, warm browser only for JS-challenge or near-empty pages; http = never opens a browser (fastest, static HTML); browser = warm-render every page (JS/SPA sites). executor=ai works on either lane.",
      -  "type": "string"
      -}
    • removedInput schema / properties / residential_country
      Removed value: -{
      -  "description": "Two-letter ISO country the residential exit should be in (e.g. 'us', 'fr'): pins the exit pool's geo for every shard. Omitted = an automatic exit. Ignored unless the session egresses residential.",
      -  "type": "string"
      -}
    • removedInput schema / properties / respect_robots
      Removed value: -{
      -  "description": "Honor robots.txt (default true).",
      -  "type": "boolean"
      -}
    • removedInput schema / properties / same_domain
      Removed value: -{
      -  "type": "boolean"
      -}
    • removedInput schema / properties / save_as / description
      Removed value: -"Saves these settings as a named, callable crawl, for one that is re-run later (a recurring pull, an API the user asked for). Saved crawls are listed to every future session as 'already collected', so a saved one-off reads as reusable data to later sessions. Reusing the same name updates that saved crawl instead of creating a duplicate."
    • changedInput schema / properties / seed_urls / description
      Previous value: -"Exact known pages to start from: the crawl collects these instead of discovering its own. The cheapest way to scrape a known set."New value: +"Exact pages to collect (no discovery)."
    • removedInput schema / properties / shard_size
      Removed value: -{
      -  "description": "URLs fetched per shard batch (default 20).",
      -  "type": "integer"
      -}
    • removedInput schema / properties / speed
      Removed value: -{
      -  "description": "Throughput tier: slow | normal (default) | fast: the share of the parallel-agent allowance the crawl uses. slow ≈ ¼ at a discounted page rate, normal ≈ ½ at standard rate, fast = all of it at a premium.",
      -  "type": "string"
      -}
    • removedInput schema / properties / timeout_seconds
      Removed value: -{
      -  "description": "Max seconds to hold when wait=true (≤75).",
      -  "type": "integer"
      -}
    • removedInput schema / properties / url / description
      Removed value: -"Seed URL (required)."
    • removedInput schema / properties / use_residential
      Removed value: -{
      -  "description": "Route every shard through the platform residential network (premium), for sites that block datacenter IPs / show a bot wall; a persona crawl forces it on automatically. Costs residential bandwidth; default off.",
      -  "type": "boolean"
      -}
    • changedInput schema / properties / wait / description
      Previous value: -"Hold until the crawl converges and return the collected pages in this call (`data.rows`). Default true for a bounded crawl (rank_cap or seed_urls: a few pages, seconds) and false for an open site crawl (returns a crawl id for writ_crawl_status). Past the 75s ceiling the answer is a 504 that still carries the crawl id."New value: +"Hold until done and return the rows (default true when bounded)."
  3. First observed

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations it discloses cost (executor='ai' at 5x the page rate), the default no-AI path, the bounded-vs-open return contract (rows vs crawl_id) and the page_budget cap. It does not address the persistence/destructive side flagged by destructiveHint=true — e.g. whether a repeat crawl overwrites an existing dataset or what gets replaced — which is the one meaningful gap for a write-flagged tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then scope, then modes, then return contract, then exclusions — a logical order with no filler sentences. The capitalized keywords and label:value shorthand make it dense but scannable; it is near the upper limit of density for a single paragraph.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 35-param, open-world, non-idempotent crawl tool with no output schema, it covers the critical decisions: how scope is chosen, which mode costs more, and what comes back in each case. It is incomplete on the dataset destination/naming params and on the destructive semantics of repeat crawls, which is the remaining risk an agent faces before calling.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 35 parameters and only 26% schema coverage, the description must carry the load; it adds real meaning for the scope-driving params (intent, rank_cap, seed_urls, page_budget) and the mode params (executor, extract_mode, extract_prompt). However, roughly two dozen options — render_mode, ocr_mode, speed, device, shard_size, same_domain, include/exclude_paths, save_as, name, timeout — are undocumented in both schema and description, deferred wholesale to writ_help.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"COLLECTS a site or a section into a queryable, change-tracked dataset (one row per page)" gives a specific verb, resource, and output shape, plus it explicitly carves out the adjacent tool ("NOT FOR: reading a few pages now (writ_scrape)"). An agent can distinguish this from writ_scrape and writ_crawl_status without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It enumerates the four scope selection mechanisms (intent, rank_cap, seed_urls, whole site with page_budget) with the condition for each, names the mutually exclusive reading tool, and routes to writ_crawl_status for the async case. The alternatives and the when-not are both stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.