CrawlForge MCP Server

map_site

Read-onlyIdempotent

Instructions

Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})

Input Schema

Name	Required	Description
`url`	Yes	The website URL to map
`search`	No	When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]
`max_urls`	No	Maximum number of URLs to discover
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`domain_filter`	No	Per-domain allow/deny lists and URL include/exclude patterns
`group_by_path`	No	Group URLs by path segments
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`include_sitemap`	No	Include sitemap.xml data in results
`include_metadata`	No	Include page metadata for each URL
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`import_filter_config`	No	JSON string of a previously exported domain-filter config

Output Schema

Name	Required	Description
`urls`	No	Flat array of URLs, or grouped-by-path object when group_by_path=true (default)
`view`	No	Whether preview and read_result offsets index a text field or the pretty-printed JSON
`_cost`	No	Cost-transparency metadata (D3.5): what this call was projected to cost and what it was charged
`cache`	No	Page cache: whether this call reused a copy of the page fetched earlier in the session
`preview`	No	The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)
`base_url`	No
`metadata`	No	Per-URL metadata when include_metadata=true
`site_map`	No	The shape of the site as counts; `urls` is the one list of URLs
`warnings`	No	Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only, or when the result is over max_inline_chars
`next_step`	No	Failed calls only: the tool or fix to try next
`truncated`	No	True when the inline result is a preview
`view_path`	No	Dotted path of the text field the view was cut from; null for the JSON view
`expires_at`	No	When the stored result is dropped (ISO 8601)
`statistics`	No
`total_urls`	No
`ranked_urls`	No	Present only when the `search` param was set
`total_chars`	No	Length of the full view in characters
`filter_stats`	No
`result_handle`	No	Handle for read_result; the full result is kept 1 hour
`domain_filter_config`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed4 schema fields changedv6.24.0Oct 23, 2026

changedOutput schema / properties / _cost / description

Previous value: -"Cost-transparency metadata (D3.5), present when injected into the text copy of the result"New value: +"Cost-transparency metadata (D3.5): what this call was projected to cost and what it was charged"

changedOutput schema / properties / _cost / properties / actual / description

Previous value: -"Credits actually charged (0 in creator mode, half-rate on error)"New value: +"Credits actually charged (0 in creator mode, and 0 for a failed call)"

addedOutput schema / properties / cache

Added value: +{
+  "additionalProperties": {},
+  "description": "Page cache: whether this call reused a copy of the page fetched earlier in the session",
+  "properties": {
+    "ageMs": {
+      "description": "Age of that copy in ms",
+      "type": [
+        "number",
+        "null"
+      ]
+    },
+    "fetchedAt": {
+      "description": "When the oldest page used was fetched (ISO 8601)",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "note": {
+      "description": "Hits only: how old the copy is and how to force a fresh fetch",
+      "type": "string"
+    },
+    "state": {
+      "description": "hit: every page came from this session's copy; bypass: fetched fresh on request (maxAge 0) or not cacheable; miss: fetched",
+      "enum": [
+        "hit",
+        "miss",
+        "bypass"
+      ],
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / next_step

Added value: +{
+  "description": "Failed calls only: the tool or fix to try next",
+  "type": "string"
+}

Changed19 schema fields changedv6.19.2Oct 6, 2026

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedOutput schema / properties / expires_at

Added value: +{
+  "description": "When the stored result is dropped (ISO 8601)",
+  "type": "string"
+}

addedOutput schema / properties / preview

Added value: +{
+  "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)",
+  "type": "string"
+}

addedOutput schema / properties / result_handle

Added value: +{
+  "description": "Handle for read_result; the full result is kept 1 hour",
+  "type": "string"
+}

addedOutput schema / properties / site_map / description

Added value: +"The shape of the site as counts; `urls` is the one list of URLs"

addedOutput schema / properties / site_map / properties / depth_levels / additionalProperties / type
```
Added value: +"number"
```
addedOutput schema / properties / site_map / properties / depth_levels / description
```
Added value: +"URLs per path depth"
```
addedOutput schema / properties / site_map / properties / root / description
```
Added value: +"How many URLs are the site root itself"
```
removedOutput schema / properties / site_map / properties / root / items
```
Removed value: -{
-  "type": "string"
-}
```
changedOutput schema / properties / site_map / properties / root / type
```
Previous value: -"array"New value: +"number"
```
addedOutput schema / properties / site_map / properties / sections / additionalProperties / additionalProperties
```
Added value: +{}
```

addedOutput schema / properties / site_map / properties / sections / additionalProperties / properties

Added value: +{
+  "count": {
+    "description": "URLs under this first path segment",
+    "type": "number"
+  },
+  "subsections": {
+    "additionalProperties": {
+      "type": "number"
+    },
+    "description": "URLs per second path segment",
+    "propertyNames": {
+      "type": "string"
+    },
+    "type": "object"
+  }
+}

addedOutput schema / properties / site_map / properties / sections / additionalProperties / type
```
Added value: +"object"
```

addedOutput schema / properties / site_map / properties / sections / description

Added value: +"Counts by first path segment; the URLs themselves are in `urls`"

addedOutput schema / properties / total_chars

Added value: +{
+  "description": "Length of the full view in characters",
+  "type": "number"
+}

addedOutput schema / properties / truncated

Added value: +{
+  "description": "True when the inline result is a preview",
+  "type": "boolean"
+}

addedOutput schema / properties / view

Added value: +{
+  "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON",
+  "enum": [
+    "text",
+    "json"
+  ],
+  "type": "string"
+}

addedOutput schema / properties / view_path

Added value: +{
+  "description": "Dotted path of the text field the view was cut from; null for the JSON view",
+  "type": [
+    "string",
+    "null"
+  ]
+}

changedOutput schema / properties / warnings / description

Previous value: -"Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only"New value: +"Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only, or when the result is over max_inline_chars"

Changed1 schema field changedv6.18.1Oct 3, 2026

addedOutput schema / properties / warnings

Added value: +{
+  "description": "Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

Changed12 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / domain_filter / additionalProperties
```
Removed value: -false
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / metadata / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / ranked_urls / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / site_map / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / site_map / properties / depth_levels / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_map / properties / sections / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / statistics / properties / file_extensions / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / properties / url_lengths / additionalProperties
```
Previous value: -trueNew value: +{}
```

changedOutput schema / properties / urls / anyOf

Previous value: -[
-  {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  {
-    "additionalProperties": {
-      "items": {
-        "type": "string"
-      },
-      "type": "array"
-    },
-    "type": "object"
-  }
-]New value: +[
+  {
+    "items": {
+      "type": "string"
+    },
+    "type": "array"
+  },
+  {
+    "additionalProperties": {
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "propertyNames": {
+      "type": "string"
+    },
+    "type": "object"
+  }
+]

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "base_url": {
+      "type": "string"
+    },
+    "domain_filter_config": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "filter_stats": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "metadata": {
+      "additionalProperties": {},
+      "description": "Per-URL metadata when include_metadata=true",
+      "type": "object"
+    },
+    "ranked_urls": {
+      "description": "Present only when the `search` param was set",
+      "items": {
+        "additionalProperties": true,
+        "properties": {
+          "score": {
+            "type": "number"
+          },
+          "url": {
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "site_map": {
+      "additionalProperties": true,
+      "properties": {
+        "depth_levels": {
+          "additionalProperties": {},
+          "type": "object"
+        },
+        "root": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "sections": {
+          "additionalProperties": {},
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "statistics": {
+      "additionalProperties": true,
+      "properties": {
+        "average_depth": {
+          "type": "number"
+        },
+        "file_extensions": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "max_depth": {
+          "type": "number"
+        },
+        "query_parameters": {
+          "type": "number"
+        },
+        "secure_urls": {
+          "type": "number"
+        },
+        "total_urls": {
+          "type": "number"
+        },
+        "unique_paths": {
+          "type": "number"
+        },
+        "url_lengths": {
+          "additionalProperties": true,
+          "properties": {
+            "average": {
+              "type": "number"
+            },
+            "max": {
+              "type": "number"
+            },
+            "min": {
+              "type": [
+                "number",
+                "null"
+              ]
+            }
+          },
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "total_urls": {
+      "type": "number"
+    },
+    "urls": {
+      "anyOf": [
+        {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        {
+          "additionalProperties": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "type": "object"
+        }
+      ],
+      "description": "Flat array of URLs, or grouped-by-path object when group_by_path=true (default)"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, open-world, idempotent, non-destructive behavior, so the bar is lower. The description adds useful operational context: no page-body fetching, sitemap.xml preference with link-following fallback, and a concrete cost of 2 credits. It does not discuss rate limits or large-result handling, but those are covered in schema details.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the primary purpose and exclusions, then adds cost and a compact example. Every sentence carries useful selection or invocation information with no repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the rich schema, output schema, and annotations, the description supplies the critical context an agent needs: purpose, alternatives, non-body-fetching behavior, sitemap fallback, and cost. Parameter and return-value details are appropriately left to the structured schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all 11 parameters are already documented in the input schema. The example illustrates a valid parameter combination (url, include_sitemap, max_urls), but the description does not add syntax or meaning beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('list a site's URLs'), clarifies the mechanism ('without fetching page bodies', sitemap fallback), and explicitly distinguishes itself from sibling tools scrape, crawl_deep, and extract_links.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use and when-not-to-use guidance: use for URL discovery without page bodies; do not use for page content (scrape, crawl_deep) or single-page links (extract_links). The named alternatives leave no inference required.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.