CrawlForge MCP Server

map_site

Read-onlyIdempotent

Discover and list a website's URLs without fetching page bodies, using sitemap.xml when available or link-following otherwise.

Instructions

Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes	The website URL to map
`search`	No	When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]
`max_urls`	No	Maximum number of URLs to discover
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`domain_filter`	No	Per-domain allow/deny lists and URL include/exclude patterns
`group_by_path`	No	Group URLs by path segments
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`include_sitemap`	No	Include sitemap.xml data in results
`include_metadata`	No	Include page metadata for each URL
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`import_filter_config`	No	JSON string of a previously exported domain-filter config

Output Schema

TableJSON Schema

Name	Required	Description
`urls`	No	Flat array of URLs, or grouped-by-path object when group_by_path=true (default)
`view`	No	Whether preview and read_result offsets index a text field or the pretty-printed JSON
`_cost`	No	Cost-transparency metadata (D3.5), present when injected into the text copy of the result
`preview`	No	The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)
`base_url`	No
`metadata`	No	Per-URL metadata when include_metadata=true
`site_map`	No	The shape of the site as counts; `urls` is the one list of URLs
`warnings`	No	Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only, or when the result is over max_inline_chars
`truncated`	No	True when the inline result is a preview
`view_path`	No	Dotted path of the text field the view was cut from; null for the JSON view
`expires_at`	No	When the stored result is dropped (ISO 8601)
`statistics`	No
`total_urls`	No
`ranked_urls`	No	Present only when the `search` param was set
`total_chars`	No	Length of the full view in characters
`filter_stats`	No
`result_handle`	No	Handle for read_result; the full result is kept 1 hour
`domain_filter_config`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed19 schema fields changedv6.19.2Oct 6, 2026

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedOutput schema / properties / expires_at

Added value: +{
+  "description": "When the stored result is dropped (ISO 8601)",
+  "type": "string"
+}

addedOutput schema / properties / preview

Added value: +{
+  "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)",
+  "type": "string"
+}

addedOutput schema / properties / result_handle

Added value: +{
+  "description": "Handle for read_result; the full result is kept 1 hour",
+  "type": "string"
+}

addedOutput schema / properties / site_map / description

Added value: +"The shape of the site as counts; `urls` is the one list of URLs"

addedOutput schema / properties / site_map / properties / depth_levels / additionalProperties / type
```
Added value: +"number"
```
addedOutput schema / properties / site_map / properties / depth_levels / description
```
Added value: +"URLs per path depth"
```
addedOutput schema / properties / site_map / properties / root / description
```
Added value: +"How many URLs are the site root itself"
```
removedOutput schema / properties / site_map / properties / root / items
```
Removed value: -{
-  "type": "string"
-}
```
changedOutput schema / properties / site_map / properties / root / type
```
Previous value: -"array"New value: +"number"
```
addedOutput schema / properties / site_map / properties / sections / additionalProperties / additionalProperties
```
Added value: +{}
```

addedOutput schema / properties / site_map / properties / sections / additionalProperties / properties

Added value: +{
+  "count": {
+    "description": "URLs under this first path segment",
+    "type": "number"
+  },
+  "subsections": {
+    "additionalProperties": {
+      "type": "number"
+    },
+    "description": "URLs per second path segment",
+    "propertyNames": {
+      "type": "string"
+    },
+    "type": "object"
+  }
+}

addedOutput schema / properties / site_map / properties / sections / additionalProperties / type
```
Added value: +"object"
```

addedOutput schema / properties / site_map / properties / sections / description

Added value: +"Counts by first path segment; the URLs themselves are in `urls`"

addedOutput schema / properties / total_chars

Added value: +{
+  "description": "Length of the full view in characters",
+  "type": "number"
+}

addedOutput schema / properties / truncated

Added value: +{
+  "description": "True when the inline result is a preview",
+  "type": "boolean"
+}

addedOutput schema / properties / view

Added value: +{
+  "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON",
+  "enum": [
+    "text",
+    "json"
+  ],
+  "type": "string"
+}

addedOutput schema / properties / view_path

Added value: +{
+  "description": "Dotted path of the text field the view was cut from; null for the JSON view",
+  "type": [
+    "string",
+    "null"
+  ]
+}

changedOutput schema / properties / warnings / description

Previous value: -"Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only"New value: +"Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only, or when the result is over max_inline_chars"

Changed1 schema field changedv6.18.1Oct 3, 2026

addedOutput schema / properties / warnings

Added value: +{
+  "description": "Present when the map is partial, e.g. the start page could not be read and the URLs come from the sitemap only",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

Changed12 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / domain_filter / additionalProperties
```
Removed value: -false
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / metadata / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / ranked_urls / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / site_map / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / site_map / properties / depth_levels / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_map / properties / sections / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / statistics / properties / file_extensions / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / properties / url_lengths / additionalProperties
```
Previous value: -trueNew value: +{}
```

changedOutput schema / properties / urls / anyOf

Previous value: -[
-  {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  {
-    "additionalProperties": {
-      "items": {
-        "type": "string"
-      },
-      "type": "array"
-    },
-    "type": "object"
-  }
-]New value: +[
+  {
+    "items": {
+      "type": "string"
+    },
+    "type": "array"
+  },
+  {
+    "additionalProperties": {
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "propertyNames": {
+      "type": "string"
+    },
+    "type": "object"
+  }
+]

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "base_url": {
+      "type": "string"
+    },
+    "domain_filter_config": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "filter_stats": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "metadata": {
+      "additionalProperties": {},
+      "description": "Per-URL metadata when include_metadata=true",
+      "type": "object"
+    },
+    "ranked_urls": {
+      "description": "Present only when the `search` param was set",
+      "items": {
+        "additionalProperties": true,
+        "properties": {
+          "score": {
+            "type": "number"
+          },
+          "url": {
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "site_map": {
+      "additionalProperties": true,
+      "properties": {
+        "depth_levels": {
+          "additionalProperties": {},
+          "type": "object"
+        },
+        "root": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "sections": {
+          "additionalProperties": {},
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "statistics": {
+      "additionalProperties": true,
+      "properties": {
+        "average_depth": {
+          "type": "number"
+        },
+        "file_extensions": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "max_depth": {
+          "type": "number"
+        },
+        "query_parameters": {
+          "type": "number"
+        },
+        "secure_urls": {
+          "type": "number"
+        },
+        "total_urls": {
+          "type": "number"
+        },
+        "unique_paths": {
+          "type": "number"
+        },
+        "url_lengths": {
+          "additionalProperties": true,
+          "properties": {
+            "average": {
+              "type": "number"
+            },
+            "max": {
+              "type": "number"
+            },
+            "min": {
+              "type": [
+                "number",
+                "null"
+              ]
+            }
+          },
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "total_urls": {
+      "type": "number"
+    },
+    "urls": {
+      "anyOf": [
+        {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        {
+          "additionalProperties": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "type": "object"
+        }
+      ],
+      "description": "Flat array of URLs, or grouped-by-path object when group_by_path=true (default)"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotency, non-destructiveness, and open-world access. The description adds meaningful behavior beyond that: it reads sitemap.xml when available and otherwise follows links, does not fetch page bodies, and costs 2 credits. This is operational context the agent can use when choosing among scraping tools.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: purpose first, then exclusions, then cost, then an example. Every sentence adds useful information, and no space is wasted restating the name or schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex 11-parameter tool with an output schema and full annotation coverage, the description supplies the routing context an agent needs: what it does, what it does not do, how it discovers URLs, and its cost. Return-value details are appropriately left to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all 11 parameters are already documented in the schema. The description mentions include_sitemap and max_urls only in an example and does not add syntax, defaults, or constraints beyond what the schema provides. This meets the baseline of 3 when the schema carries parameter semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: list a site's URLs without fetching page bodies. It also distinguishes the tool from siblings by naming scrape, crawl_deep, and extract_links as wrong alternatives. An agent can identify the correct tool without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-to-use guidance ('list a site's URLs without fetching page bodies') and when-not-to-use guidance ('Not for page content', 'not for the links on one page'), naming the alternatives for each excluded case. The sitemap-first fallback behavior also tells the agent how the tool will go about discovery.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.