CrawlForge MCP Server

map_site

Read-onlyIdempotent

Discover a website's URLs by reading sitemap.xml or following links, returning a clean site map without fetching page bodies. Ideal for site audits and navigation planning.

Instructions

Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})

Input Schema

TableJSON Schema

Name	Required	Description
`url`	Yes	The website URL to map
`search`	No	When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]
`max_urls`	No	Maximum number of URLs to discover
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`domain_filter`	No	Per-domain allow/deny lists and URL include/exclude patterns
`group_by_path`	No	Group URLs by path segments
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`include_sitemap`	No	Include sitemap.xml data in results
`include_metadata`	No	Include page metadata for each URL
`import_filter_config`	No	JSON string of a previously exported domain-filter config

Output Schema

TableJSON Schema

Name	Required	Description
`urls`	No	Flat array of URLs, or grouped-by-path object when group_by_path=true (default)
`_cost`	No	Cost-transparency metadata (D3.5), present when injected into the text copy of the result
`base_url`	No
`metadata`	No	Per-URL metadata when include_metadata=true
`site_map`	No
`statistics`	No
`total_urls`	No
`ranked_urls`	No	Present only when the `search` param was set
`filter_stats`	No
`domain_filter_config`	No

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

Changed12 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / domain_filter / additionalProperties
```
Removed value: -false
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / metadata / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / ranked_urls / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / site_map / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / site_map / properties / depth_levels / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_map / properties / sections / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / statistics / properties / file_extensions / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / statistics / properties / url_lengths / additionalProperties
```
Previous value: -trueNew value: +{}
```

changedOutput schema / properties / urls / anyOf

Previous value: -[
-  {
-    "items": {
-      "type": "string"
-    },
-    "type": "array"
-  },
-  {
-    "additionalProperties": {
-      "items": {
-        "type": "string"
-      },
-      "type": "array"
-    },
-    "type": "object"
-  }
-]New value: +[
+  {
+    "items": {
+      "type": "string"
+    },
+    "type": "array"
+  },
+  {
+    "additionalProperties": {
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "propertyNames": {
+      "type": "string"
+    },
+    "type": "object"
+  }
+]

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "base_url": {
+      "type": "string"
+    },
+    "domain_filter_config": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "filter_stats": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "metadata": {
+      "additionalProperties": {},
+      "description": "Per-URL metadata when include_metadata=true",
+      "type": "object"
+    },
+    "ranked_urls": {
+      "description": "Present only when the `search` param was set",
+      "items": {
+        "additionalProperties": true,
+        "properties": {
+          "score": {
+            "type": "number"
+          },
+          "url": {
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "site_map": {
+      "additionalProperties": true,
+      "properties": {
+        "depth_levels": {
+          "additionalProperties": {},
+          "type": "object"
+        },
+        "root": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "sections": {
+          "additionalProperties": {},
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "statistics": {
+      "additionalProperties": true,
+      "properties": {
+        "average_depth": {
+          "type": "number"
+        },
+        "file_extensions": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "max_depth": {
+          "type": "number"
+        },
+        "query_parameters": {
+          "type": "number"
+        },
+        "secure_urls": {
+          "type": "number"
+        },
+        "total_urls": {
+          "type": "number"
+        },
+        "unique_paths": {
+          "type": "number"
+        },
+        "url_lengths": {
+          "additionalProperties": true,
+          "properties": {
+            "average": {
+              "type": "number"
+            },
+            "max": {
+              "type": "number"
+            },
+            "min": {
+              "type": [
+                "number",
+                "null"
+              ]
+            }
+          },
+          "type": "object"
+        }
+      },
+      "type": "object"
+    },
+    "total_urls": {
+      "type": "number"
+    },
+    "urls": {
+      "anyOf": [
+        {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        {
+          "additionalProperties": {
+            "items": {
+              "type": "string"
+            },
+            "type": "array"
+          },
+          "type": "object"
+        }
+      ],
+      "description": "Flat array of URLs, or grouped-by-path object when group_by_path=true (default)"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds behavioral detail beyond annotations: it 'reads sitemap.xml when available, otherwise follows links' and notes the 2-credit cost, which helps set expectations about how the tool operates and what it costs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the core purpose, followed by sibling exclusions, cost, and an example. It is slightly longer than strictly necessary because the example overlaps somewhat with the schema, but every sentence earns its place and none merely restate the name or schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter tool with an output schema and rich annotations, the description covers the essential context: what the tool returns conceptually, when to use alternatives, the cost, and an example. It does not explain the domain_filter or search parameters, but those are fully documented in the schema, so the description is complete enough for routing and correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all 10 parameters. The description adds minimal parameter semantics beyond the schema, mostly via the example invocation that references include_sitemap and max_urls. This matches the baseline of 3 when the schema carries the parameter documentation burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'list a site's URLs without fetching page bodies.' It explicitly distinguishes itself from siblings by naming what it is not for ('not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links)'). An agent can immediately tell this apart from scrape, crawl_deep, and extract_links.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives explicit when-to-use guidance ('list a site's URLs without fetching page bodies') and explicit exclusions with named alternatives. It also includes cost and a concrete example, which helps an agent decide if invoking this tool is appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/mysleekdesigns/crawlforge-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server