CrawlForge MCP Server

crawl_deep

Read-only

Instructions

Use this to fetch many pages of one site by following links - a knowledge base, a docs index, a full-site audit. Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper). Cost: 4 credits base, grows with page count. Example: crawl_deep({url: "https://docs.example.com", max_depth: 3, max_pages: 200, extract_content: true})

Input Schema

Name	Required	Description
`url`	Yes	Starting URL for the crawl
`session`	No	Shared cookie-jar/session for login-then-crawl workflows
`max_depth`	No	Maximum crawl depth from starting URL
`max_pages`	No	Maximum number of pages to crawl
`redact_pii`	No	Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off
`concurrency`	No	Number of concurrent requests
`domain_filter`	No	Per-domain allow/deny lists and crawl rules
`respect_robots`	No	Respect robots.txt directives
`extract_content`	No	Extract page content during crawl
`follow_external`	No	Follow links to external domains
`exclude_patterns`	No	URL patterns to exclude (regex)
`include_patterns`	No	URL patterns to include (regex)
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`content_max_length`	No	Maximum characters of page content to include per page (default 500); sets a truncated flag when trimmed
`enable_link_analysis`	No	Compute PageRank/link-graph analysis over crawled pages
`import_filter_config`	No	JSON string of a previously exported domain-filter config
`link_analysis_options`	No	PageRank tuning options

Output Schema

Name	Required	Description
`url`	No
`view`	No	Whether preview and read_result offsets index a text field or the pretty-printed JSON
`_cost`	No	Cost-transparency metadata (D3.5): what this call was projected to cost and what it was charged
`cache`	No	Page cache: whether this call reused a copy of the page fetched earlier in the session
`error`	No
`stats`	No
`cached`	No	True when this response was replayed from an earlier crawl rather than crawled now; crawled_at gives its age
`errors`	No
`preview`	No	The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)
`results`	No
`session`	No
`success`	No	False only when the crawl was cancelled via elicitation decline
`warnings`	No	Notes on this result; over max_inline_chars, where the full result is kept and how to read it
`next_step`	No	Failed calls only: the tool or fix to try next
`redaction`	No	Present when redact_pii was set: what was redacted from the text of this result
`truncated`	No	True when the inline result is a preview
`view_path`	No	Dotted path of the text field the view was cut from; null for the JSON view
`crawled_at`	No	When the pages were actually fetched (ISO 8601)
`expires_at`	No	When the stored result is dropped (ISO 8601)
`crawl_depth`	No
`duration_ms`	No
`error_count`	No	URLs that failed; one entry each in errors[]
`pages_found`	No	Same as pages_crawled; kept for existing clients
`total_chars`	No	Length of the full view in characters
`link_analysis`	No
`pages_crawled`	No	Pages fetched and returned in results
`result_handle`	No	Handle for read_result; the full result is kept 1 hour
`site_structure`	No
`pages_attempted`	No	URLs the crawl tried: pages_crawled + error_count
`pages_per_second`	No
`domain_filter_config`	No

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedv6.24.0Oct 23, 2026

changedOutput schema / properties / _cost / description

Previous value: -"Cost-transparency metadata (D3.5), present when injected into the text copy of the result"New value: +"Cost-transparency metadata (D3.5): what this call was projected to cost and what it was charged"

changedOutput schema / properties / _cost / properties / actual / description

Previous value: -"Credits actually charged (0 in creator mode, half-rate on error)"New value: +"Credits actually charged (0 in creator mode, and 0 for a failed call)"

addedOutput schema / properties / cache

Added value: +{
+  "additionalProperties": {},
+  "description": "Page cache: whether this call reused a copy of the page fetched earlier in the session",
+  "properties": {
+    "ageMs": {
+      "description": "Age of that copy in ms",
+      "type": [
+        "number",
+        "null"
+      ]
+    },
+    "fetchedAt": {
+      "description": "When the oldest page used was fetched (ISO 8601)",
+      "type": [
+        "string",
+        "null"
+      ]
+    },
+    "note": {
+      "description": "Hits only: how old the copy is and how to force a fresh fetch",
+      "type": "string"
+    },
+    "state": {
+      "description": "hit: every page came from this session's copy; bypass: fetched fresh on request (maxAge 0) or not cacheable; miss: fetched",
+      "enum": [
+        "hit",
+        "miss",
+        "bypass"
+      ],
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / next_step

Added value: +{
+  "description": "Failed calls only: the tool or fix to try next",
+  "type": "string"
+}

addedOutput schema / properties / warnings

Added value: +{
+  "description": "Notes on this result; over max_inline_chars, where the full result is kept and how to read it",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}

Changed5 schema fields changedv6.18.1Oct 3, 2026

addedOutput schema / properties / error_count / description

Added value: +"URLs that failed; one entry each in errors[]"

addedOutput schema / properties / pages_attempted

Added value: +{
+  "description": "URLs the crawl tried: pages_crawled + error_count",
+  "type": "number"
+}

addedOutput schema / properties / pages_crawled / description
```
Added value: +"Pages fetched and returned in results"
```

addedOutput schema / properties / pages_found / description

Added value: +"Same as pages_crawled; kept for existing clients"

addedOutput schema / properties / site_structure / properties / total_pages / description
```
Added value: +"Equals pages_crawled; failed URLs are not counted"
```

Changed28 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / domain_filter / additionalProperties
```
Removed value: -false
```
addedInput schema / properties / domain_filter / properties / blacklist / items
```
Added value: +{}
```
addedInput schema / properties / domain_filter / properties / domain_rules / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedInput schema / properties / domain_filter / properties / whitelist / items
```
Added value: +{}
```
removedInput schema / properties / link_analysis_options / additionalProperties
```
Removed value: -false
```

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedInput schema / properties / redact_pii

Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "properties": {
+        "entities": {
+          "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected",
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "mode": {
+          "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)",
+          "enum": [
+            "fast",
+            "model"
+          ],
+          "type": "string"
+        },
+        "replace_style": {
+          "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value",
+          "enum": [
+            "tag",
+            "mask",
+            "remove"
+          ],
+          "type": "string"
+        }
+      },
+      "type": "object"
+    }
+  ],
+  "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off"
+}

removedInput schema / properties / session / additionalProperties
```
Removed value: -false
```
addedInput schema / properties / session / properties / headers / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
removedInput schema / properties / session / properties / initialRequest / additionalProperties
```
Removed value: -false
```
addedInput schema / properties / session / properties / initialRequest / properties / headers / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / expires_at

Added value: +{
+  "description": "When the stored result is dropped (ISO 8601)",
+  "type": "string"
+}

addedOutput schema / properties / preview

Added value: +{
+  "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)",
+  "type": "string"
+}

addedOutput schema / properties / redaction

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when redact_pii was set: what was redacted from the text of this result",
+  "properties": {
+    "count": {
+      "description": "Total spans replaced",
+      "type": "number"
+    },
+    "entities": {
+      "additionalProperties": {
+        "type": "number"
+      },
+      "description": "How many spans were replaced, by entity class; a class with no hits is omitted",
+      "propertyNames": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    "mode": {
+      "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION",
+      "enum": [
+        "fast",
+        "model"
+      ],
+      "type": "string"
+    },
+    "model_ran": {
+      "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged",
+      "type": "boolean"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / result_handle

Added value: +{
+  "description": "Handle for read_result; the full result is kept 1 hour",
+  "type": "string"
+}

changedOutput schema / properties / results / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / session / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / site_structure / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / site_structure / properties / depth_distribution / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_structure / properties / file_types / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_structure / properties / path_depth_distribution / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / site_structure / properties / path_patterns / propertyNames
```
Added value: +{
+  "type": "string"
+}
```

addedOutput schema / properties / total_chars

Added value: +{
+  "description": "Length of the full view in characters",
+  "type": "number"
+}

addedOutput schema / properties / truncated

Added value: +{
+  "description": "True when the inline result is a preview",
+  "type": "boolean"
+}

addedOutput schema / properties / view

Added value: +{
+  "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON",
+  "enum": [
+    "text",
+    "json"
+  ],
+  "type": "string"
+}

addedOutput schema / properties / view_path

Added value: +{
+  "description": "Dotted path of the text field the view was cut from; null for the JSON view",
+  "type": [
+    "string",
+    "null"
+  ]
+}

Changed2 schema fields changedv5.4.0Aug 29, 2026
- addedOutput schema / properties / site_structure / properties / depth_distribution / description
```
Added value: +"Pages per crawl depth (links from the start URL)"
```
- addedOutput schema / properties / site_structure / properties / path_depth_distribution
```
Added value: +{
+  "additionalProperties": {
+    "type": "number"
+  },
+  "description": "Pages per URL path-segment depth",
+  "type": "object"
+}
```

Changed2 schema fields changedv5.1.0Aug 26, 2026

addedOutput schema / properties / cached

Added value: +{
+  "description": "True when this response was replayed from an earlier crawl rather than crawled now; crawled_at gives its age",
+  "type": "boolean"
+}

addedOutput schema / properties / crawled_at

Added value: +{
+  "description": "When the pages were actually fetched (ISO 8601)",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "crawl_depth": {
+      "type": "number"
+    },
+    "domain_filter_config": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "duration_ms": {
+      "type": "number"
+    },
+    "error": {
+      "type": "string"
+    },
+    "error_count": {
+      "type": "number"
+    },
+    "errors": {
+      "items": {},
+      "type": "array"
+    },
+    "link_analysis": {
+      "anyOf": [
+        {},
+        {
+          "type": "null"
+        }
+      ]
+    },
+    "pages_crawled": {
+      "type": "number"
+    },
+    "pages_found": {
+      "type": "number"
+    },
+    "pages_per_second": {
+      "type": "number"
+    },
+    "results": {
+      "items": {
+        "additionalProperties": true,
+        "properties": {
+          "content": {
+            "type": "string"
+          },
+          "content_length": {
+            "type": "number"
+          },
+          "depth": {
+            "type": "number"
+          },
+          "links_count": {
+            "type": "number"
+          },
+          "metadata": {},
+          "timestamp": {
+            "type": [
+              "string",
+              "number"
+            ]
+          },
+          "title": {
+            "type": "string"
+          },
+          "truncated": {
+            "type": "boolean"
+          },
+          "url": {
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "session": {
+      "additionalProperties": true,
+      "properties": {
+        "cookies_captured": {
+          "type": "number"
+        },
+        "enabled": {
+          "type": "boolean"
+        }
+      },
+      "type": "object"
+    },
+    "site_structure": {
+      "additionalProperties": true,
+      "properties": {
+        "depth_distribution": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "file_types": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "path_patterns": {
+          "additionalProperties": {
+            "type": "number"
+          },
+          "type": "object"
+        },
+        "subdomains": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "total_pages": {
+          "type": "number"
+        }
+      },
+      "type": "object"
+    },
+    "stats": {},
+    "success": {
+      "description": "False only when the crawl was cancelled via elicitation decline",
+      "type": "boolean"
+    },
+    "url": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, openWorld, non-idempotent, non-destructive, so the safety profile is covered. The description adds genuinely new information by disclosing the credit price and that cost grows with page count, but says nothing about crawl duration, rate limiting, or that huge results spill into a result_handle (that detail lives only in the schema).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tightly packed sentences: purpose, exclusions with alternatives, cost, then an example. Front-loaded and no filler; every clause carries routing or pricing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 17-parameter, nested-schema tool with an output schema and full annotations, the description covers the decision-relevant essentials: scope, alternatives, cost, and a usage example. Minor gap: it never hints at the login/session workflow or the preview-plus-handle return path, though both are documented in the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so a 3 is the baseline; the description earns a bump by showing a realistic invocation that anchors typical values for max_depth, max_pages, and extract_content on a docs site. It does not touch the session, redaction, or inline-limit parameters, which is acceptable given the schema already documents them fully.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource (fetch many pages of one site by following links) plus concrete scenarios (knowledge base, docs index, full-site audit). It also explicitly sets itself apart from scrape, batch_scrape, and map_site, so an agent can route without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use framing and three named alternatives with the condition that selects each ('Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper)'). The cost comparison to map_site further aids the decision.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.