CrawlForge MCP Server

scrape

Read-onlyIdempotent

Fetch any single web page and extract clean Markdown, HTML, text, links, metadata, or JSON in one call. Handles blocked sites with a stealth retry.

Instructions

Use this to read one page - markdown by default, plus any of "html", "rawHtml", "text", "links", "metadata", "branding" (static design tokens: colors, fonts, logo), "screenshot" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:"json",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for "highlights" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it comes back walled does the stealth browser retry and return the page - projected at 7, charged 2 when the plain fetch worked. Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:"https://example.com", formats:["markdown","links","metadata"]})

Input Schema

TableJSON Schema

Name	Required	Description	Default
`url`	Yes	The URL to scrape
`formats`	No	Formats to return (default: ["markdown"])
`escalate`	No	When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false
`timeoutMs`	No	Fetch timeout in ms
`redact_pii`	No	Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`brandingOptions`	No	Options for the "branding" format
`escalate_engine`	No	Stealth engine for the escalated retry: "auto" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), "playwright" (Chromium) or "camoufox"	auto
`onlyMainContent`	No	Strip boilerplate via Readability (default: true)
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`screenshotOptions`	No	Options for the "screenshot" format

Output Schema

TableJSON Schema

Name	Required	Description
`url`	No	Final URL after redirects
`view`	No	Whether preview and read_result offsets index a text field or the pretty-printed JSON
`_cost`	No	Cost-transparency metadata (D3.5), present when injected into the text copy of the result
`error`	No	Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content
`title`	No	Document title; present when success is false
`status`	No	HTTP status of the fetch; present when success is false
`blocked`	No	Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next
`content`	No	One key per requested format
`preview`	No	The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)
`stealth`	No	Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none)
`success`	No	Whether the scrape completed
`warnings`	No	Per-format warnings; partial success never fails the whole call
`escalated`	No	Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price
`redaction`	No	Present when redact_pii was set: what was redacted from the text of this result
`truncated`	No	True when the inline result is a preview
`view_path`	No	Dotted path of the text field the view was cut from; null for the JSON view
`expires_at`	No	When the stored result is dropped (ISO 8601)
`total_chars`	No	Length of the full view in characters
`result_handle`	No	Handle for read_result; the full result is kept 1 hour

Schema Changelog

Changes observed during successful MCP inspections.

Changed3 schema fields changedv6.8.0Sep 23, 2026

changedInput schema / properties / escalate_engine / default
```
Previous value: -"playwright"New value: +"auto"
```

changedInput schema / properties / escalate_engine / description

Previous value: -"Stealth engine for the escalated retry (default: \"playwright\")"New value: +"Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\""

changedInput schema / properties / escalate_engine / enum

Previous value: -[
-  "playwright",
-  "camoufox"
-]New value: +[
+  "auto",
+  "playwright",
+  "camoufox"
+]

Changed33 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / brandingOptions / additionalProperties
```
Removed value: -false
```

addedInput schema / properties / escalate

Added value: +{
+  "default": false,
+  "description": "When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false",
+  "type": "boolean"
+}

addedInput schema / properties / escalate_engine

Added value: +{
+  "default": "playwright",
+  "description": "Stealth engine for the escalated retry (default: \"playwright\")",
+  "enum": [
+    "playwright",
+    "camoufox"
+  ],
+  "type": "string"
+}

changedInput schema / properties / formats / items / anyOf

Previous value: -[
-  {
-    "enum": [
-      "markdown",
-      "html",
-      "rawHtml",
-      "text",
-      "links",
-      "metadata",
-      "screenshot",
-      "branding"
-    ],
-    "type": "string"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "prompt": {
-        "description": "Extraction instruction for the LLM",
-        "type": "string"
-      },
-      "schema": {
-        "additionalProperties": {},
-        "description": "JSON schema for extraction",
-        "type": "object"
-      },
-      "type": {
-        "const": "json",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "enum": [
+      "markdown",
+      "html",
+      "rawHtml",
+      "text",
+      "links",
+      "metadata",
+      "screenshot",
+      "branding"
+    ],
+    "type": "string"
+  },
+  {
+    "properties": {
+      "prompt": {
+        "description": "Extraction instruction for the LLM",
+        "type": "string"
+      },
+      "schema": {
+        "additionalProperties": {},
+        "description": "JSON schema for extraction",
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "type": {
+        "const": "json",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type"
+    ],
+    "type": "object"
+  },
+  {
+    "properties": {
+      "max_highlights": {
+        "default": 10,
+        "description": "How many units to return (default 10)",
+        "maximum": 50,
+        "minimum": 1,
+        "type": "integer"
+      },
+      "mode": {
+        "default": "extractive",
+        "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)",
+        "enum": [
+          "extractive",
+          "model"
+        ],
+        "type": "string"
+      },
+      "query": {
+        "description": "What to look for; the matching sentences, table rows and code blocks come back verbatim with offsets into the markdown",
+        "maxLength": 500,
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "highlights",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "query"
+    ],
+    "type": "object"
+  },
+  {
+    "properties": {
+      "mode": {
+        "default": "extractive",
+        "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)",
+        "enum": [
+          "extractive",
+          "model"
+        ],
+        "type": "string"
+      },
+      "question": {
+        "description": "The question to answer from the page; the evidence units come back verbatim with offsets",
+        "maxLength": 500,
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "question",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "question"
+    ],
+    "type": "object"
+  }
+]

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedInput schema / properties / redact_pii

Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "properties": {
+        "entities": {
+          "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected",
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "mode": {
+          "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)",
+          "enum": [
+            "fast",
+            "model"
+          ],
+          "type": "string"
+        },
+        "replace_style": {
+          "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value",
+          "enum": [
+            "tag",
+            "mask",
+            "remove"
+          ],
+          "type": "string"
+        }
+      },
+      "type": "object"
+    }
+  ],
+  "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off"
+}

removedInput schema / properties / screenshotOptions / additionalProperties
```
Removed value: -false
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / blocked

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next",
+  "properties": {
+    "evidence": {
+      "type": "string"
+    },
+    "vendor": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

changedOutput schema / properties / content / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / content / properties / answer

Added value: +{
+  "additionalProperties": {},
+  "description": "Result of the {type:\"question\"} format",
+  "properties": {
+    "evidence": {
+      "description": "The units the answer rests on, verbatim with offsets",
+      "items": {
+        "additionalProperties": {},
+        "properties": {
+          "kind": {
+            "enum": [
+              "sentence",
+              "table_row",
+              "code_block"
+            ],
+            "type": "string"
+          },
+          "length": {
+            "type": "number"
+          },
+          "offset": {
+            "description": "JS string index into the markdown format of this call",
+            "type": "number"
+          },
+          "score": {
+            "description": "BM25 relevance to the query, higher is better",
+            "type": "number"
+          },
+          "text": {
+            "description": "Verbatim page text: markdown.slice(offset, offset + length) === text",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "grounded": {
+      "description": "True when every number and proper noun in text appears in the evidence or the question; always true in extractive mode",
+      "type": "boolean"
+    },
+    "text": {
+      "description": "Extractive mode: the evidence texts joined; model mode: the model's answer",
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / content / properties / branding / propertyNames
```
Added value: +{
+  "type": "string"
+}
```

addedOutput schema / properties / content / properties / highlights

Added value: +{
+  "description": "Result of the {type:\"highlights\"} format: the units matching the query, best first, verbatim with offsets",
+  "items": {
+    "additionalProperties": {},
+    "properties": {
+      "kind": {
+        "enum": [
+          "sentence",
+          "table_row",
+          "code_block"
+        ],
+        "type": "string"
+      },
+      "length": {
+        "type": "number"
+      },
+      "offset": {
+        "description": "JS string index into the markdown format of this call",
+        "type": "number"
+      },
+      "score": {
+        "description": "BM25 relevance to the query, higher is better",
+        "type": "number"
+      },
+      "text": {
+        "description": "Verbatim page text: markdown.slice(offset, offset + length) === text",
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

changedOutput schema / properties / content / properties / links / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / content / properties / links / properties / links / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / content / properties / metadata / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / content / properties / metadata / properties / og_tags / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / content / properties / metadata / properties / twitter_tags / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / content / properties / screenshots / items / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / error

Added value: +{
+  "description": "Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content",
+  "type": "string"
+}

addedOutput schema / properties / escalated

Added value: +{
+  "description": "Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price",
+  "type": "boolean"
+}

addedOutput schema / properties / expires_at

Added value: +{
+  "description": "When the stored result is dropped (ISO 8601)",
+  "type": "string"
+}

addedOutput schema / properties / preview

Added value: +{
+  "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)",
+  "type": "string"
+}

addedOutput schema / properties / redaction

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when redact_pii was set: what was redacted from the text of this result",
+  "properties": {
+    "count": {
+      "description": "Total spans replaced",
+      "type": "number"
+    },
+    "entities": {
+      "additionalProperties": {
+        "type": "number"
+      },
+      "description": "How many spans were replaced, by entity class; a class with no hits is omitted",
+      "propertyNames": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    "mode": {
+      "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION",
+      "enum": [
+        "fast",
+        "model"
+      ],
+      "type": "string"
+    },
+    "model_ran": {
+      "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged",
+      "type": "boolean"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / result_handle

Added value: +{
+  "description": "Handle for read_result; the full result is kept 1 hour",
+  "type": "string"
+}

addedOutput schema / properties / status

Added value: +{
+  "description": "HTTP status of the fetch; present when success is false",
+  "type": "number"
+}

addedOutput schema / properties / stealth

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none)",
+  "properties": {
+    "engine": {
+      "type": "string"
+    },
+    "vendor_detected": {
+      "type": [
+        "string",
+        "null"
+      ]
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / title

Added value: +{
+  "description": "Document title; present when success is false",
+  "type": "string"
+}

addedOutput schema / properties / total_chars

Added value: +{
+  "description": "Length of the full view in characters",
+  "type": "number"
+}

addedOutput schema / properties / truncated

Added value: +{
+  "description": "True when the inline result is a preview",
+  "type": "boolean"
+}

addedOutput schema / properties / view

Added value: +{
+  "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON",
+  "enum": [
+    "text",
+    "json"
+  ],
+  "type": "string"
+}

addedOutput schema / properties / view_path

Added value: +{
+  "description": "Dotted path of the text field the view was cut from; null for the JSON view",
+  "type": [
+    "string",
+    "null"
+  ]
+}

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "content": {
+      "additionalProperties": true,
+      "description": "One key per requested format",
+      "properties": {
+        "branding": {
+          "additionalProperties": {},
+          "description": "Static design tokens: colors, fonts, logo",
+          "type": "object"
+        },
+        "html": {
+          "type": "string"
+        },
+        "json": {
+          "description": "Result of the {type:\"json\"} format (LLM-structured extraction)"
+        },
+        "links": {
+          "additionalProperties": true,
+          "properties": {
+            "external_count": {
+              "type": "number"
+            },
+            "internal_count": {
+              "type": "number"
+            },
+            "links": {
+              "items": {
+                "additionalProperties": true,
+                "properties": {
+                  "href": {
+                    "type": "string"
+                  },
+                  "is_external": {
+                    "type": "boolean"
+                  },
+                  "original_href": {
+                    "type": "string"
+                  },
+                  "text": {
+                    "type": "string"
+                  }
+                },
+                "type": "object"
+              },
+              "type": "array"
+            },
+            "total_count": {
+              "type": "number"
+            }
+          },
+          "type": "object"
+        },
+        "markdown": {
+          "type": "string"
+        },
+        "metadata": {
+          "additionalProperties": true,
+          "properties": {
+            "author": {
+              "type": "string"
+            },
+            "canonical_url": {
+              "type": "string"
+            },
+            "description": {
+              "type": "string"
+            },
+            "json_ld": {
+              "items": {},
+              "type": "array"
+            },
+            "keywords": {
+              "items": {
+                "type": "string"
+              },
+              "type": "array"
+            },
+            "microdata": {
+              "items": {},
+              "type": "array"
+            },
+            "og_tags": {
+              "additionalProperties": {},
+              "type": "object"
+            },
+            "robots": {
+              "type": "string"
+            },
+            "title": {
+              "type": "string"
+            },
+            "twitter_tags": {
+              "additionalProperties": {},
+              "type": "object"
+            },
+            "url": {
+              "type": "string"
+            },
+            "viewport": {
+              "type": "string"
+            }
+          },
+          "type": "object"
+        },
+        "rawHtml": {
+          "type": "string"
+        },
+        "screenshots": {
+          "description": "Present for the \"screenshot\" format; each item carries a resourceUri once published",
+          "items": {
+            "additionalProperties": true,
+            "properties": {},
+            "type": "object"
+          },
+          "type": "array"
+        },
+        "text": {
+          "type": "string"
+        }
+      },
+      "type": "object"
+    },
+    "success": {
+      "description": "Whether the scrape completed",
+      "type": "boolean"
+    },
+    "url": {
+      "description": "Final URL after redirects",
+      "type": "string"
+    },
+    "warnings": {
+      "description": "Per-format warnings; partial success never fails the whole call",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.9/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark it readOnly, idempotent, and non-destructive, so the description's added behavior is all bonus: onlyMainContent strips boilerplate via Readability, partial format failures do not fail the whole call, escalate runs the plain fetch first then retries in a stealth browser, and it discloses the credit cost. This is far beyond what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but every sentence earns its place: core purpose, format list with semantics, cost, exclusions, escalation behavior, and a concrete example. The most important scoping and usage guidance is front-loaded before the parameter-level detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parameter, nested-object, multi-format tool, the description tells an agent everything needed to select and call it correctly: what it can return, when to prefer it, when not to use it, what escalation does, how partial failures behave, and what it costs. The output schema and rich input schema cover the remaining low-level details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents each parameter. The description adds useful conceptual meaning beyond that: 'all from one fetch', the key difference between the highlight modes ('1 extra credit, no model'), the escalate pricing model ('projected at 7, charged 2 when the plain fetch worked'), and the default main-content stripping. It does not restate parameter mechanics, which is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource ('read one page') and enumerates the exact formats it can return in one fetch. It explicitly names distinguishing exclusions (fetch_url, scrape_with_actions, batch_scrape), which differentiates it from siblings without needing to open their schemas.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use guidance: prefer it over the client's built-in web fetch, request all needed formats in one call instead of fetching then extracting, and set escalate:true for blocking sites. It also states clear when-not-to-use conditions: raw API/JSON bodies, pages needing clicks or login, and multi-URL jobs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.