CrawlForge MCP Server

scrape

Read-onlyIdempotent

Fetch a web page and extract content as markdown, HTML, text, links, metadata, screenshots, or structured JSON. Supports targeted highlights and direct question answering from the page.

Instructions

Use this to read one page - markdown by default, plus any of "html", "rawHtml", "text", "links", "metadata", "branding" (static design tokens: colors, fonts, logo), "screenshot" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:"json",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for "highlights" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it comes back walled does the stealth browser retry and return the page - projected at 7, charged 2 when the plain fetch worked. Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:"https://example.com", formats:["markdown","links","metadata"]})

Input Schema

TableJSON Schema

Name	Required	Description	Default
`url`	Yes	The URL to scrape
`formats`	No	Formats to return (default: ["markdown"])
`escalate`	No	When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false
`timeoutMs`	No	Fetch timeout in ms
`redact_pii`	No	Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`brandingOptions`	No	Options for the "branding" format
`escalate_engine`	No	Stealth engine for the escalated retry (default: "playwright")	playwright
`onlyMainContent`	No	Strip boilerplate via Readability (default: true)
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`screenshotOptions`	No	Options for the "screenshot" format

Output Schema

TableJSON Schema

Name	Required	Description
`url`	No	Final URL after redirects
`view`	No	Whether preview and read_result offsets index a text field or the pretty-printed JSON
`_cost`	No	Cost-transparency metadata (D3.5), present when injected into the text copy of the result
`error`	No	Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content
`title`	No	Document title; present when success is false
`status`	No	HTTP status of the fetch; present when success is false
`blocked`	No	Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next
`content`	No	One key per requested format
`preview`	No	The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)
`stealth`	No	Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none)
`success`	No	Whether the scrape completed
`warnings`	No	Per-format warnings; partial success never fails the whole call
`escalated`	No	Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price
`redaction`	No	Present when redact_pii was set: what was redacted from the text of this result
`truncated`	No	True when the inline result is a preview
`view_path`	No	Dotted path of the text field the view was cut from; null for the JSON view
`expires_at`	No	When the stored result is dropped (ISO 8601)
`total_chars`	No	Length of the full view in characters
`result_handle`	No	Handle for read_result; the full result is kept 1 hour

Schema Changelog

Changes observed during successful MCP inspections. Dates show when Glama detected each change.

Changed33 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
removedInput schema / properties / brandingOptions / additionalProperties
```
Removed value: -false
```

addedInput schema / properties / escalate

Added value: +{
+  "default": false,
+  "description": "When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false",
+  "type": "boolean"
+}

addedInput schema / properties / escalate_engine

Added value: +{
+  "default": "playwright",
+  "description": "Stealth engine for the escalated retry (default: \"playwright\")",
+  "enum": [
+    "playwright",
+    "camoufox"
+  ],
+  "type": "string"
+}

changedInput schema / properties / formats / items / anyOf

Previous value: -[
-  {
-    "enum": [
-      "markdown",
-      "html",
-      "rawHtml",
-      "text",
-      "links",
-      "metadata",
-      "screenshot",
-      "branding"
-    ],
-    "type": "string"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "prompt": {
-        "description": "Extraction instruction for the LLM",
-        "type": "string"
-      },
-      "schema": {
-        "additionalProperties": {},
-        "description": "JSON schema for extraction",
-        "type": "object"
-      },
-      "type": {
-        "const": "json",
-        "type": "string"
-      }
-    },
-    "required": [
-      "type"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "enum": [
+      "markdown",
+      "html",
+      "rawHtml",
+      "text",
+      "links",
+      "metadata",
+      "screenshot",
+      "branding"
+    ],
+    "type": "string"
+  },
+  {
+    "properties": {
+      "prompt": {
+        "description": "Extraction instruction for the LLM",
+        "type": "string"
+      },
+      "schema": {
+        "additionalProperties": {},
+        "description": "JSON schema for extraction",
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "type": {
+        "const": "json",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type"
+    ],
+    "type": "object"
+  },
+  {
+    "properties": {
+      "max_highlights": {
+        "default": 10,
+        "description": "How many units to return (default 10)",
+        "maximum": 50,
+        "minimum": 1,
+        "type": "integer"
+      },
+      "mode": {
+        "default": "extractive",
+        "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)",
+        "enum": [
+          "extractive",
+          "model"
+        ],
+        "type": "string"
+      },
+      "query": {
+        "description": "What to look for; the matching sentences, table rows and code blocks come back verbatim with offsets into the markdown",
+        "maxLength": 500,
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "highlights",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "query"
+    ],
+    "type": "object"
+  },
+  {
+    "properties": {
+      "mode": {
+        "default": "extractive",
+        "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)",
+        "enum": [
+          "extractive",
+          "model"
+        ],
+        "type": "string"
+      },
+      "question": {
+        "description": "The question to answer from the page; the evidence units come back verbatim with offsets",
+        "maxLength": 500,
+        "minLength": 1,
+        "type": "string"
+      },
+      "type": {
+        "const": "question",
+        "type": "string"
+      }
+    },
+    "required": [
+      "type",
+      "question"
+    ],
+    "type": "object"
+  }
+]

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedInput schema / properties / redact_pii

Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "properties": {
+        "entities": {
+          "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected",
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "mode": {
+          "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)",
+          "enum": [
+            "fast",
+            "model"
+          ],
+          "type": "string"
+        },
+        "replace_style": {
+          "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value",
+          "enum": [
+            "tag",
+            "mask",
+            "remove"
+          ],
+          "type": "string"
+        }
+      },
+      "type": "object"
+    }
+  ],
+  "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off"
+}

removedInput schema / properties / screenshotOptions / additionalProperties
```
Removed value: -false
```
changedOutput schema / properties / _cost / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / blocked

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next",
+  "properties": {
+    "evidence": {
+      "type": "string"
+    },
+    "vendor": {
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

changedOutput schema / properties / content / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / content / properties / answer

Added value: +{
+  "additionalProperties": {},
+  "description": "Result of the {type:\"question\"} format",
+  "properties": {
+    "evidence": {
+      "description": "The units the answer rests on, verbatim with offsets",
+      "items": {
+        "additionalProperties": {},
+        "properties": {
+          "kind": {
+            "enum": [
+              "sentence",
+              "table_row",
+              "code_block"
+            ],
+            "type": "string"
+          },
+          "length": {
+            "type": "number"
+          },
+          "offset": {
+            "description": "JS string index into the markdown format of this call",
+            "type": "number"
+          },
+          "score": {
+            "description": "BM25 relevance to the query, higher is better",
+            "type": "number"
+          },
+          "text": {
+            "description": "Verbatim page text: markdown.slice(offset, offset + length) === text",
+            "type": "string"
+          }
+        },
+        "type": "object"
+      },
+      "type": "array"
+    },
+    "grounded": {
+      "description": "True when every number and proper noun in text appears in the evidence or the question; always true in extractive mode",
+      "type": "boolean"
+    },
+    "text": {
+      "description": "Extractive mode: the evidence texts joined; model mode: the model's answer",
+      "type": "string"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / content / properties / branding / propertyNames
```
Added value: +{
+  "type": "string"
+}
```

addedOutput schema / properties / content / properties / highlights

Added value: +{
+  "description": "Result of the {type:\"highlights\"} format: the units matching the query, best first, verbatim with offsets",
+  "items": {
+    "additionalProperties": {},
+    "properties": {
+      "kind": {
+        "enum": [
+          "sentence",
+          "table_row",
+          "code_block"
+        ],
+        "type": "string"
+      },
+      "length": {
+        "type": "number"
+      },
+      "offset": {
+        "description": "JS string index into the markdown format of this call",
+        "type": "number"
+      },
+      "score": {
+        "description": "BM25 relevance to the query, higher is better",
+        "type": "number"
+      },
+      "text": {
+        "description": "Verbatim page text: markdown.slice(offset, offset + length) === text",
+        "type": "string"
+      }
+    },
+    "type": "object"
+  },
+  "type": "array"
+}

changedOutput schema / properties / content / properties / links / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / content / properties / links / properties / links / items / additionalProperties
```
Previous value: -trueNew value: +{}
```
changedOutput schema / properties / content / properties / metadata / additionalProperties
```
Previous value: -trueNew value: +{}
```
addedOutput schema / properties / content / properties / metadata / properties / og_tags / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
addedOutput schema / properties / content / properties / metadata / properties / twitter_tags / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
changedOutput schema / properties / content / properties / screenshots / items / additionalProperties
```
Previous value: -trueNew value: +{}
```

addedOutput schema / properties / error

Added value: +{
+  "description": "Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content",
+  "type": "string"
+}

addedOutput schema / properties / escalated

Added value: +{
+  "description": "Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price",
+  "type": "boolean"
+}

addedOutput schema / properties / expires_at

Added value: +{
+  "description": "When the stored result is dropped (ISO 8601)",
+  "type": "string"
+}

addedOutput schema / properties / preview

Added value: +{
+  "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)",
+  "type": "string"
+}

addedOutput schema / properties / redaction

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when redact_pii was set: what was redacted from the text of this result",
+  "properties": {
+    "count": {
+      "description": "Total spans replaced",
+      "type": "number"
+    },
+    "entities": {
+      "additionalProperties": {
+        "type": "number"
+      },
+      "description": "How many spans were replaced, by entity class; a class with no hits is omitted",
+      "propertyNames": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    "mode": {
+      "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION",
+      "enum": [
+        "fast",
+        "model"
+      ],
+      "type": "string"
+    },
+    "model_ran": {
+      "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged",
+      "type": "boolean"
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / result_handle

Added value: +{
+  "description": "Handle for read_result; the full result is kept 1 hour",
+  "type": "string"
+}

addedOutput schema / properties / status

Added value: +{
+  "description": "HTTP status of the fetch; present when success is false",
+  "type": "number"
+}

addedOutput schema / properties / stealth

Added value: +{
+  "additionalProperties": {},
+  "description": "Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none)",
+  "properties": {
+    "engine": {
+      "type": "string"
+    },
+    "vendor_detected": {
+      "type": [
+        "string",
+        "null"
+      ]
+    }
+  },
+  "type": "object"
+}

addedOutput schema / properties / title

Added value: +{
+  "description": "Document title; present when success is false",
+  "type": "string"
+}

addedOutput schema / properties / total_chars

Added value: +{
+  "description": "Length of the full view in characters",
+  "type": "number"
+}

addedOutput schema / properties / truncated

Added value: +{
+  "description": "True when the inline result is a preview",
+  "type": "boolean"
+}

addedOutput schema / properties / view

Added value: +{
+  "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON",
+  "enum": [
+    "text",
+    "json"
+  ],
+  "type": "string"
+}

addedOutput schema / properties / view_path

Added value: +{
+  "description": "Dotted path of the text field the view was cut from; null for the JSON view",
+  "type": [
+    "string",
+    "null"
+  ]
+}

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed2 schema fields changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "additionalProperties": false,
+  "properties": {
+    "_cost": {
+      "additionalProperties": true,
+      "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result",
+      "properties": {
+        "actual": {
+          "description": "Credits actually charged (0 in creator mode, half-rate on error)",
+          "type": "number"
+        },
+        "projected": {
+          "description": "Credits projected for this call before execution",
+          "type": "number"
+        },
+        "projection_note": {
+          "description": "Human-readable note about how the cost was projected",
+          "type": "string"
+        },
+        "remaining_credits": {
+          "description": "Credits remaining on the account after this call, if known",
+          "type": [
+            "number",
+            "null"
+          ]
+        }
+      },
+      "type": "object"
+    },
+    "content": {
+      "additionalProperties": true,
+      "description": "One key per requested format",
+      "properties": {
+        "branding": {
+          "additionalProperties": {},
+          "description": "Static design tokens: colors, fonts, logo",
+          "type": "object"
+        },
+        "html": {
+          "type": "string"
+        },
+        "json": {
+          "description": "Result of the {type:\"json\"} format (LLM-structured extraction)"
+        },
+        "links": {
+          "additionalProperties": true,
+          "properties": {
+            "external_count": {
+              "type": "number"
+            },
+            "internal_count": {
+              "type": "number"
+            },
+            "links": {
+              "items": {
+                "additionalProperties": true,
+                "properties": {
+                  "href": {
+                    "type": "string"
+                  },
+                  "is_external": {
+                    "type": "boolean"
+                  },
+                  "original_href": {
+                    "type": "string"
+                  },
+                  "text": {
+                    "type": "string"
+                  }
+                },
+                "type": "object"
+              },
+              "type": "array"
+            },
+            "total_count": {
+              "type": "number"
+            }
+          },
+          "type": "object"
+        },
+        "markdown": {
+          "type": "string"
+        },
+        "metadata": {
+          "additionalProperties": true,
+          "properties": {
+            "author": {
+              "type": "string"
+            },
+            "canonical_url": {
+              "type": "string"
+            },
+            "description": {
+              "type": "string"
+            },
+            "json_ld": {
+              "items": {},
+              "type": "array"
+            },
+            "keywords": {
+              "items": {
+                "type": "string"
+              },
+              "type": "array"
+            },
+            "microdata": {
+              "items": {},
+              "type": "array"
+            },
+            "og_tags": {
+              "additionalProperties": {},
+              "type": "object"
+            },
+            "robots": {
+              "type": "string"
+            },
+            "title": {
+              "type": "string"
+            },
+            "twitter_tags": {
+              "additionalProperties": {},
+              "type": "object"
+            },
+            "url": {
+              "type": "string"
+            },
+            "viewport": {
+              "type": "string"
+            }
+          },
+          "type": "object"
+        },
+        "rawHtml": {
+          "type": "string"
+        },
+        "screenshots": {
+          "description": "Present for the \"screenshot\" format; each item carries a resourceUri once published",
+          "items": {
+            "additionalProperties": true,
+            "properties": {},
+            "type": "object"
+          },
+          "type": "array"
+        },
+        "text": {
+          "type": "string"
+        }
+      },
+      "type": "object"
+    },
+    "success": {
+      "description": "Whether the scrape completed",
+      "type": "boolean"
+    },
+    "url": {
+      "description": "Final URL after redirects",
+      "type": "string"
+    },
+    "warnings": {
+      "description": "Per-format warnings; partial success never fails the whole call",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    }
+  },
+  "type": "object"
+}

First observedv4.10.0Jul 29, 2026

TDQS

A4.9/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint=true, idempotentHint=true, destructiveHint=false; the description adds rich behavior beyond them: per-format warnings never fail the whole call, onlyMainContent strips via Readability, escalate runs the plain fetch first and only retries in the stealth browser when walled, plus cost and example. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Long but dense: every sentence earns its place. Purpose and core behavior are front-loaded, then formats, exclusions, cost, and an example. The grouped clause style keeps a high-information paragraph readable without waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 12-parameter tool with nested objects and multiple formats, the description covers selection criteria, exclusions, cost model, partial-success behavior, escalation, and an example. An output schema exists, so return values need no explanation; the schema fills any remaining parameter-level details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds real meaning beyond the schema: explains branding as static design tokens, screenshot returns crawlforge:// resources, highlights query costs 1 extra credit with no model, and gives a concrete call example. It does not cover every parameter, but the schema already documents them well.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb and resource: 'Use this to read one page'. Enumerates the exact formats, and explicitly distinguishes from siblings: 'Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape)'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States when to use this tool and names concrete alternatives with the conditions that select them: fetch_url for API/JSON, scrape_with_actions for click/login, batch_scrape for 2+ URLs. Also says 'Preferred over the client's built-in web fetch' and tells when to set escalate:true.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/mysleekdesigns/crawlforge-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server