CrawlForge MCP Server

batch_scrape

Read-only

Scrape 2-50 URLs in one call instead of looping. Sync mode returns results directly; async mode with a webhook handles larger batches.

Instructions

Use this to scrape 2-50 URLs in one call - product pages, news articles, competitor pages. Never loop scrape over a URL list. mode:"sync" returns results directly for up to ~25 URLs; mode:"async" with a webhook for larger batches, then get_batch_results. Not for one URL (scrape) or for discovering URLs (map_site). Cost: 5 credits. Example: batch_scrape({urls: ["https://a.com","https://b.com"], formats: ["json"], maxConcurrency: 5})

Input Schema

TableJSON Schema

Name	Required	Description	Default
`mode`	No	Processing mode: sync (wait) or async (background)	sync
`urls`	Yes	Array of URLs or URL objects to scrape
`formats`	No	Output formats for scraped content
`webhook`	No	Webhook configuration for async job notifications
`pageSize`	No	Number of results per page
`jobOptions`	No	Job management options for async processing
`redact_pii`	No	Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off
`user_agent`	No	Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
`includeFailed`	No	List failed URLs in results. false hides the entries; failedUrls still counts them
`maxConcurrency`	No	Maximum concurrent scraping requests
`respect_robots`	No	Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
`includeMetadata`	No	Include page metadata in results
`extractionSchema`	No	Schema for structured data extraction from each URL
`max_inline_chars`	No	Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)
`delayBetweenRequests`	No	Delay in milliseconds between requests

Schema Changelog

Changes observed during successful MCP inspections.

Changed1 schema field changedv6.18.1Oct 3, 2026

changedInput schema / properties / includeFailed / description

Previous value: -"Include failed URLs in results"New value: +"List failed URLs in results. false hides the entries; failedUrls still counts them"

Changed8 schema fields changedv6.0.0Sep 6, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```
addedInput schema / properties / extractionSchema / propertyNames
```
Added value: +{
+  "type": "string"
+}
```
removedInput schema / properties / jobOptions / additionalProperties
```
Removed value: -false
```

addedInput schema / properties / max_inline_chars

Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}

addedInput schema / properties / redact_pii

Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "properties": {
+        "entities": {
+          "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected",
+          "items": {
+            "type": "string"
+          },
+          "type": "array"
+        },
+        "mode": {
+          "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)",
+          "enum": [
+            "fast",
+            "model"
+          ],
+          "type": "string"
+        },
+        "replace_style": {
+          "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value",
+          "enum": [
+            "tag",
+            "mask",
+            "remove"
+          ],
+          "type": "string"
+        }
+      },
+      "type": "object"
+    }
+  ],
+  "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off"
+}

changedInput schema / properties / urls / items / anyOf

Previous value: -[
-  {
-    "format": "uri",
-    "type": "string"
-  },
-  {
-    "additionalProperties": false,
-    "properties": {
-      "headers": {
-        "additionalProperties": {
-          "type": "string"
-        },
-        "type": "object"
-      },
-      "metadata": {
-        "additionalProperties": {},
-        "type": "object"
-      },
-      "selectors": {
-        "additionalProperties": {
-          "type": "string"
-        },
-        "type": "object"
-      },
-      "timeout": {
-        "maximum": 30000,
-        "minimum": 1000,
-        "type": "number"
-      },
-      "url": {
-        "format": "uri",
-        "type": "string"
-      }
-    },
-    "required": [
-      "url"
-    ],
-    "type": "object"
-  }
-]New value: +[
+  {
+    "format": "uri",
+    "type": "string"
+  },
+  {
+    "properties": {
+      "headers": {
+        "additionalProperties": {
+          "type": "string"
+        },
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "metadata": {
+        "additionalProperties": {},
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "selectors": {
+        "additionalProperties": {
+          "type": "string"
+        },
+        "propertyNames": {
+          "type": "string"
+        },
+        "type": "object"
+      },
+      "timeout": {
+        "maximum": 30000,
+        "minimum": 1000,
+        "type": "number"
+      },
+      "url": {
+        "format": "uri",
+        "type": "string"
+      }
+    },
+    "required": [
+      "url"
+    ],
+    "type": "object"
+  }
+]

removedInput schema / properties / webhook / additionalProperties
```
Removed value: -false
```
addedInput schema / properties / webhook / properties / headers / propertyNames
```
Added value: +{
+  "type": "string"
+}
```

Changed2 schema fields changedv5.4.0Aug 29, 2026

addedInput schema / properties / respect_robots

Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}

addedInput schema / properties / user_agent

Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}

Changed1 schema field changedv5.0.4Aug 21, 2026

changedInput schema / $schema

Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"

First observedv4.10.0Jul 29, 2026

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, openWorld, and non-destructive. The description adds real context beyond that: cost (5 credits), the sync-vs-async behavioral split with a URL-count threshold, and the async follow-up step via get_batch_results. It omits behavioral notes on pagination, redaction, and the robots.txt warning, which the schema carries instead.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the primary action and scope, then routing, then mode mechanics, then cost and example. Every sentence carries distinct information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter tool with no output schema, it covers the critical decision path: batch scope, mode choice, async retrieval, cost, and alternatives. It leaves advanced parameters (redaction, robots, concurrency, pagination) to the schema, which is reasonable given 100% schema coverage, but a brief note on the async result-retrieval handle would round it out.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3. The description still adds value the schema doesn't: the ~25-URL practical threshold for sync mode and the cost figure. The worked example (urls, formats, maxConcurrency) illustrates expected shape. Marginal lift above baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (scrape) and resource (2-50 URLs in one call), scoping the batch use case precisely. It explicitly distinguishes itself from siblings: 'Not for one URL (scrape) or for discovering URLs (map_site).' An agent can select it without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use (2-50 URLs, 'Never loop scrape over a URL list'), when-not (single URL, discovery), and mode-selection criteria ('sync returns results directly for up to ~25 URLs; async with a webhook for larger batches, then get_batch_results'). Alternatives are named and conditions that select them are clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

batch_scrape

Instructions

Input Schema

Schema Changelog

TDQS

Other Tools