Skip to main content
Glama
Sign In
CrawlForge MCP Server

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

fetch_url

Read-onlyIdempotent

Use this for a raw HTTP body - JSON, XML, plain text, an API response - or for the status code, headers and response time. Returns the body unprocessed.

Instructions

Use this for a raw HTTP body - JSON, XML, plain text, an API response - or for the status code, headers and response time. Returns the body unprocessed. Not for HTML you intend to read: scrape returns markdown from one fetch, so fetch_url followed by extract_* is a double fetch. Not for JS-rendered or bot-protected pages (scrape, then stealth_mode). Supports custom headers (e.g. auth tokens) and a timeout; the result reports redirected and final_url. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: fetch_url({url: "https://api.example.com/v1/items", timeout: 15000})

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to fetch content from
maxAgeNoReuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.
headersNoCustom HTTP headers to include in the request
timeoutNoRequest timeout in milliseconds (1000-30000)
user_agentNoOverride the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
respect_robotsNoRespect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
max_inline_charsNoLargest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv6.24.0
    • addedInput schema / properties / maxAge
      Added value: +{
+  "description": "Reuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.",
+  "maximum": 9007199254740991,
+  "minimum": 0,
+  "type": "integer"
+}
  2. Changed3 schema fields changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
    • addedInput schema / properties / headers / propertyNames
      Added value: +{
+  "type": "string"
+}
    • addedInput schema / properties / max_inline_chars
      Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}
  3. Changed2 schema fields changedv5.4.0
    • addedInput schema / properties / respect_robots
      Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}
    • addedInput schema / properties / user_agent
      Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}
  4. Changed1 schema field changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  5. First observedv4.10.0

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnly, idempotent, openWorld), the description discloses the 10-minute session cache and 1-credit cost, how maxAge: 0 forces a fresh fetch, that redirects and final_url are reported, and that headers/timeout are supported. These are non-obvious operational traits an agent needs before calling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded and every sentence carries routing, cost, or caching information with no filler. It is dense and slightly long, but the density is justified by the number of distinct behaviors covered.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, yet the description explicitly lists return contents (body, status code, headers, response time, redirected, final_url), plus cost, caching, and the double-fetch pitfall. Nothing an agent needs to invoke this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is already 100%, so the baseline is 3; the description still adds value by explaining the credit/caching semantics of maxAge and giving a concrete usage pattern for headers (auth tokens) plus an example call. It doesn't expand on user_agent or respect_robots, which remain schema-only.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb and resource ('raw HTTP body - JSON, XML, plain text, an API response') and explicitly states what it returns ('Returns the body unprocessed'). It names the exact sibling it is not (scrape, extract_*, stealth_mode), so an agent can route without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use (raw bodies, status codes, headers, timing) and explicit when-not-to-use with the named alternative for each case: HTML to read → scrape; JS-rendered/bot-protected → scrape then stealth_mode. Alternatives and conditions are fully spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools