Skip to main content
Glama
mysleekdesigns

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

extract_links

Read-onlyIdempotent

List the hyperlinks on a single web page for seed lists, broken-link audits, or resource discovery. Filter out internal links and read rendered pages when a site blocks the plain fetch.

Instructions

Use this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true drops internal links; mailto:, tel: and javascript: links are type "other". Not for a whole site (map_site), and not alongside a scrape of the same URL: scrape formats:["markdown","links"] returns both in one fetch. A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Cost: 1 credit. Example: extract_links({url: "https://example.com", filter_external: true})

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to extract links from
base_urlNoBase URL for resolving relative links
escalateNoWhen the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine "auto" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false
user_agentNoOverride the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
respect_robotsNoRespect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
escalate_engineNoStealth engine for the escalated retry: "auto" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), "playwright" (Chromium) or "camoufox"auto
filter_externalNoDrop internal (same-host) links
max_inline_charsNoLargest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv6.19.2
    • addedInput schema / properties / max_inline_chars
      Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}
  2. Changed3 schema fields changedv6.18.1
    • addedInput schema / properties / escalate
      Added value: +{
+  "default": false,
+  "description": "When the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false",
+  "type": "boolean"
+}
    • addedInput schema / properties / escalate_engine
      Added value: +{
+  "default": "auto",
+  "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"",
+  "enum": [
+    "auto",
+    "playwright",
+    "camoufox"
+  ],
+  "type": "string"
+}
    • changedInput schema / properties / filter_external / description
      Previous value: -"Only return external links"New value: +"Drop internal (same-host) links"
  3. Changed1 schema field changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
  4. Changed2 schema fields changedv5.4.0
    • addedInput schema / properties / respect_robots
      Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}
    • addedInput schema / properties / user_agent
      Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}
  5. Changed1 schema field changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  6. First observedv4.10.0

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly/idempotent/non-destructive), yet the description adds substantial operational detail: escalate runs the plain fetch first and only re-reads in the stealth browser if walled, projected 6 vs charged 1, a fixed 1-credit cost, PDF/binary refusal, and an empty client-rendered shell returning rendered:false with a warning. This is well beyond what the annotations provide and consistent with them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with purpose, then alternatives, then behavior, then cost, then an example — a logical progression with no filler sentences. It is dense and somewhat long, but every clause carries operational information an agent would otherwise have to infer.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description still explains return behavior: inline result vs preview plus result_handle, rendered:false with a warning on empty shells, and warning-recording when robots is disabled. For an 8-parameter, multi-mode tool, nothing an agent needs to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the schema already documents url, escalate, filter_external, and the rest, setting the baseline at 3. The description still adds usable context by tying filter_external to its intent (dropping internal links) and naming the escalate trigger in plain language, though it largely restates rather than extends the schema's own parameter prose.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource ('list the hyperlinks on one page') and immediately frames the concrete use cases (crawl seed list, broken-link audit, related resources). It explicitly distinguishes itself from map_site, scrape, and process_document, so an agent can route without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States both when-not conditions ('Not for a whole site (map_site)', 'not alongside a scrape of the same URL') and names the alternative that handles each case. It also gives the escalation trigger ('when the site is known to block') and the no-escalation cases (404/5xx), which is exactly the when-to-use guidance an agent needs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools