Skip to main content
Glama
Sign In
CrawlForge MCP Server

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

extract_links

Read-onlyIdempotent

Use this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true drops internal links; mailto:, tel: and javascript: links are type "other".

Instructions

Use this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true drops internal links; mailto:, tel: and javascript: links are type "other". Not for a whole site (map_site), and not alongside a scrape of the same URL: scrape formats:["markdown","links"] returns both in one fetch. A PDF or other binary is refused (process_document reads documents); an empty client-rendered shell succeeds with rendered:false and a warning. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page - projected at 6, charged 1 when the plain fetch worked. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: extract_links({url: "https://example.com", filter_external: true})

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to extract links from
maxAgeNoReuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.
base_urlNoBase URL for resolving relative links
escalateNoWhen the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine "auto" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false
user_agentNoOverride the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
respect_robotsNoRespect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
escalate_engineNoStealth engine for the escalated retry: "auto" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), "playwright" (Chromium) or "camoufox"auto
filter_externalNoDrop internal (same-host) links
max_inline_charsNoLargest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv6.24.0
    • addedInput schema / properties / maxAge
      Added value: +{
+  "description": "Reuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.",
+  "maximum": 9007199254740991,
+  "minimum": 0,
+  "type": "integer"
+}
  2. Changed1 schema field changedv6.19.2
    • addedInput schema / properties / max_inline_chars
      Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}
  3. Changed3 schema fields changedv6.18.1
    • addedInput schema / properties / escalate
      Added value: +{
+  "default": false,
+  "description": "When the plain fetch comes back blocked (403/429/444/challenge page/empty shell), re-read the page once in the stealth browser and extract from the rendered document. Under escalate_engine \"auto\" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 1+5; the actual charge stays at 1 when the plain fetch succeeded. Default: false",
+  "type": "boolean"
+}
    • addedInput schema / properties / escalate_engine
      Added value: +{
+  "default": "auto",
+  "description": "Stealth engine for the escalated retry: \"auto\" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), \"playwright\" (Chromium) or \"camoufox\"",
+  "enum": [
+    "auto",
+    "playwright",
+    "camoufox"
+  ],
+  "type": "string"
+}
    • changedInput schema / properties / filter_external / description
      Previous value: -"Only return external links"New value: +"Drop internal (same-host) links"
  4. Changed1 schema field changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
  5. Changed2 schema fields changedv5.4.0
    • addedInput schema / properties / respect_robots
      Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}
    • addedInput schema / properties / user_agent
      Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}
  6. Changed1 schema field changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  7. First observedv4.10.0

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover readOnly/idempotent/non-destructive, and the description adds genuinely new behavior: PDFs and binaries are refused with a pointer to process_document, empty client-rendered shells succeed with rendered:false plus a warning, and the escalated retry only fires when the plain fetch is walled. It also discloses the economics (1 credit, projected 6 on escalation, charged 1 when the plain path works) and the 10-minute session cache.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded, then exclusions, then edge-case behavior, then cost, then a concrete example — a sensible ordering. It is a single very dense paragraph with stacked clauses (escalation pricing, cache, robots) that takes effort to parse, but almost every sentence carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates by explaining what the caller will see: rendered:false with a warning on empty shells, type "other" for non-http schemes, and a result_handle path for oversized results. For a 9-parameter fetch tool with a non-trivial cost model, nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, and the schema descriptions for escalate, maxAge and max_inline_chars are already very detailed. The description still adds value the schema does not: filter_external's effect in output terms and the fact that mailto:, tel: and javascript: links come back as type "other".

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Starts with a specific verb and resource ('list the hyperlinks on one page') and immediately supplies three concrete use cases (crawl seed list, broken-link audit, related resources). It explicitly distinguishes itself from map_site for whole-site scope and from scrape with links format, so an agent can pick it without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-not guidance: not for a whole site (use map_site) and not alongside a scrape of the same URL (scrape's links format returns both in one fetch). It also states when to set escalate:true and when to pass maxAge:0, which is real routing guidance rather than implied context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools