Skip to main content
Glama
Sign In
CrawlForge MCP Server

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

extract_metadata

Read-onlyIdempotent

Use this for a page's SEO metadata only: title, meta description, Open Graph tags, canonical URL, schema.org data. url is the final URL; redirected:true (with requested_url) says a redirect moved the request to another page.

Instructions

Use this for a page's SEO metadata only: title, meta description, Open Graph tags, canonical URL, schema.org data. url is the final URL; redirected:true (with requested_url) says a redirect moved the request to another page. Not alongside a scrape of the same URL: scrape formats:["markdown","metadata"] returns both in one fetch. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 1 credit. Example: extract_metadata({url: "https://example.com"})

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to extract metadata from
maxAgeNoReuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.
user_agentNoOverride the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
json_ld_typesNoFilter the returned JSON-LD to nodes of these schema.org types, e.g. ["Product","Offer"]. Subtypes match their parent: "Event" returns MusicEvent, "Offer" returns AggregateOffer, "ItemList" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node; a match nested inside another returned node comes back inside it, not again on its own. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type, nested ones included. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly.
respect_robotsNoRespect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.
max_inline_charsNoLargest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv6.24.0
    • addedInput schema / properties / maxAge
      Added value: +{
+  "description": "Reuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.",
+  "maximum": 9007199254740991,
+  "minimum": 0,
+  "type": "integer"
+}
  2. Changed2 schema fields changedv6.19.2
    • changedInput schema / properties / json_ld_types / description
      Previous value: -"Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly."New value: +"Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node; a match nested inside another returned node comes back inside it, not again on its own. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type, nested ones included. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly."
    • addedInput schema / properties / max_inline_chars
      Added value: +{
+  "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)",
+  "maximum": 10000000,
+  "minimum": 1000,
+  "type": "integer"
+}
  3. Changed1 schema field changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
  4. Changed3 schema fields changedv5.4.0
    • addedInput schema / properties / json_ld_types
      Added value: +{
+  "description": "Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly.",
+  "items": {
+    "type": "string"
+  },
+  "type": "array"
+}
    • addedInput schema / properties / respect_robots
      Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}
    • addedInput schema / properties / user_agent
      Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}
  5. Changed1 schema field changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  6. First observedv4.10.0

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive/openWorld, so the safety profile is covered; the description layers on non-obvious behavior the annotations cannot express: redirect provenance (redirected:true plus requested_url), a 10-minute session cache, and a stated cost of 1 credit per call. That is exactly the additional context this dimension rewards.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads the purpose and the exclusion before caching and cost details, and ends with a runnable example. It is dense — the middle sentence packs redirect semantics, the scrape alternative, caching, credit cost and maxAge into one long clause — but nearly every clause carries information an agent needs.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 6-parameter read tool with no output schema, the description names the returned fields, explains redirect output, discloses caching and cost, and names the cheaper alternative. The remaining parameters are exhaustively documented in the schema, so an agent has everything needed to call this correctly and to decide against it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, and the description earns an increment by clarifying that the returned `url` is the final post-redirect URL and by surfacing the credit cost that no schema field states. It does not touch user_agent, respect_robots, json_ld_types or max_inline_chars, but those are fully documented in the schema, so the gap is small.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific scope statement — 'a page's SEO metadata only' — followed by an explicit enumeration of what comes back (title, meta description, OG tags, canonical URL, schema.org data). That verb+resource+scope combination makes it immediately distinguishable from generic scrape and from extract_structured/extract_links siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent elsewhere — 'Not alongside a scrape of the same URL: scrape formats:["markdown","metadata"] returns both in one fetch' — naming the sibling and the exact parameter combination. It also states the repeat-request rule (same page with different formats/paths/selectors reuses the session copy for 10 minutes at 1 credit) and the escape hatch maxAge: 0.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools