Skip to main content
Glama
Sign In
CrawlForge MCP Server

CrawlForge MCP Server

by mysleekdesigns
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

scrape_structured

Read-onlyIdempotent

Use this when you know the exact CSS selectors for the data you want - e.g. a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages.

Instructions

Use this when you know the exact CSS selectors for the data you want - e.g. a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead - one object per row, null for a field the row lacks. Not for pages whose markup varies or where you cannot name the selectors (extract_structured, LLM-driven). Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 2 credits. Example: scrape_structured({url: "https://shop.com/products", row_selector: ".product-card", selectors: {price: ".price", name: ".product-title"}})

Input Schema

NameRequiredDescriptionDefault
urlYesThe URL to scrape
maxAgeNoReuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.
selectorsYesCSS selectors mapping field names to selectors. Append @attr to extract an attribute instead of text (e.g. "a.link@href", "img@src"); @href and @src values are resolved to absolute URLs. A field may also be {selector, multiple: true} to return every match as an array - inside a row_selector row a field otherwise keeps only the first match
user_agentNoOverride the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.
max_resultsNoMaximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set
row_selectorNoCSS selector for the repeating row/container element. When set, each field in selectors is matched inside each row and data is an array of row-aligned records ({field: value|null}) instead of parallel arrays
respect_robotsNoRespect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed4 schema fields changedv6.24.0
    • addedInput schema / properties / maxAge
      Added value: +{
+  "description": "Reuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh.",
+  "maximum": 9007199254740991,
+  "minimum": 0,
+  "type": "integer"
+}
    • addedInput schema / properties / selectors / additionalProperties / anyOf
      Added value: +[
+  {
+    "type": "string"
+  },
+  {
+    "properties": {
+      "multiple": {
+        "type": "boolean"
+      },
+      "selector": {
+        "type": "string"
+      }
+    },
+    "required": [
+      "selector"
+    ],
+    "type": "object"
+  }
+]
    • removedInput schema / properties / selectors / additionalProperties / type
      Removed value: -"string"
    • changedInput schema / properties / selectors / description
      Previous value: -"CSS selectors mapping field names to selectors. Append @attr to extract an attribute instead of text (e.g. \"a.link@href\", \"img@src\")"New value: +"CSS selectors mapping field names to selectors. Append @attr to extract an attribute instead of text (e.g. \"a.link@href\", \"img@src\"); @href and @src values are resolved to absolute URLs. A field may also be {selector, multiple: true} to return every match as an array - inside a row_selector row a field otherwise keeps only the first match"
  2. Changed3 schema fields changedv6.0.0
    • removedInput schema / additionalProperties
      Removed value: -false
    • addedInput schema / properties / max_results / maximum
      Added value: +9007199254740991
    • addedInput schema / properties / selectors / propertyNames
      Added value: +{
+  "type": "string"
+}
  3. Changed4 schema fields changedv5.4.0
    • changedInput schema / properties / max_results / description
      Previous value: -"Maximum number of matches to return per field when a selector matches multiple elements"New value: +"Maximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set"
    • addedInput schema / properties / respect_robots
      Added value: +{
+  "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.",
+  "type": "boolean"
+}
    • addedInput schema / properties / row_selector
      Added value: +{
+  "description": "CSS selector for the repeating row/container element. When set, each field in selectors is matched inside each row and data is an array of row-aligned records ({field: value|null}) instead of parallel arrays",
+  "type": "string"
+}
    • addedInput schema / properties / user_agent
      Added value: +{
+  "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.",
+  "type": "string"
+}
  4. Changed1 schema field changedv5.0.4
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  5. First observedv4.10.0

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, open-world, and non-destructive behavior. The description adds important context beyond annotations: default selector results are independent arrays rather than row-aligned, row_selector changes output to aligned records with nulls for missing fields, repeated requests reuse the session copy for 10 minutes at 1 credit, maxAge: 0 forces a fresh fetch, and the cost is 2 credits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but front-loaded and well organized: use case first, then default behavior, row_selector correction, exclusions, caching and cost, and finally a concrete example. Every sentence carries actionable information, and there is no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter nested-object tool with no output schema, the description covers the critical gaps: selector alignment behavior, row_selector semantics, caching and cost, and the alternative LLM-based tool. The remaining parameters such as user_agent and respect_robots are fully documented in the schema, so the definition is complete enough for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; however, the description adds meaningful interaction semantics beyond the schema. It explains that selectors are matched independently by default, that the resulting arrays are not row-aligned, and that row_selector produces aligned records with nulls for missing fields. The worked example clarifies how url, row_selector, and selectors combine in practice.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: scrape structured data using exact CSS selectors. It distinguishes itself from LLM-driven extraction by naming extract_structured as the alternative for variable markup, so an agent can route correctly without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use it (known exact selectors, consistent markup such as pricing tables or product lists) and when not to use it (markup varies, selectors cannot be named, use extract_structured instead). It also explains the caching and maxAge behavior for repeat requests, leaving little ambiguity.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools