Skip to main content
Glama
Sign In
Crawlora MCP

Crawlora MCP

Official
by Crawlora-org
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript
Remote

web_scrape

Fetch and convert a public URL into clean markdown, HTML, links, or metadata. Strips navigation and boilerplate so you can analyze page content without clutter.

Instructions

Scrape a URL into markdown, HTML, links or metadata. Fetches a single public URL and returns clean content in the requested formats (markdown, html, raw_html, links, metadata). The request body IS the ScrapeOption object itself — e.g. {"url": "https://example.com"} — do not wrap it in an extra key. With render=auto the request starts as a fast HTTP fetch and escalates to a real browser when the page is blocked or rendered with JavaScript. only_main_content (default true) strips navigation, headers, footers and other boilerplate before conversion. Only public pages are supported; respect each site's terms of use and robots directives. A handful of popular sites (Amazon, Reddit, Yelp, LinkedIn, and others) already have a dedicated, more reliable endpoint elsewhere in this API — a failed scrape against one of them names it. A Reddit URL that maps onto a dedicated Reddit endpoint (listings, search, comment threads, user and domain pages) is answered by that endpoint instead, and its response is returned as-is in data with an X-Routed-To header; send dedicated=false to always get the scraped page.

Input Schema

NameRequiredDescriptionDefault
scrapeOptionYesScrape options

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv1.17.9
    • addedInput schema / properties / scrapeOption / properties / dedicated
      Added value: +{
+  "type": "boolean"
+}
  2. Changed2 schema fields changedv1.17.5
    • addedInput schema / properties / scrapeOption / properties
      Added value: +{
+  "backend": {
+    "type": "string"
+  },
+  "formats": {
+    "items": {
+      "type": "string"
+    },
+    "type": "array"
+  },
+  "max_age": {
+    "type": "integer"
+  },
+  "only_main_content": {
+    "type": "boolean"
+  },
+  "proxy": {
+    "type": "string"
+  },
+  "render": {
+    "enum": [
+      "auto",
+      "http",
+      "browser",
+      "unblocker"
+    ],
+    "type": "string"
+  },
+  "store_in_cache": {
+    "type": "boolean"
+  },
+  "url": {
+    "type": "string"
+  },
+  "wait_for": {
+    "type": "integer"
+  }
+}
    • addedInput schema / properties / scrapeOption / required
      Added value: +[
+  "url"
+]
  3. Addedv1.1.0

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses the render=auto HTTP-to-browser escalation, the only_main_content default of true and what it strips, the public-pages-only restriction, robots/terms expectations, and the dedicated-endpoint bypass with an X-Routed-To response header. It omits rate limits, caching (store_in_cache) and auth/cost implications, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose and the critical request-body warning ("do not wrap it in an extra key") before behavioral detail. Dense and mostly waste-free, though the trailing Reddit-specific routing paragraph is more verbose than the general case warrants.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-param tool with a nested option object, no annotations and no output schema, the description supplies the enum behaviors, defaults, routing rules and the important body-shape warning. An agent has what it needs to invoke it correctly; only cost/rate-limit context is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Although schema coverage is reported at 100%, the nested properties have no per-field descriptions, and the description compensates: it explains render=auto behavior, the only_main_content default and effect, dedicated=false, and the format values. It adds real meaning beyond the bare enum/type definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource ("Scrape a URL") with the target output formats enumerated (markdown, html, raw_html, links, metadata). "Fetches a single public URL" clearly distinguishes it from multi-page siblings like web_crawl and from search tools. An agent can identify what this does without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explains the routing condition against dedicated endpoints (Amazon, Reddit, etc.) and how to override it with dedicated=false, which is genuine when-to-use guidance. It doesn't, however, compare itself to generic alternatives such as extract or web_search, so the agent is left to infer those boundaries.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools