Skip to main content
Glama
mxchris18
by mxchris18

scrape_url

Scrape a web URL and split its content into structured chunks for RAG pipelines. Ideal for summarizing or processing page content without field extraction.

Instructions

Scrape a single web URL, chunk its content into RAG-ready text segments, and return the structured chunks. No embedding or vector DB required — this is the fastest and cheapest operation.

Use this when the user wants to read, summarize, or process the content of a specific web page WITHOUT extracting structured fields.

If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Ask user to choose a model from the list

  3. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

  4. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesThe web URL to scrape and chunk.
overlapNoToken overlap between consecutive chunks. Default: 50.
selectorNoOptional CSS selector to target a specific element (e.g. 'article', '.content', 'main').
js_renderNoUse headless Chromium to render JavaScript before scraping. Required for SPAs and JS-heavy pages. Ask the user before enabling. Adds a small surcharge.
llm_modelNoLLM model name. MUST be chosen from the list returned by verify_provider_key — do not guess or hardcode.
chunk_sizeNoTarget token count per chunk. Default: 512. Range: 64–4096.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. One of: 'openai', 'anthropic', 'gemini'. Verify with verify_provider_key first.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment. An LLM generates a unique context string for each chunk, boosting retrieval accuracy by 35–50%. Present this as a recommended upgrade. Requires llm_provider and llm_model (from verify_provider_key).
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses chunking behavior, cost (surcharge for js_render, ~$0.001/chunk for contextual retrieval), prerequisite steps (verify_provider_key), and important flags (js_render for SPAs). However, it omits output format details, error handling, rate limits, or caching behavior, which would increase transparency for a tool with 9 parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise in the first paragraph but includes a detailed multi-step pre-flight sequence that, while informative, makes it longer than necessary. Every sentence serves a purpose, but the pre-flight instructions could be more succinct. Overall structured logically: purpose, usage, special case.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 9 parameters, no output schema, and moderate complexity. The description covers the main use case and the contextual retrieval flow. However, it does not describe the return format (e.g., array of chunks with text, metadata) or error handling. Without output schema, the agent needs more detail on what to expect. Given the richness of the description for the main flow, it's adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds value by explaining when to use js_render (JavaScript-heavy pages, SPAs), that llm_model must come from verify_provider_key, and that llm_api_key can be an env var. The pre-flight sequence for contextual_retrieval adds operational context beyond schema. However, some schema descriptions already suffice, limiting extra contribution.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool scrapes a single URL, chunks it into RAG-ready segments, and returns structured chunks. It distinguishes from sibling tools like extract_data (which extracts structured fields) and crawl_site (which does multiple pages). The verb 'scrape' and resource 'single web URL' are specific and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use: 'when the user wants to read, summarize, or process content of a specific web page WITHOUT extracting structured fields.' Provides a detailed pre-flight sequence for contextual_retrieval. Does not explicitly list when not to use, but the positive criteria imply exclusion of structured extraction. Sibling names provide context, but description could be more explicit about alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/mxchris18/scrapedatshi-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server