scrape_url
Scrape a web page, divide its content into structured text chunks optimized for RAG retrieval, and return them ready for downstream processing. No embedding or vector database required.
Instructions
Scrape a single web URL, chunk its content into RAG-ready text segments, and return the structured chunks. No embedding or vector DB required — this is the fastest and cheapest operation.
Use this when the user wants to read, summarize, or process the content of a specific web page WITHOUT extracting structured fields.
If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:
Call verify_provider_key(provider, 'llm') → get live model list
Ask user to choose a model from the list
Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'
LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The web URL to scrape and chunk. | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector to target a specific element (e.g. 'article', '.content', 'main'). | |
| js_render | No | Use headless Chromium to render JavaScript before scraping. Required for SPAs and JS-heavy pages. Ask the user before enabling. Adds a small surcharge. | |
| llm_model | No | LLM model name. MUST be chosen from the list returned by verify_provider_key — do not guess or hardcode. | |
| chunk_size | No | Target token count per chunk. Default: 512. Range: 64–4096. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. One of: 'openai', 'anthropic', 'gemini'. Verify with verify_provider_key first. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment. An LLM generates a unique context string for each chunk, boosting retrieval accuracy by 35–50%. Present this as a recommended upgrade. Requires llm_provider and llm_model (from verify_provider_key). |