scrapedatshi-mcp
The scrapedatshi-mcp server enables AI-powered web scraping, crawling, data extraction, and RAG pipeline orchestration directly from Claude Desktop — no code required.
Scraping & Chunking
scrape_url— Scrape a single URL and return RAG-ready text chunks (with optional JS rendering and RAG 2.0 contextual enrichment).chunk_file— Upload and chunk local files (PDF, MD, TXT, YAML, JSON) into structured text segments.
Crawling
crawl_site— Crawl an entire website via sitemap or spider mode, returning chunks from all pages with auto-batching for large sites (200+ pages).
Structured Data Extraction
extract_data— Use an LLM to extract specific schema fields (e.g. product name, price, stock) from a single URL.extract_crawl— Crawl a site and extract structured fields from every page.
Vector DB / RAG Pipelines
sync_to_vectordb— Scrape a URL, embed chunks, and inject into a vector DB in one call.ingest_file— Upload a local file, embed its chunks, and inject into a vector DB.autorag— Full AutoRAG pipeline: crawl an entire domain, chunk, embed, and inject all content into your vector DB with automatic batching.inspect_vectordb— Read vector DB metadata (dimensions, vector count, suggested models).query_vectordb— Semantically query your vector DB for the most relevant chunks.
Supported embedding providers: OpenAI, Cohere, Gemini, Mistral, Voyage AI, Ollama. Supported vector databases: Pinecone, Qdrant, ChromaDB, Supabase, Weaviate, MongoDB, Azure Cosmos DB, LanceDB.
Provider & Key Management
verify_provider_key— Validate LLM/embedding API keys and retrieve live model lists.list_embedding_providers— List all supported embedding providers with notes.list_vector_db_providers— List all supported vector DBs with required config fields.
Guidance
get_usage_guide— Access a guided workflow wizard to select the right tool and follow the correct pre-flight sequence.
API keys are configured securely via environment variables in the Claude Desktop config, not entered in chat.
Provides both LLM and embedding capabilities, supporting models like gemini-1.5-flash and text-embedding-004.
Allows syncing scraped data to MongoDB Atlas as a vector database for RAG pipelines.
Provides local embedding models for vector generation, such as nomic-embed-text and mxbai-embed-large.
Provides LLM and embedding capabilities, supporting models like gpt-4o-mini and text-embedding-3-small.
Allows syncing scraped data to Supabase (pgvector) as a vector database.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@scrapedatshi-mcpScrape https://docs.example.com/getting-started and show me the chunks"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
scrapedatshi-mcp
MCP (Model Context Protocol) server for the scrapedatshi RAG pipeline API.
Use scrapedatshi's scraping, crawling, extraction, and vector DB sync tools directly from Claude Desktop — no code required.
What you can do
Just talk to Claude naturally:
"Scrape https://docs.example.com and give me the chunks"
"Extract the text from this PDF: https://example.com/annual-report.pdf"
"Extract all tables from this local PDF: C:/Users/me/Documents/financials.pdf"
"Chunk this PDF URL: https://my-bucket.s3.amazonaws.com/report.pdf" — PDF URLs are automatically detected and extracted
"Crawl https://example.com/products and extract the title and price from every page"
"Sync https://docs.example.com to my Pinecone index using OpenAI embeddings"
"Crawl the entire docs.stripe.com site (all 800 pages) and inject it into my Pinecone index" — large sites are auto-batched server-side, no manual pagination needed
"What embedding providers does scrapedatshi support?"
"Inspect my Pinecone index and tell me what embedding model was used"
"Query my Pinecone index for information about API authentication"
"Query my LanceDB with hybrid search — I need to find exact IDs and names, not just semantic matches"
"Chunk https://docs.example.com using hierarchical chunking so the LLM gets full context on retrieval"
"Ingest all the JSON files in my ./scrapy_output/ folder into my Pinecone index"
Related MCP server: Firecrawl MCP Server
Tools exposed
Tool | What it does |
| Verify an LLM or embedding API key + get live model list |
| Returns the guided wizard flow and tool selection reference |
| Scrape a URL and return clean Markdown — no chunking, just the raw text |
| Extract text or tables from a PDF (URL or local file) — no chunking, no embedding needed |
| Scrape & chunk a single URL into RAG-ready text segments |
| Upload a local file (PDF, MD, TXT, CSV, XLSX, DOCX, IPYNB, HTML, XML, code files, etc.) and chunk it into RAG-ready segments |
| Crawl an entire site (sitemap or spider mode) and return all chunks |
| Extract structured schema fields from a URL using your LLM |
| Multi-page schema extraction via site crawl |
| Full pipeline: scrape URL → embed → inject into your vector DB |
| Full pipeline: upload local file → embed → inject into your vector DB |
| Full pipeline: bulk-ingest a folder of pre-scraped files → embed → inject into your vector DB |
| Full pipeline: crawl entire site → chunk → embed → inject into your vector DB (large sites auto-batched) |
| Read vector DB metadata: dimension, vector count, suggested embedding models (free) |
| Semantic search: embed a query and retrieve the most relevant chunks from your vector DB. Supports |
| RAG Chat: retrieve top-N chunks from your vector DB and generate a grounded LLM answer. Supports |
| Discover supported embedding providers + model notes |
| Discover supported vector DBs + required config fields |
Prerequisites
scrapedatshi account — Sign up at scrapedatshi.com
Add credits — Billing portal
Get your API key — starts with
sds_...Claude Desktop — Download here
Python 3.10+ — python.org
Installation
Option A — Install from PyPI (recommended, works with uvx)
pip install scrapedatshi-mcpOr use uv for isolated installs:
uv tool install scrapedatshi-mcpOption B — Install from source (local development)
git clone https://github.com/scrapedatshi/scrapedatshi-mcp.git
cd scrapedatshi-mcp
pip install -e .Claude Desktop configuration
Easiest way to find your config file: Open Claude Desktop → Settings → Developer → Edit Config
Alternatively, the file is located at:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%\Claude\claude_desktop_config.json
Recommended — uvx with all provider SDKs (auto-updates on restart)
{
"mcpServers": {
"scrapedatshi": {
"command": "uvx",
"args": [
"--from", "scrapedatshi-mcp[all]",
"--refresh",
"scrapedatshi-mcp"
],
"env": {
"SCRAPEDATSHI_API_KEY": "sds_your_key_here"
}
}
}
}[all]installs all provider SDKs (OpenAI, Anthropic, Gemini, Voyage AI) soverify_provider_keyworks for any provider--refreshchecks PyPI for updates every time Claude Desktop starts — no manual reinstalls needed
If installed via pip (using python)
{
"mcpServers": {
"scrapedatshi": {
"command": "python",
"args": ["-m", "scrapedatshi_mcp.server"],
"env": {
"SCRAPEDATSHI_API_KEY": "sds_your_key_here"
}
}
}
}If cloned from source (absolute path)
{
"mcpServers": {
"scrapedatshi": {
"command": "python",
"args": ["/absolute/path/to/scrapedatshi-mcp/scrapedatshi_mcp/server.py"],
"env": {
"SCRAPEDATSHI_API_KEY": "sds_your_key_here"
}
}
}
}Restart Claude Desktop after saving the config.
Secure key configuration (BYOK)
You bring your own LLM, embedding, and vector DB keys. The server resolves keys in this priority order:
Argument passed in the tool call — explicit override
Environment variable in the MCP config — preferred secure path (keys never appear in chat)
Clear error message if neither is found
Add your provider keys to the env block in claude_desktop_config.json:
{
"mcpServers": {
"scrapedatshi": {
"command": "uvx",
"args": [
"--from", "scrapedatshi-mcp[all]",
"--refresh",
"scrapedatshi-mcp"
],
"env": {
"SCRAPEDATSHI_API_KEY": "sds_your_key_here",
"OPENAI_API_KEY": "sk-...",
"ANTHROPIC_API_KEY": "sk-ant-...",
"GEMINI_API_KEY": "AIza...",
"COHERE_API_KEY": "...",
"MISTRAL_API_KEY": "...",
"VOYAGE_API_KEY": "...",
"PINECONE_API_KEY": "pc-...",
"QDRANT_API_KEY": "...",
"WEAVIATE_API_KEY": "..."
}
}
}
}Once set, Claude will automatically use these keys without asking you to type them in chat.
Fetch Mode
Starting in v0.5.0, the MCP server uses local-fetch mode by default — URLs are fetched on the machine running Claude Desktop (your IP), and only the HTML processing runs on our server. This is cheaper and keeps your IP off our server.
SCRAPEDATSHI_FETCH_MODE=local (default)
The MCP server fetches URLs using the machine's own IP address, then submits the raw HTML to our server for processing.
✅ Your IP is used — not our server's
✅ Billed at the standard per-URL rate ($0.0020)
✅ Faster — no double-hop latency
SCRAPEDATSHI_FETCH_MODE=server
Our server fetches the URL. Use this if Claude Desktop is running in a restricted environment without outbound HTTP access, or if you need server-managed IP rotation.
⚠️ Our server's IP is used
⚠️ Billed at 2× the standard rate ($0.0040 / URL)
✅ Works from restricted environments
To enable server fetch, add SCRAPEDATSHI_FETCH_MODE to your MCP config:
{
"mcpServers": {
"scrapedatshi": {
"command": "uvx",
"args": ["--from", "scrapedatshi-mcp[all]", "--refresh", "scrapedatshi-mcp"],
"env": {
"SCRAPEDATSHI_API_KEY": "sds_your_key_here",
"SCRAPEDATSHI_FETCH_MODE": "server"
}
}
}
}Supported environment variables
Variable | Used for |
| scrapedatshi API key (required) |
|
|
| OpenAI LLM + embedding |
| Anthropic LLM (Claude) |
| Google Gemini LLM + embedding |
| Cohere embedding |
| Mistral embedding |
| Voyage AI embedding |
| Pinecone vector DB |
| Qdrant vector DB (optional for local) |
| Weaviate vector DB (optional for local) |
Authenticated Scraping (v0.5.1+)
For pages behind a login wall, you can pass your session cookies and/or custom headers to scrape_url and crawl_site. Credentials are only sent to URLs within the permitted domain scope — they are never leaked to external domains.
Scrape a login-walled page
Just tell Claude:
"Scrape https://internal.company.com/wiki/api-docs — use my session cookie: abc123"
Claude will call scrape_url with:
{
"url": "https://internal.company.com/wiki/api-docs",
"cookies": {"session": "abc123"},
"headers": {"Authorization": "Bearer eyJ..."}
}Authenticated crawl with subdomain scope
"Crawl https://company.com including wiki.company.com and docs.company.com — use session cookie abc123"
Claude will call crawl_site with:
{
"url": "https://company.com",
"cookies": {"session": "abc123"},
"allow_subdomains": true,
"max_pages": 20
}Security model:
Cookies and headers are only sent to URLs within the permitted domain scope — never to external domains discovered during crawling
allow_subdomains: false(default): only the exact hostname receives credentialsallow_subdomains: true: credentials are shared with subdomains of the root domain (e.g.wiki.company.comwhen root iscompany.com). Multi-part TLDs (.co.uk,.com.br) are handled safely.Credentials are never forwarded to the scrapedatshi server — they stay on the machine running Claude Desktop
Enterprise SSO / MFA — Session Capture (v0.6.4+)
For enterprise portals protected by Okta, Duo, or any SSO/MFA flow that blocks automated login, use the SDK's capture_session() utility to authenticate manually in a real browser, then pass the captured session state to Claude via the storage_state parameter.
Step 1 — Capture the session locally (run once):
pip install scrapedatshi[auth]
playwright install chromiumfrom scrapedatshi.auth import capture_session
import json
state = capture_session(
"https://internal.company.com/login",
save_to="session.auth.json", # gitignored automatically
)This opens a real browser window. Log in manually (including any MFA prompts), then press Enter. The session state is saved to session.auth.json.
Step 2 — Tell Claude to use the saved session:
"Crawl https://internal.company.com using the session state in session.auth.json"
Claude will call crawl_site with the storage_state parameter containing the captured session.
⚠ Security:
session.auth.jsoncontains live authentication tokens. Never commit it to version control. The SDK's.gitignoretemplate automatically filters*.auth.jsonfiles.
Example conversations
Get clean Markdown from a page
You: Scrape https://docs.example.com/getting-started and show me the content.
Claude calls scrape_url and returns the full page as clean Markdown — title, credits used, and the raw text in one piece.
The response also includes selectors_found — a list of CSS selectors for the main content sections detected on the page. Claude can use these to re-scrape just a specific section:
You: Now scrape just the pricing section.
Claude calls scrape_url again with selector="section#pricing" (from selectors_found).
Chunk a page for RAG
You: Chunk https://docs.example.com/getting-started into RAG-ready segments.
Claude calls chunk_url and returns the structured chunks with token counts and credit usage.
Crawl a documentation site
You: Crawl https://docs.example.com — just the first 5 pages.
Claude calls crawl_site with max_pages=5 and returns all chunks from all pages.
Extract structured data from a product page
You: Extract the product name, price, and whether it's in stock from https://example.com/products/widget-pro
Claude calls extract_data with a schema it constructs from your request, using your OpenAI key from the env config.
Extract data from an entire product catalogue
You: Crawl https://example.com/products and extract the title and price from every product page. Limit to 10 pages.
Claude calls extract_crawl with max_pages=10 and returns per-page extraction results.
Sync a page to your vector DB
You: Sync https://docs.example.com to my Pinecone index. The index host is https://my-index-abc123.svc.pinecone.io. Use OpenAI text-embedding-3-small.
Claude calls sync_to_vectordb. If OPENAI_API_KEY and PINECONE_API_KEY are set in your env config, no keys need to be typed in chat.
Query your vector database
You: Query my Pinecone index for "how do I authenticate with the API?"
Claude calls inspect_vectordb first to confirm the embedding model, then calls query_vectordb and returns the top matching chunks.
Hybrid search — exact terms + semantic
You: Query my LanceDB for "Project Alpha manager" using hybrid search — I need exact name matching.
Claude calls query_vectordb with hybrid_search=true, combining BM25 keyword search with vector similarity using Reciprocal Rank Fusion (RRF). Results include rrf_score and hybrid_sources showing which search method found each chunk.
RAG chat with query rewriting
You: Ask my knowledge base: "what about the second pricing tier?" — use hybrid search and rewrite the query first.
Claude calls rag_chat with hybrid_search=true and query_rewrite=true. The query is rewritten into a crisp search term before embedding (using the same LLM configured for answer generation — no extra keys needed), then hybrid search retrieves the most relevant chunks, and the LLM generates a grounded answer. The response shows the rewritten query so you can see what was actually searched.
Discover what's supported
You: What embedding providers does scrapedatshi support?
Claude calls list_embedding_providers and returns a formatted list with model notes.
You: What fields do I need to configure for Qdrant?
Claude calls list_vector_db_providers and returns the required and optional fields for each provider.
Supported providers
Embedding providers
Key | Provider |
| OpenAI (text-embedding-3-small, text-embedding-3-large, ada-002) |
| Cohere (embed-english-v3.0, embed-multilingual-v3.0) |
| Google Gemini (text-embedding-004, gemini-embedding-001) |
| Mistral (mistral-embed) |
| Voyage AI (voyage-3, voyage-3-lite, voyage-code-3) |
| Ollama local (nomic-embed-text, mxbai-embed-large, etc.) |
Vector databases
Key | Provider |
| Pinecone |
| Qdrant |
| ChromaDB (local) |
| Supabase (pgvector) |
| Weaviate |
| MongoDB Atlas |
| Azure Cosmos DB (NoSQL) |
| Azure Cosmos DB (MongoDB API) |
| LanceDB (local) |
LLM providers (for extraction + contextual retrieval)
Key | Provider |
| OpenAI (gpt-4o-mini, gpt-4o, etc.) |
| Anthropic (claude-3-haiku, claude-3-5-sonnet, etc.) |
| Google Gemini (gemini-1.5-flash, gemini-1.5-pro, etc.) |
Billing
Credits are deducted from your scrapedatshi account after each successful API call
Failed requests are not charged
Every tool response includes
credits_usedandcredits_remainingLLM, embedding, and vector DB costs are billed directly by your chosen providers — scrapedatshi only charges for scraping and orchestration
Top up at scrapedatshi.com/portal/billing
Per-URL rates
Mode | Rate | When |
Local fetch (default) | $0.0020 / URL |
|
Server fetch | $0.0040 / URL |
|
Spider crawl (server) | $0.0050 / URL |
|
Chunk fee | $0.0005 / chunk | All routes |
Injection fee | $0.0030 / chunk | sync_to_vectordb, ingest_file, autorag |
Contextual Retrieval | $0.0010 / chunk | When |
Vector query | $0.0002 / chunk | query_vectordb, rag_chat |
Auto-Batching for Large Sites
When you ask Claude to crawl a large site (more than 200 pages), the autorag and crawl_site tools automatically split the job into sequential batches server-side. You don't need to do anything special — just ask Claude to crawl the site and it handles the rest.
You: Crawl the entire docs.stripe.com site and inject everything into my Pinecone index.
Claude calls autorag with a high max_pages value. If the site has 600 pages, the server processes it as 3 batches of 200 pages each and returns the combined result.
The response will include auto_batched: true and batches_processed: N when batching occurred.
Safety limits
To prevent runaway credit usage and client timeouts:
crawl_site: defaults to 10 pages, maximum 200 per batch (auto-batched for larger jobs)autorag: defaults to 5 pages, no hard upper limit — large jobs are auto-batchedextract_crawl: defaults to 5 pages, maximum 50 per call
Claude will always confirm page limits with you before calling multi-page tools.
Troubleshooting
Contextual Retrieval fails — "model no longer available"
LLM providers periodically deprecate older models. If you see an error like "This model is no longer available", run verify_provider_key again to get the current list of available models for your key, then select a current model.
Current recommended models for contextual retrieval:
Gemini:
gemini-2.5-flashorgemini-2.0-flash-001(notgemini-2.0-flash— deprecated)OpenAI: any current
gpt-4oorgpt-4.1series modelAnthropic: any current
claude-3-5orclaude-3-7series model
Provider model & deprecation pages:
OpenAI: platform.openai.com/docs/deprecations
Anthropic: docs.anthropic.com/en/docs/about-claude/models
Google Gemini: ai.google.dev/gemini-api/docs/models
Cohere: docs.cohere.com/docs/models
Mistral: docs.mistral.ai/getting-started/models
Voyage AI: docs.voyageai.com/docs/embeddings
Contextual Retrieval fails — "quota exceeded"
Your LLM provider API key has no remaining credits. Add credits at your provider's billing page. Note that scrapedatshi credits are separate from your LLM provider credits — you need both.
verify_provider_key returns no models
If key verification succeeds but returns an empty model list, your API key may be restricted to specific model families or your account may have limited access. Check your provider's dashboard for account restrictions.
Claude Desktop doesn't show scrapedatshi tools
Make sure you saved
claude_desktop_config.jsoncorrectly (valid JSON, no trailing commas)Fully quit and reopen Claude Desktop — a simple window close is not enough
Check that
uvxis installed: runuvx --versionin your terminalIf using
--refresh, the first startup may take a few seconds to download the package
License
MIT — see LICENSE
Available Tools
12 toolsautoragA
Full AutoRAG pipeline: crawl an entire domain, chunk every page, embed all chunks, and inject into your vector database — all in a single call.
Use this when the user wants to bulk-ingest an entire website into their vector DB. This combines crawl_site + sync_to_vectordb into one operation.
⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 5 pages. Each page is fetched, chunked, embedded, and injected. For large sites, warn about credit usage and wait times first.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list
Present models to user, ask them to choose one
Call list_vector_db_providers if user is unsure what config fields are needed
Confirm max_pages with the user
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'
If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too
Keys can be omitted if set as environment variables.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain to crawl (e.g. 'https://docs.example.com'). | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector applied to every page before chunking. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| max_pages | No | Maximum pages to crawl and inject. Default: 5. Maximum: 200. Always confirm with user for large sites. | |
| vector_db | Yes | Vector DB provider. Call list_vector_db_providers to see required config fields for each. | |
| chunk_size | No | Target token count per chunk. Default: 512. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site). | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first. | |
| embedding_model | No | Embedding model name from verify_provider_key. Do not guess or hardcode. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/docs/'). | |
| vector_db_config | Yes | Provider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. | |
| embedding_api_key | No | API key for the embedding provider. Can be omitted if set as env var. | |
| embedding_provider | Yes | Embedding provider. Call verify_provider_key(provider, 'embedding') first to get available models. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description details the full pipeline, default settings (max_pages=5), crawl modes, and key handling (env vars). It also explains contextual retrieval's cost and benefits, leaving no behavioral ambiguity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with sections, bullet points, and warnings. Slightly verbose but every sentence serves a purpose. Could be tightened slightly, but excellent overall.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers the complex pipeline thoroughly, including pre-flight checks and parameter interactions. Lacks error-handling details or return value description, but the absence of an output schema makes this acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 100% schema coverage, the description adds critical context: instructions to verify providers, not guess models, confirm limits, and call list_vector_db_providers for config. This far exceeds the schema's basic descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it is a 'Full AutoRAG pipeline' combining crawling, chunking, embedding, and injection into one call. It explicitly distinguishes from sibling tools like crawl_site and sync_to_vectordb.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit when-to-use guidance ('bulk-ingest an entire website'), mandatory pre-flight steps with tool calls, warnings about max_pages and credit usage, and alternative upgrade recommendation (Contextual Retrieval).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
chunk_fileA
Upload a local file, chunk its content into RAG-ready text segments, and return the structured chunks as JSON. No embedding or vector DB required.
Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB
Use this when the user says 'chunk this PDF', 'process this document', 'read this file', or wants to extract text from a local file.
Provide the ABSOLUTE path to the file on the user's local machine (e.g. 'C:/Users/user/Documents/report.pdf' or '/home/user/docs/manual.pdf').
If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:
Call verify_provider_key(provider, 'llm') → get live model list
Ask user to choose a model
Present Contextual Retrieval as a recommended upgrade
LLM keys can be omitted if set as environment variables.
| Name | Required | Description | Default |
|---|---|---|---|
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| file_path | Yes | Absolute path to the local file to chunk. Supported: .pdf, .md, .txt, .yaml, .yml, .json. Example: 'C:/Users/user/Documents/report.pdf' | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| chunk_size | No | Target token count per chunk. Default: 512. Range: 64–4096. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key first. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment. Present as a recommended upgrade. Requires llm_provider and llm_model from verify_provider_key. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It discloses file format support, size limit, absolute path requirement, and the optional contextual retrieval workflow. Lacks details on error handling or file disposal, but adequate for typical use.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with bullet points and ordered sections. Every sentence adds value. Slightly lengthy due to detailed pre-flight sequence, but still efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers input requirements, supported formats, size limit, and optional features. Does not detail return structure or error cases, but the tool's output is implied. Overall sufficient for agent invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but description adds value by explaining absolute path, env var fallback for llm_api_key, and the verify_provider_key dependency for contextual_retrieval parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (upload, chunk) and resource (local file) and distinguishes from siblings like 'ingest_file' and 'autorag' by specifying it returns RAG-ready chunks without embedding or vector DB.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit trigger phrases ('chunk this PDF', 'process this document') and a detailed pre-flight sequence for contextual retrieval. Lacks explicit exclusion cases but gives clear usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_siteA
Crawl an entire website, chunk all pages, and return structured JSON chunks. Two modes: 'sitemap' (reads sitemap.xml — best for docs/blogs) and 'spider' (follows links — works on any site).
Use this when the user wants chunks from MULTIPLE pages WITHOUT extracting structured fields. For structured field extraction across pages, use extract_crawl.
⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 10 pages. For large sites, warn about credit usage first.
If contextual_retrieval is requested, follow the PRE-FLIGHT sequence:
Call verify_provider_key(provider, 'llm') → get live model list
Ask user to choose a model
Ask about JS rendering
Present Contextual Retrieval as a recommended upgrade
LLM keys can be omitted if set as environment variables.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain or sitemap URL to crawl. | |
| selector | No | Optional CSS selector applied to every crawled page. | |
| js_render | No | Use headless browser to render JS before scraping each page. Ask the user before enabling. Adds surcharge per page. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| max_pages | No | Maximum pages to crawl. Default: 10. Maximum: 200. Always confirm with user for large sites. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site). | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key first. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/docs/'). | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment. Present as a recommended upgrade. Requires llm_provider and llm_model from verify_provider_key. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite no annotations, the description discloses behavioral traits like two modes, default max_pages, credit usage warning, surcharge for js_render, and pre-flight requirements. Could mention non-destructive nature, but overall strong.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with sections, bullet points, and front-loaded purpose. Slightly lengthy but every part earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters, no output schema, and no annotations, the description covers the tool's behavior, parameters, and workflow comprehensively. Lacks detail on output format but still strong.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds context beyond the schema, e.g., explaining crawl_mode types, surcharge for js_render, and the pre-flight sequence for contextual_retrieval. Adds value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Crawl an entire website, chunk all pages, and return structured JSON chunks.' It distinguishes two modes and contrasts with extract_crawl for structured field extraction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit guidance is given: use when user wants chunks from multiple pages without structured extraction; use extract_crawl for structured extraction. Includes a pre-flight sequence for contextual_retrieval and warns about confirming max_pages.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_crawlA
Crawl a domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.
Use this when the user wants structured FIELDS from MULTIPLE pages (e.g. extract title + price from every product page on a site).
⚠️ Each page takes 5–15 seconds. Default is 5 pages. For more than 20 pages, warn the user about wait times and credit usage before proceeding.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(provider, 'llm') → get live model list
Present models to user, ask them to choose one
Ask: 'Is this a JavaScript-heavy site?' → js_render (not available for extract_crawl, note this)
Confirm max_pages with the user
LLM keys can be omitted if set as environment variables.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain to crawl. | |
| schema | Yes | Dict mapping field names to description strings. Example: {"title": "string — the product name", "price": "number — price in USD"} | |
| selector | No | Optional CSS selector applied to every page before extraction. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. Advanced models (not mini/flash/haiku) use 30k char context — better for long pages. | |
| max_pages | No | Maximum pages to crawl and extract. Default: 5. Maximum: 50. Always confirm with user before setting above 20. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml. 'spider': follows links from root URL. | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | Yes | LLM provider. One of: 'openai', 'anthropic', 'gemini'. Call verify_provider_key first. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| extract_as_list | No | If true, extracts ALL matching items on each page as a JSON array. | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/products/'). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses page independence, billing for successful extractions, per-page time estimates, and model context size. Missing rate limits or concurrency details, but the key behaviors are well covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured with a clear purpose sentence, use-case line, warnings, and a numbered pre-flight list. Every sentence adds value; no redundancy. Length is appropriate for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers the main aspects: purpose, usage, pre-flight, timing, billing, parameter guidance. However, it lacks description of the output format (what the extracted data looks like) and does not address error handling for non-page failures. Given no output schema, this is a minor gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds significant context: instructions to not hardcode llm_model, to confirm max_pages, and explanation of crawl_mode options. It also describes pre-flight steps that affect parameter usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb-resource pair 'crawl a domain and extract structured data' and distinguishes from siblings like extract_data (single page) and crawl_site (no structured extraction). It provides a concrete use case example.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use (multiple pages with structured fields) and provides a detailed pre-flight checklist including calling verify_provider_key, confirming max_pages, and noting js_render unavailability. Warnings for large page counts are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_dataA
Scrape a URL and extract structured data matching a user-defined schema using an LLM. Returns a JSON object (or array if extract_as_list=true).
Use this when the user wants specific FIELDS from a page (e.g. product name, price, stock status; article author, date, summary).
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(provider, 'llm') → get live model list
Present models to user, ask them to choose one
Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render
Present Contextual Retrieval is NOT applicable here (extraction only)
LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The web URL to scrape and extract structured data from. | |
| schema | Yes | Dict mapping field names to description strings. Example: {"title": "string — the product name", "price": "number — price in USD", "in_stock": "boolean — whether in stock"} | |
| selector | No | Optional CSS selector to target a specific section before extraction. | |
| js_render | No | Use headless browser to render JS before extracting. Ask the user before enabling. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. Use an advanced model (not mini/flash/haiku) for long-form pages like documentation or legal docs. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | Yes | LLM provider. One of: 'openai', 'anthropic', 'gemini'. Call verify_provider_key first. | |
| click_selector | No | CSS selector for an element to click after page load (tabs, accordions, load-more). Only used when js_render=true. | |
| extract_as_list | No | If true, extracts ALL matching items on the page as a JSON array. Use for listing pages (product catalogues, article feeds). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that the tool uses an LLM for extraction, requires a provider key verification, supports optional headless rendering, and allows API keys via env vars. However, it does not mention rate limits, error handling, timeouts, or any side effects (though likely read-only). The transparency is adequate but could be more comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a main purpose, usage examples, and a clear 'PRE-FLIGHT REQUIRED' section. It is concise, with each sentence adding value. The use of bullet points in the pre-flight steps enhances readability. Slightly verbose in parts but overall efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 parameters, no output schema), the description covers the main purpose, pre-flight process, and parameter details. However, it lacks details on the output format beyond 'returns a JSON object', error handling, or what happens if the schema cannot be matched. For a tool with nested objects and multiple optional parameters, this is a notable gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, so the baseline is 3. The description adds significant meaning beyond the schema: it explains the purpose of the 'schema' parameter with an example, advises using advanced models for long pages, clarifies that 'llm_api_key' can be omitted if set as an env var, and explains when to use 'extract_as_list'. This extra context improves understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description starts with a specific verb+resource combination ('scrape a URL and extract structured data') and includes examples of use cases (e.g., product name, price, stock status). It clearly conveys the tool's function of extracting user-defined fields via LLM. However, it does not explicitly differentiate from its sibling tool 'scrape_url', which also deals with URL content, leaving some ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear guidance on when to use this tool (when the user wants specific fields from a page) and includes a 'PRE-FLIGHT REQUIRED' section with step-by-step instructions for the agent (call verify_provider_key, present models, ask about js_render). It also clarifies that Contextual Retrieval is not applicable. However, it does not explicitly state when NOT to use it or name alternative tools like 'scrape_url' for raw HTML extraction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_usage_guideA
Returns the complete guided workflow for using scrapedatshi tools. Call this at the start of any scrapedatshi conversation to understand which tool to use for each task and the required pre-flight sequence.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It states the tool returns a workflow, but does not disclose behavioral traits such as side effects, rate limits, or whether it requires authentication. However, as a read-only guide, its behavior is relatively simple.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences with no wasted words. It front-loads the purpose and provides a clear usage scenario. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description could elaborate on what the guided workflow includes (e.g., tool list, sequence steps). It only says 'complete guided workflow', which is adequate but not fully descriptive for an agent to set expectations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has no parameters, so the input schema provides full coverage. The description does not need to add parameter information. Baseline for 0 parameters is 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns a 'complete guided workflow' for using scrapedatshi tools. It specifies the action ('returns') and the resource ('guided workflow'), and distinguishes itself from sibling action tools by being a meta-guide.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises to call it 'at the start of any scrapedatshi conversation' and explains why (to understand tool selection and pre-flight sequence). It does not mention alternatives, but the context is clear enough for an agent to know when to invoke it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ingest_fileA
Full RAG pipeline for local files: upload a file, embed the chunks using your embedding provider, and inject the vectors into your vector database.
Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB
Use this when the user wants to ADD a local document (PDF, markdown, etc.) to their vector DB. This is the file-based equivalent of sync_to_vectordb.
Provide the ABSOLUTE path to the file on the user's local machine.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list
Present models to user, ask them to choose one
Call list_vector_db_providers if user is unsure what config fields are needed
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'
If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too
Keys can be omitted if set as environment variables.
| Name | Required | Description | Default |
|---|---|---|---|
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| file_path | Yes | Absolute path to the local file to ingest. Supported: .pdf, .md, .txt, .yaml, .yml, .json. Example: 'C:/Users/user/Documents/report.pdf' | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db | Yes | Vector DB provider. Call list_vector_db_providers to see required config fields for each. | |
| chunk_size | No | Target token count per chunk. Default: 512. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first. | |
| embedding_model | No | Embedding model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db_config | Yes | Provider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. | |
| embedding_api_key | No | API key for the embedding provider. Can be omitted if set as env var. | |
| embedding_endpoint | No | Public HTTPS endpoint for Ollama only (e.g. from ngrok). Not needed for cloud providers. | |
| embedding_provider | Yes | Embedding provider. Call verify_provider_key(provider, 'embedding') first to get available models. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. Describes the full pipeline, supported formats, max file size, and pre-flight requirements. Mentions cost for contextual retrieval and that keys can be omitted as env vars. Does not detail error handling or edge cases, but provides substantial behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Well-structured: purpose first, then supported formats/size, then usage instruction, then numbered pre-flight list. Sentences are informative and not redundant. Slightly long but each part earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers purpose, usage, parameters, pre-flight dependencies. Missing description of return value/output (no output schema provided). Does not explain error scenarios or what happens on failure. Adequate for a complex tool but incomplete in capturing all aspects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% description coverage. Description adds some value by reinforcing that file_path must be absolute, embedding_model should come from verify_provider_key, and pre-flight steps relate to parameters. However, the schema already covers these details, so the description provides marginal additional meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Full RAG pipeline for local files: upload a file, embed the chunks... and inject the vectors into your vector database.' It distinguishes from sibling tools like sync_to_vectordb by calling itself the file-based equivalent, and from chunk_file by implying it does the full pipeline.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'Use this when the user wants to ADD a local document... to their vector DB.' Provides a detailed pre-flight checklist. Does not explicitly say when not to use or list alternatives beyond the sibling, but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_embedding_providersA
Returns all supported embedding providers with labels and notes. Call this to help the user choose an embedding provider before sync_to_vectordb. After the user chooses, call verify_provider_key to get the live model list.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description explains it returns labels and notes, which is sufficient for a read-only listing tool. No annotations are present, so the description carries the burden; it is brief but adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with the primary action, no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no parameters, the description covers the return values and workflow steps. It is reasonably complete for a helper tool, though could mention if the list is static.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist, so the description does not need to add parameter info. The schema coverage is 100%, meeting the baseline for zero parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns all supported embedding providers with labels and notes. It distinguishes itself from siblings by specifying its role before sync_to_vectordb.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly advises when to call (before sync_to_vectordb) and the next step (call verify_provider_key after user choice), providing a clear workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_vector_db_providersA
Returns all supported vector database providers with required config fields, optional fields, and setup notes. Call this before sync_to_vectordb to help the user understand what vector_db_config fields they need to provide.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description doesn't add behavioral traits beyond purpose. For a read-only list, minimal but adequate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences front-loading the purpose and usage, with no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with no parameters and no output schema, the description adequately covers what the tool returns and why it should be called.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters exist; schema coverage is 100%. Description adds value by explaining what the output contains (config fields, optional fields, setup notes).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states it returns supported vector database providers with config fields and setup notes, and distinguishes itself from sibling tools like list_embedding_providers.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Clearly says to call this before sync_to_vectordb to understand required fields, providing explicit when-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_urlA
Scrape a single web URL, chunk its content into RAG-ready text segments, and return the structured chunks. No embedding or vector DB required — this is the fastest and cheapest operation.
Use this when the user wants to read, summarize, or process the content of a specific web page WITHOUT extracting structured fields.
If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:
Call verify_provider_key(provider, 'llm') → get live model list
Ask user to choose a model from the list
Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'
LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The web URL to scrape and chunk. | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector to target a specific element (e.g. 'article', '.content', 'main'). | |
| js_render | No | Use headless Chromium to render JavaScript before scraping. Required for SPAs and JS-heavy pages. Ask the user before enabling. Adds a small surcharge. | |
| llm_model | No | LLM model name. MUST be chosen from the list returned by verify_provider_key — do not guess or hardcode. | |
| chunk_size | No | Target token count per chunk. Default: 512. Range: 64–4096. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. One of: 'openai', 'anthropic', 'gemini'. Verify with verify_provider_key first. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment. An LLM generates a unique context string for each chunk, boosting retrieval accuracy by 35–50%. Present this as a recommended upgrade. Requires llm_provider and llm_model (from verify_provider_key). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses chunking behavior, cost (surcharge for js_render, ~$0.001/chunk for contextual retrieval), prerequisite steps (verify_provider_key), and important flags (js_render for SPAs). However, it omits output format details, error handling, rate limits, or caching behavior, which would increase transparency for a tool with 9 parameters.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise in the first paragraph but includes a detailed multi-step pre-flight sequence that, while informative, makes it longer than necessary. Every sentence serves a purpose, but the pre-flight instructions could be more succinct. Overall structured logically: purpose, usage, special case.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 9 parameters, no output schema, and moderate complexity. The description covers the main use case and the contextual retrieval flow. However, it does not describe the return format (e.g., array of chunks with text, metadata) or error handling. Without output schema, the agent needs more detail on what to expect. Given the richness of the description for the main flow, it's adequate but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds value by explaining when to use js_render (JavaScript-heavy pages, SPAs), that llm_model must come from verify_provider_key, and that llm_api_key can be an env var. The pre-flight sequence for contextual_retrieval adds operational context beyond schema. However, some schema descriptions already suffice, limiting extra contribution.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool scrapes a single URL, chunks it into RAG-ready segments, and returns structured chunks. It distinguishes from sibling tools like extract_data (which extracts structured fields) and crawl_site (which does multiple pages). The verb 'scrape' and resource 'single web URL' are specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: 'when the user wants to read, summarize, or process content of a specific web page WITHOUT extracting structured fields.' Provides a detailed pre-flight sequence for contextual_retrieval. Does not explicitly list when not to use, but the positive criteria imply exclusion of structured extraction. Sibling names provide context, but description could be more explicit about alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
sync_to_vectordbA
Full RAG pipeline: scrape a URL, embed the chunks using your embedding provider, and inject the vectors into your vector database — all in one call.
Use this when the user wants to ADD web content to their vector DB for later retrieval. The user brings their own embedding provider and vector DB.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list
Present models to user, ask them to choose one
Call list_vector_db_providers if user is unsure what config fields are needed
Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra. If yes, I'll also need your LLM provider and model.'
If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too
Keys can be omitted if set as environment variables (OPENAI_API_KEY, PINECONE_API_KEY, etc.).
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The web URL to scrape, embed, and inject into the vector DB. | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector to target a specific page section. | |
| js_render | No | Use headless browser to render JS before scraping. Ask the user before enabling. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db | Yes | Vector DB provider. Call list_vector_db_providers to see required config fields for each. | |
| chunk_size | No | Target token count per chunk. Default: 512. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first. | |
| embedding_model | No | Embedding model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db_config | Yes | Provider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. Examples: pinecone: {"index_host": "https://my-index.svc.pinecone.io"} | qdrant: {"url": "https://cluster.qdrant.io", "collection_name": "docs"} | supabase: {"connection_string": "postgresql://...", "table_name": "documents"} | chroma: {"collection_name": "docs"} | |
| embedding_api_key | No | API key for the embedding provider. Can be omitted if set as env var. | |
| embedding_endpoint | No | Public HTTPS endpoint for Ollama only (e.g. from ngrok). Not needed for cloud providers. | |
| embedding_provider | Yes | Embedding provider. Call verify_provider_key(provider, 'embedding') first to get available models. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses the composite operation, pre-flight requirements, headless browser for js_render, and extra cost for contextual_retrieval. It does not cover rate limits, error handling, or idempotency, but provides sufficient transparency for a complex tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy but well-structured with numbered steps and bullet points. It front-loads the purpose and systematically guides the user. Slightly verbose but efficient for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given high complexity (15 params, nested objects, no output schema), the description covers the workflow and pre-flight thoroughly. It lacks explicit return value description, but the process is clear enough for an agent to proceed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds value by explaining workflow context (e.g., embedding_endpoint only for Ollama, env var fallbacks, and pre-flight steps). It does not repeat schema but enhances usability beyond schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool is a full RAG pipeline that scrapes, embeds, and injects into a vector DB. It explicitly says 'Use this when the user wants to ADD web content to their vector DB for later retrieval,' distinguishing it from sibling tools like chunk_file or ingest_file that handle only parts of the pipeline.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage context ('ADD web content') and a detailed pre-flight checklist with steps. However, it does not explicitly mention when NOT to use this tool or compare directly with siblings like autorag, leaving some ambiguity about alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
verify_provider_keyA
Verify an LLM or embedding API key and return the live list of models available for that key. Call this BEFORE any operation that requires an LLM or embedding provider — never assume or hardcode model names.
Returns: key validity, list of available model names (live from the provider's API), and an error message if the key is invalid.
Supported LLM providers: openai, anthropic, gemini Supported embedding providers: openai, cohere, gemini, mistral, voyage
The API key can be omitted if the corresponding env var is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, COHERE_API_KEY, MISTRAL_API_KEY, VOYAGE_API_KEY).
| Name | Required | Description | Default |
|---|---|---|---|
| api_key | No | The API key to verify. Can be omitted if the corresponding env var is set. | |
| provider | Yes | Provider to verify. LLM: 'openai', 'anthropic', 'gemini'. Embedding: 'openai', 'cohere', 'gemini', 'mistral', 'voyage'. | |
| provider_type | Yes | 'llm' for text generation models (used in extract_data, extract_crawl, contextual_retrieval). 'embedding' for vector embedding models (used in sync_to_vectordb). |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses return values (key validity, model list, error message) and mentions that API keys can be omitted. It does not mention side effects, rate limits, or idempotency, but for a verification tool, the behavioral traits are adequately covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, comprising three clear sentences that front-load the primary purpose. It states the action, return values, and supported providers without any extraneous information. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given there is no output schema, the description fully describes return values (key validity, model list, error). It lists all supported providers and explains how the key can be omitted. The tool is straightforward, and the description covers everything needed for an agent to use it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds value by explaining that 'provider_type' indicates whether models are used for LLM or embedding, with examples of where they are applied (extract_data, sync_to_vectordb). It also elaborates on the 'api_key' parameter's optionality with env var names, going beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: verifying an API key and returning live available models. It uses specific verbs ('verify', 'return') and identifies the resource (LLM/embedding API key). This clearly distinguishes it from sibling tools like 'extract_data' or 'sync_to_vectordb' which perform different operations.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool: 'Call this BEFORE any operation that requires an LLM or embedding provider' and advises against hardcoding model names. It lists supported providers and notes that the API key can be omitted if env vars are set. However, it does not explicitly discuss scenarios where the tool should not be used or contrast it with alternatives, though the context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
12 tool updates
v0.2.1- First observed
autorag - First observed
chunk_file - First observed
crawl_site - First observed
extract_crawl - First observed
extract_data - First observed
get_usage_guide - First observed
ingest_file - First observed
list_embedding_providers - First observed
list_vector_db_providers - First observed
scrape_url - First observed
sync_to_vectordb - First observed
verify_provider_key
TDQS
Tools are mostly distinct, with clear purposes like scrape_url vs crawl_site vs extract_data. However, some overlap exists: autorag combines crawl_site and sync_to_vectordb, and ingest_file is similar to the file-based equivalent. Descriptions help disambiguate.
Naming follows a consistent verb_noun pattern (scrape_url, list_embedding_providers, verify_provider_key). Minor deviations: autorag is a brand name, get_usage_guide uses 'get' instead of a verb like 'show', and sync_to_vectordb has a preposition. Overall mostly consistent.
12 tools is an appropriate number for a web scraping and RAG pipeline server. Each tool covers a distinct step or combination of steps, and the count is neither too few nor too many for the scope.
The tool set covers the full workflow: provider discovery, key verification, scraping (single/multi page, structured extraction), file processing, and full RAG pipelines (chunk, embed, inject). Includes a usage guide. No obvious gaps for the stated purpose.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables web scraping, crawling, structured data extraction, and browser automation through multiple AI agents including OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use.17MIT
- AlicenseNot gradedqualityDmaintenanceEnables Claude to scrape and crawl websites via a self-hosted Firecrawl instance through the Model Context Protocol. Provides tools for single/multi-URL scraping, site mapping, and full-site crawling operations.28,8531MIT
- FlicenseAqualityDmaintenanceEnables Claude Desktop to perform advanced web scraping and crawling operations, extracting structured data, analyzing website architectures, and discovering content relationships through natural conversation.54-

spidra-mcp-serverofficial
AlicenseAqualityBmaintenanceEnables AI assistants to scrape pages, batch-process URLs, and crawl entire websites with AI-powered extraction.1237MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/scrapedatshi/scrapedatshi-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server