Skip to main content
Glama

scrapedatshi-mcp

MCP (Model Context Protocol) server for the scrapedatshi RAG pipeline API.

Use scrapedatshi's scraping, crawling, extraction, and vector DB sync tools directly from Claude Desktop — no code required.


What you can do

Just talk to Claude naturally:

  • "Scrape https://docs.example.com and give me the chunks"

  • "Extract the text from this PDF: https://example.com/annual-report.pdf"

  • "Extract all tables from this local PDF: C:/Users/me/Documents/financials.pdf"

  • "Chunk this PDF URL: https://my-bucket.s3.amazonaws.com/report.pdf" — PDF URLs are automatically detected and extracted

  • "Crawl https://example.com/products and extract the title and price from every page"

  • "Sync https://docs.example.com to my Pinecone index using OpenAI embeddings"

  • "Crawl the entire docs.stripe.com site (all 800 pages) and inject it into my Pinecone index" — large sites are auto-batched server-side, no manual pagination needed

  • "What embedding providers does scrapedatshi support?"

  • "Inspect my Pinecone index and tell me what embedding model was used"

  • "Query my Pinecone index for information about API authentication"

  • "Query my LanceDB with hybrid search — I need to find exact IDs and names, not just semantic matches"

  • "Chunk https://docs.example.com using hierarchical chunking so the LLM gets full context on retrieval"

  • "Ingest all the JSON files in my ./scrapy_output/ folder into my Pinecone index"


Related MCP server: Firecrawl MCP Server

Tools exposed

Tool

What it does

verify_provider_key

Verify an LLM or embedding API key + get live model list

get_usage_guide

Returns the guided wizard flow and tool selection reference

scrape_url

Scrape a URL and return clean Markdown — no chunking, just the raw text

pdf_extract

Extract text or tables from a PDF (URL or local file) — no chunking, no embedding needed

chunk_url

Scrape & chunk a single URL into RAG-ready text segments

chunk_file

Upload a local file (PDF, MD, TXT, CSV, XLSX, DOCX, IPYNB, HTML, XML, code files, etc.) and chunk it into RAG-ready segments

crawl_site

Crawl an entire site (sitemap or spider mode) and return all chunks

extract_data

Extract structured schema fields from a URL using your LLM

extract_crawl

Multi-page schema extraction via site crawl

sync_to_vectordb

Full pipeline: scrape URL → embed → inject into your vector DB

ingest_file

Full pipeline: upload local file → embed → inject into your vector DB

ingest_scraped

Full pipeline: bulk-ingest a folder of pre-scraped files → embed → inject into your vector DB

autorag

Full pipeline: crawl entire site → chunk → embed → inject into your vector DB (large sites auto-batched)

inspect_vectordb

Read vector DB metadata: dimension, vector count, suggested embedding models (free)

query_vectordb

Semantic search: embed a query and retrieve the most relevant chunks from your vector DB. Supports hybrid_search=true (vector + BM25 + RRF) and optional query_rewrite (LLM-powered query rewriting before embedding)

rag_chat

RAG Chat: retrieve top-N chunks from your vector DB and generate a grounded LLM answer. Supports hybrid_search=true, query_rewrite=true (uses the same LLM — no extra keys), and conversation_history for pronoun resolution

list_embedding_providers

Discover supported embedding providers + model notes

list_vector_db_providers

Discover supported vector DBs + required config fields


Prerequisites

  1. scrapedatshi accountSign up at scrapedatshi.com

  2. Add creditsBilling portal

  3. Get your API key — starts with sds_...

  4. Claude DesktopDownload here

  5. Python 3.10+python.org


Installation

pip install scrapedatshi-mcp

Or use uv for isolated installs:

uv tool install scrapedatshi-mcp

Option B — Install from source (local development)

git clone https://github.com/scrapedatshi/scrapedatshi-mcp.git
cd scrapedatshi-mcp
pip install -e .

Claude Desktop configuration

Easiest way to find your config file: Open Claude Desktop → SettingsDeveloperEdit Config

Alternatively, the file is located at:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": [
        "--from", "scrapedatshi-mcp[all]",
        "--refresh",
        "scrapedatshi-mcp"
      ],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here"
      }
    }
  }
}
  • [all] installs all provider SDKs (OpenAI, Anthropic, Gemini, Voyage AI) so verify_provider_key works for any provider

  • --refresh checks PyPI for updates every time Claude Desktop starts — no manual reinstalls needed

If installed via pip (using python)

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "python",
      "args": ["-m", "scrapedatshi_mcp.server"],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here"
      }
    }
  }
}

If cloned from source (absolute path)

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "python",
      "args": ["/absolute/path/to/scrapedatshi-mcp/scrapedatshi_mcp/server.py"],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here"
      }
    }
  }
}

Restart Claude Desktop after saving the config.


Secure key configuration (BYOK)

You bring your own LLM, embedding, and vector DB keys. The server resolves keys in this priority order:

  1. Argument passed in the tool call — explicit override

  2. Environment variable in the MCP config — preferred secure path (keys never appear in chat)

  3. Clear error message if neither is found

Add your provider keys to the env block in claude_desktop_config.json:

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": [
        "--from", "scrapedatshi-mcp[all]",
        "--refresh",
        "scrapedatshi-mcp"
      ],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here",

        "OPENAI_API_KEY": "sk-...",
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "GEMINI_API_KEY": "AIza...",

        "COHERE_API_KEY": "...",
        "MISTRAL_API_KEY": "...",
        "VOYAGE_API_KEY": "...",

        "PINECONE_API_KEY": "pc-...",
        "QDRANT_API_KEY": "...",
        "WEAVIATE_API_KEY": "..."
      }
    }
  }
}

Once set, Claude will automatically use these keys without asking you to type them in chat.


Fetch Mode

Starting in v0.5.0, the MCP server uses local-fetch mode by default — URLs are fetched on the machine running Claude Desktop (your IP), and only the HTML processing runs on our server. This is cheaper and keeps your IP off our server.

SCRAPEDATSHI_FETCH_MODE=local (default)

The MCP server fetches URLs using the machine's own IP address, then submits the raw HTML to our server for processing.

  • ✅ Your IP is used — not our server's

  • ✅ Billed at the standard per-URL rate ($0.0020)

  • ✅ Faster — no double-hop latency

SCRAPEDATSHI_FETCH_MODE=server

Our server fetches the URL. Use this if Claude Desktop is running in a restricted environment without outbound HTTP access, or if you need server-managed IP rotation.

  • ⚠️ Our server's IP is used

  • ⚠️ Billed at 2× the standard rate ($0.0040 / URL)

  • ✅ Works from restricted environments

To enable server fetch, add SCRAPEDATSHI_FETCH_MODE to your MCP config:

{
  "mcpServers": {
    "scrapedatshi": {
      "command": "uvx",
      "args": ["--from", "scrapedatshi-mcp[all]", "--refresh", "scrapedatshi-mcp"],
      "env": {
        "SCRAPEDATSHI_API_KEY": "sds_your_key_here",
        "SCRAPEDATSHI_FETCH_MODE": "server"
      }
    }
  }
}

Supported environment variables

Variable

Used for

SCRAPEDATSHI_API_KEY

scrapedatshi API key (required)

SCRAPEDATSHI_FETCH_MODE

local (default) or server — see Fetch Mode above

OPENAI_API_KEY

OpenAI LLM + embedding

ANTHROPIC_API_KEY

Anthropic LLM (Claude)

GEMINI_API_KEY

Google Gemini LLM + embedding

COHERE_API_KEY

Cohere embedding

MISTRAL_API_KEY

Mistral embedding

VOYAGE_API_KEY

Voyage AI embedding

PINECONE_API_KEY

Pinecone vector DB

QDRANT_API_KEY

Qdrant vector DB (optional for local)

WEAVIATE_API_KEY

Weaviate vector DB (optional for local)


Authenticated Scraping (v0.5.1+)

For pages behind a login wall, you can pass your session cookies and/or custom headers to scrape_url and crawl_site. Credentials are only sent to URLs within the permitted domain scope — they are never leaked to external domains.

Scrape a login-walled page

Just tell Claude:

"Scrape https://internal.company.com/wiki/api-docs — use my session cookie: abc123"

Claude will call scrape_url with:

{
  "url": "https://internal.company.com/wiki/api-docs",
  "cookies": {"session": "abc123"},
  "headers": {"Authorization": "Bearer eyJ..."}
}

Authenticated crawl with subdomain scope

"Crawl https://company.com including wiki.company.com and docs.company.com — use session cookie abc123"

Claude will call crawl_site with:

{
  "url": "https://company.com",
  "cookies": {"session": "abc123"},
  "allow_subdomains": true,
  "max_pages": 20
}

Security model:

  • Cookies and headers are only sent to URLs within the permitted domain scope — never to external domains discovered during crawling

  • allow_subdomains: false (default): only the exact hostname receives credentials

  • allow_subdomains: true: credentials are shared with subdomains of the root domain (e.g. wiki.company.com when root is company.com). Multi-part TLDs (.co.uk, .com.br) are handled safely.

  • Credentials are never forwarded to the scrapedatshi server — they stay on the machine running Claude Desktop

Enterprise SSO / MFA — Session Capture (v0.6.4+)

For enterprise portals protected by Okta, Duo, or any SSO/MFA flow that blocks automated login, use the SDK's capture_session() utility to authenticate manually in a real browser, then pass the captured session state to Claude via the storage_state parameter.

Step 1 — Capture the session locally (run once):

pip install scrapedatshi[auth]
playwright install chromium
from scrapedatshi.auth import capture_session
import json

state = capture_session(
    "https://internal.company.com/login",
    save_to="session.auth.json",   # gitignored automatically
)

This opens a real browser window. Log in manually (including any MFA prompts), then press Enter. The session state is saved to session.auth.json.

Step 2 — Tell Claude to use the saved session:

"Crawl https://internal.company.com using the session state in session.auth.json"

Claude will call crawl_site with the storage_state parameter containing the captured session.

⚠ Security: session.auth.json contains live authentication tokens. Never commit it to version control. The SDK's .gitignore template automatically filters *.auth.json files.


Example conversations

Get clean Markdown from a page

You: Scrape https://docs.example.com/getting-started and show me the content.

Claude calls scrape_url and returns the full page as clean Markdown — title, credits used, and the raw text in one piece.

The response also includes selectors_found — a list of CSS selectors for the main content sections detected on the page. Claude can use these to re-scrape just a specific section:

You: Now scrape just the pricing section.

Claude calls scrape_url again with selector="section#pricing" (from selectors_found).


Chunk a page for RAG

You: Chunk https://docs.example.com/getting-started into RAG-ready segments.

Claude calls chunk_url and returns the structured chunks with token counts and credit usage.


Crawl a documentation site

You: Crawl https://docs.example.com — just the first 5 pages.

Claude calls crawl_site with max_pages=5 and returns all chunks from all pages.


Extract structured data from a product page

You: Extract the product name, price, and whether it's in stock from https://example.com/products/widget-pro

Claude calls extract_data with a schema it constructs from your request, using your OpenAI key from the env config.


Extract data from an entire product catalogue

You: Crawl https://example.com/products and extract the title and price from every product page. Limit to 10 pages.

Claude calls extract_crawl with max_pages=10 and returns per-page extraction results.


Sync a page to your vector DB

You: Sync https://docs.example.com to my Pinecone index. The index host is https://my-index-abc123.svc.pinecone.io. Use OpenAI text-embedding-3-small.

Claude calls sync_to_vectordb. If OPENAI_API_KEY and PINECONE_API_KEY are set in your env config, no keys need to be typed in chat.


Query your vector database

You: Query my Pinecone index for "how do I authenticate with the API?"

Claude calls inspect_vectordb first to confirm the embedding model, then calls query_vectordb and returns the top matching chunks.


Hybrid search — exact terms + semantic

You: Query my LanceDB for "Project Alpha manager" using hybrid search — I need exact name matching.

Claude calls query_vectordb with hybrid_search=true, combining BM25 keyword search with vector similarity using Reciprocal Rank Fusion (RRF). Results include rrf_score and hybrid_sources showing which search method found each chunk.


RAG chat with query rewriting

You: Ask my knowledge base: "what about the second pricing tier?" — use hybrid search and rewrite the query first.

Claude calls rag_chat with hybrid_search=true and query_rewrite=true. The query is rewritten into a crisp search term before embedding (using the same LLM configured for answer generation — no extra keys needed), then hybrid search retrieves the most relevant chunks, and the LLM generates a grounded answer. The response shows the rewritten query so you can see what was actually searched.


Discover what's supported

You: What embedding providers does scrapedatshi support?

Claude calls list_embedding_providers and returns a formatted list with model notes.

You: What fields do I need to configure for Qdrant?

Claude calls list_vector_db_providers and returns the required and optional fields for each provider.


Supported providers

Embedding providers

Key

Provider

openai

OpenAI (text-embedding-3-small, text-embedding-3-large, ada-002)

cohere

Cohere (embed-english-v3.0, embed-multilingual-v3.0)

gemini

Google Gemini (text-embedding-004, gemini-embedding-001)

mistral

Mistral (mistral-embed)

voyage

Voyage AI (voyage-3, voyage-3-lite, voyage-code-3)

ollama

Ollama local (nomic-embed-text, mxbai-embed-large, etc.)

Vector databases

Key

Provider

pinecone

Pinecone

qdrant

Qdrant

chroma

ChromaDB (local)

supabase

Supabase (pgvector)

weaviate

Weaviate

mongodb

MongoDB Atlas

azure_cosmos

Azure Cosmos DB (NoSQL)

azure_cosmos_mongo

Azure Cosmos DB (MongoDB API)

lancedb

LanceDB (local)

LLM providers (for extraction + contextual retrieval)

Key

Provider

openai

OpenAI (gpt-4o-mini, gpt-4o, etc.)

anthropic

Anthropic (claude-3-haiku, claude-3-5-sonnet, etc.)

gemini

Google Gemini (gemini-1.5-flash, gemini-1.5-pro, etc.)


Billing

  • Credits are deducted from your scrapedatshi account after each successful API call

  • Failed requests are not charged

  • Every tool response includes credits_used and credits_remaining

  • LLM, embedding, and vector DB costs are billed directly by your chosen providers — scrapedatshi only charges for scraping and orchestration

  • Top up at scrapedatshi.com/portal/billing

Per-URL rates

Mode

Rate

When

Local fetch (default)

$0.0020 / URL

SCRAPEDATSHI_FETCH_MODE=local (default)

Server fetch

$0.0040 / URL

SCRAPEDATSHI_FETCH_MODE=server

Spider crawl (server)

$0.0050 / URL

/v1/spider — server-side link-following

Chunk fee

$0.0005 / chunk

All routes

Injection fee

$0.0030 / chunk

sync_to_vectordb, ingest_file, autorag

Contextual Retrieval

$0.0010 / chunk

When contextual_retrieval=true

Vector query

$0.0002 / chunk

query_vectordb, rag_chat


Auto-Batching for Large Sites

When you ask Claude to crawl a large site (more than 200 pages), the autorag and crawl_site tools automatically split the job into sequential batches server-side. You don't need to do anything special — just ask Claude to crawl the site and it handles the rest.

You: Crawl the entire docs.stripe.com site and inject everything into my Pinecone index.

Claude calls autorag with a high max_pages value. If the site has 600 pages, the server processes it as 3 batches of 200 pages each and returns the combined result.

The response will include auto_batched: true and batches_processed: N when batching occurred.


Safety limits

To prevent runaway credit usage and client timeouts:

  • crawl_site: defaults to 10 pages, maximum 200 per batch (auto-batched for larger jobs)

  • autorag: defaults to 5 pages, no hard upper limit — large jobs are auto-batched

  • extract_crawl: defaults to 5 pages, maximum 50 per call

Claude will always confirm page limits with you before calling multi-page tools.


Troubleshooting

Contextual Retrieval fails — "model no longer available"

LLM providers periodically deprecate older models. If you see an error like "This model is no longer available", run verify_provider_key again to get the current list of available models for your key, then select a current model.

Current recommended models for contextual retrieval:

  • Gemini: gemini-2.5-flash or gemini-2.0-flash-001 (not gemini-2.0-flash — deprecated)

  • OpenAI: any current gpt-4o or gpt-4.1 series model

  • Anthropic: any current claude-3-5 or claude-3-7 series model

Provider model & deprecation pages:

  • OpenAI: platform.openai.com/docs/deprecations

  • Anthropic: docs.anthropic.com/en/docs/about-claude/models

  • Google Gemini: ai.google.dev/gemini-api/docs/models

  • Cohere: docs.cohere.com/docs/models

  • Mistral: docs.mistral.ai/getting-started/models

  • Voyage AI: docs.voyageai.com/docs/embeddings


Contextual Retrieval fails — "quota exceeded"

Your LLM provider API key has no remaining credits. Add credits at your provider's billing page. Note that scrapedatshi credits are separate from your LLM provider credits — you need both.


verify_provider_key returns no models

If key verification succeeds but returns an empty model list, your API key may be restricted to specific model families or your account may have limited access. Check your provider's dashboard for account restrictions.


Claude Desktop doesn't show scrapedatshi tools

  1. Make sure you saved claude_desktop_config.json correctly (valid JSON, no trailing commas)

  2. Fully quit and reopen Claude Desktop — a simple window close is not enough

  3. Check that uvx is installed: run uvx --version in your terminal

  4. If using --refresh, the first startup may take a few seconds to download the package


License

MIT — see LICENSE

Available Tools

12 tools
autoragA

Full AutoRAG pipeline: crawl an entire domain, chunk every page, embed all chunks, and inject into your vector database — all in a single call.

Use this when the user wants to bulk-ingest an entire website into their vector DB. This combines crawl_site + sync_to_vectordb into one operation.

⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 5 pages. Each page is fetched, chunked, embedded, and injected. For large sites, warn about credit usage and wait times first.

PRE-FLIGHT REQUIRED — before calling:

  1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

  2. Present models to user, ask them to choose one

  3. Call list_vector_db_providers if user is unsure what config fields are needed

  4. Confirm max_pages with the user

  5. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

  6. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe root domain to crawl (e.g. 'https://docs.example.com').
overlapNoToken overlap between consecutive chunks. Default: 50.
selectorNoOptional CSS selector applied to every page before chunking.
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode.
max_pagesNoMaximum pages to crawl and inject. Default: 5. Maximum: 200. Always confirm with user for large sites.
vector_dbYesVector DB provider. Call list_vector_db_providers to see required config fields for each.
chunk_sizeNoTarget token count per chunk. Default: 512.
crawl_modeNo'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site).sitemap
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first.
embedding_modelNoEmbedding model name from verify_provider_key. Do not guess or hardcode.
exclude_patternNoSkip URLs containing this substring (e.g. '/blog/').
include_patternNoOnly crawl URLs containing this substring (e.g. '/docs/').
vector_db_configYesProvider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars.
embedding_api_keyNoAPI key for the embedding provider. Can be omitted if set as env var.
embedding_providerYesEmbedding provider. Call verify_provider_key(provider, 'embedding') first to get available models.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model.

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description details the full pipeline, default settings (max_pages=5), crawl modes, and key handling (env vars). It also explains contextual retrieval's cost and benefits, leaving no behavioral ambiguity.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with sections, bullet points, and warnings. Slightly verbose but every sentence serves a purpose. Could be tightened slightly, but excellent overall.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the complex pipeline thoroughly, including pre-flight checks and parameter interactions. Lacks error-handling details or return value description, but the absence of an output schema makes this acceptable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 100% schema coverage, the description adds critical context: instructions to verify providers, not guess models, confirm limits, and call list_vector_db_providers for config. This far exceeds the schema's basic descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it is a 'Full AutoRAG pipeline' combining crawling, chunking, embedding, and injection into one call. It explicitly distinguishes from sibling tools like crawl_site and sync_to_vectordb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use guidance ('bulk-ingest an entire website'), mandatory pre-flight steps with tool calls, warnings about max_pages and credit usage, and alternative upgrade recommendation (Contextual Retrieval).

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

chunk_fileA

Upload a local file, chunk its content into RAG-ready text segments, and return the structured chunks as JSON. No embedding or vector DB required.

Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB

Use this when the user says 'chunk this PDF', 'process this document', 'read this file', or wants to extract text from a local file.

Provide the ABSOLUTE path to the file on the user's local machine (e.g. 'C:/Users/user/Documents/report.pdf' or '/home/user/docs/manual.pdf').

If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Ask user to choose a model

  3. Present Contextual Retrieval as a recommended upgrade

LLM keys can be omitted if set as environment variables.

ParametersJSON Schema
NameRequiredDescriptionDefault
overlapNoToken overlap between consecutive chunks. Default: 50.
file_pathYesAbsolute path to the local file to chunk. Supported: .pdf, .md, .txt, .yaml, .yml, .json. Example: 'C:/Users/user/Documents/report.pdf'
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode.
chunk_sizeNoTarget token count per chunk. Default: 512. Range: 64–4096.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. Verify with verify_provider_key first.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment. Present as a recommended upgrade. Requires llm_provider and llm_model from verify_provider_key.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. It discloses file format support, size limit, absolute path requirement, and the optional contextual retrieval workflow. Lacks details on error handling or file disposal, but adequate for typical use.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with bullet points and ordered sections. Every sentence adds value. Slightly lengthy due to detailed pre-flight sequence, but still efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers input requirements, supported formats, size limit, and optional features. Does not detail return structure or error cases, but the tool's output is implied. Overall sufficient for agent invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but description adds value by explaining absolute path, env var fallback for llm_api_key, and the verify_provider_key dependency for contextual_retrieval parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (upload, chunk) and resource (local file) and distinguishes from siblings like 'ingest_file' and 'autorag' by specifying it returns RAG-ready chunks without embedding or vector DB.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit trigger phrases ('chunk this PDF', 'process this document') and a detailed pre-flight sequence for contextual retrieval. Lacks explicit exclusion cases but gives clear usage context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_siteA

Crawl an entire website, chunk all pages, and return structured JSON chunks. Two modes: 'sitemap' (reads sitemap.xml — best for docs/blogs) and 'spider' (follows links — works on any site).

Use this when the user wants chunks from MULTIPLE pages WITHOUT extracting structured fields. For structured field extraction across pages, use extract_crawl.

⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 10 pages. For large sites, warn about credit usage first.

If contextual_retrieval is requested, follow the PRE-FLIGHT sequence:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Ask user to choose a model

  3. Ask about JS rendering

  4. Present Contextual Retrieval as a recommended upgrade

LLM keys can be omitted if set as environment variables.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe root domain or sitemap URL to crawl.
selectorNoOptional CSS selector applied to every crawled page.
js_renderNoUse headless browser to render JS before scraping each page. Ask the user before enabling. Adds surcharge per page.
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode.
max_pagesNoMaximum pages to crawl. Default: 10. Maximum: 200. Always confirm with user for large sites.
crawl_modeNo'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site).sitemap
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. Verify with verify_provider_key first.
exclude_patternNoSkip URLs containing this substring (e.g. '/blog/').
include_patternNoOnly crawl URLs containing this substring (e.g. '/docs/').
contextual_retrievalNoEnable RAG 2.0 contextual enrichment. Present as a recommended upgrade. Requires llm_provider and llm_model from verify_provider_key.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Despite no annotations, the description discloses behavioral traits like two modes, default max_pages, credit usage warning, surcharge for js_render, and pre-flight requirements. Could mention non-destructive nature, but overall strong.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with sections, bullet points, and front-loaded purpose. Slightly lengthy but every part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 11 parameters, no output schema, and no annotations, the description covers the tool's behavior, parameters, and workflow comprehensively. Lacks detail on output format but still strong.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds context beyond the schema, e.g., explaining crawl_mode types, surcharge for js_render, and the pre-flight sequence for contextual_retrieval. Adds value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Crawl an entire website, chunk all pages, and return structured JSON chunks.' It distinguishes two modes and contrasts with extract_crawl for structured field extraction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance is given: use when user wants chunks from multiple pages without structured extraction; use extract_crawl for structured extraction. Includes a pre-flight sequence for contextual_retrieval and warns about confirming max_pages.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_crawlA

Crawl a domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.

Use this when the user wants structured FIELDS from MULTIPLE pages (e.g. extract title + price from every product page on a site).

⚠️ Each page takes 5–15 seconds. Default is 5 pages. For more than 20 pages, warn the user about wait times and credit usage before proceeding.

PRE-FLIGHT REQUIRED — before calling:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Present models to user, ask them to choose one

  3. Ask: 'Is this a JavaScript-heavy site?' → js_render (not available for extract_crawl, note this)

  4. Confirm max_pages with the user

LLM keys can be omitted if set as environment variables.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe root domain to crawl.
schemaYesDict mapping field names to description strings. Example: {"title": "string — the product name", "price": "number — price in USD"}
selectorNoOptional CSS selector applied to every page before extraction.
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode. Advanced models (not mini/flash/haiku) use 30k char context — better for long pages.
max_pagesNoMaximum pages to crawl and extract. Default: 5. Maximum: 50. Always confirm with user before setting above 20.
crawl_modeNo'sitemap': reads sitemap.xml. 'spider': follows links from root URL.sitemap
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerYesLLM provider. One of: 'openai', 'anthropic', 'gemini'. Call verify_provider_key first.
exclude_patternNoSkip URLs containing this substring (e.g. '/blog/').
extract_as_listNoIf true, extracts ALL matching items on each page as a JSON array.
include_patternNoOnly crawl URLs containing this substring (e.g. '/products/').

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses page independence, billing for successful extractions, per-page time estimates, and model context size. Missing rate limits or concurrency details, but the key behaviors are well covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured with a clear purpose sentence, use-case line, warnings, and a numbered pre-flight list. Every sentence adds value; no redundancy. Length is appropriate for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers the main aspects: purpose, usage, pre-flight, timing, billing, parameter guidance. However, it lacks description of the output format (what the extracted data looks like) and does not address error handling for non-page failures. Given no output schema, this is a minor gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds significant context: instructions to not hardcode llm_model, to confirm max_pages, and explanation of crawl_mode options. It also describes pre-flight steps that affect parameter usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb-resource pair 'crawl a domain and extract structured data' and distinguishes from siblings like extract_data (single page) and crawl_site (no structured extraction). It provides a concrete use case example.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use (multiple pages with structured fields) and provides a detailed pre-flight checklist including calling verify_provider_key, confirming max_pages, and noting js_render unavailability. Warnings for large page counts are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_dataA

Scrape a URL and extract structured data matching a user-defined schema using an LLM. Returns a JSON object (or array if extract_as_list=true).

Use this when the user wants specific FIELDS from a page (e.g. product name, price, stock status; article author, date, summary).

PRE-FLIGHT REQUIRED — before calling:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Present models to user, ask them to choose one

  3. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

  4. Present Contextual Retrieval is NOT applicable here (extraction only)

LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe web URL to scrape and extract structured data from.
schemaYesDict mapping field names to description strings. Example: {"title": "string — the product name", "price": "number — price in USD", "in_stock": "boolean — whether in stock"}
selectorNoOptional CSS selector to target a specific section before extraction.
js_renderNoUse headless browser to render JS before extracting. Ask the user before enabling.
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode. Use an advanced model (not mini/flash/haiku) for long-form pages like documentation or legal docs.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerYesLLM provider. One of: 'openai', 'anthropic', 'gemini'. Call verify_provider_key first.
click_selectorNoCSS selector for an element to click after page load (tabs, accordions, load-more). Only used when js_render=true.
extract_as_listNoIf true, extracts ALL matching items on the page as a JSON array. Use for listing pages (product catalogues, article feeds).

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses that the tool uses an LLM for extraction, requires a provider key verification, supports optional headless rendering, and allows API keys via env vars. However, it does not mention rate limits, error handling, timeouts, or any side effects (though likely read-only). The transparency is adequate but could be more comprehensive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a main purpose, usage examples, and a clear 'PRE-FLIGHT REQUIRED' section. It is concise, with each sentence adding value. The use of bullet points in the pre-flight steps enhances readability. Slightly verbose in parts but overall efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (9 parameters, no output schema), the description covers the main purpose, pre-flight process, and parameter details. However, it lacks details on the output format beyond 'returns a JSON object', error handling, or what happens if the schema cannot be matched. For a tool with nested objects and multiple optional parameters, this is a notable gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% description coverage, so the baseline is 3. The description adds significant meaning beyond the schema: it explains the purpose of the 'schema' parameter with an example, advises using advanced models for long pages, clarifies that 'llm_api_key' can be omitted if set as an env var, and explains when to use 'extract_as_list'. This extra context improves understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description starts with a specific verb+resource combination ('scrape a URL and extract structured data') and includes examples of use cases (e.g., product name, price, stock status). It clearly conveys the tool's function of extracting user-defined fields via LLM. However, it does not explicitly differentiate from its sibling tool 'scrape_url', which also deals with URL content, leaving some ambiguity.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear guidance on when to use this tool (when the user wants specific fields from a page) and includes a 'PRE-FLIGHT REQUIRED' section with step-by-step instructions for the agent (call verify_provider_key, present models, ask about js_render). It also clarifies that Contextual Retrieval is not applicable. However, it does not explicitly state when NOT to use it or name alternative tools like 'scrape_url' for raw HTML extraction.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_usage_guideA

Returns the complete guided workflow for using scrapedatshi tools. Call this at the start of any scrapedatshi conversation to understand which tool to use for each task and the required pre-flight sequence.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It states the tool returns a workflow, but does not disclose behavioral traits such as side effects, rate limits, or whether it requires authentication. However, as a read-only guide, its behavior is relatively simple.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences with no wasted words. It front-loads the purpose and provides a clear usage scenario. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema, the description could elaborate on what the guided workflow includes (e.g., tool list, sequence steps). It only says 'complete guided workflow', which is adequate but not fully descriptive for an agent to set expectations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has no parameters, so the input schema provides full coverage. The description does not need to add parameter information. Baseline for 0 parameters is 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns a 'complete guided workflow' for using scrapedatshi tools. It specifies the action ('returns') and the resource ('guided workflow'), and distinguishes itself from sibling action tools by being a meta-guide.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly advises to call it 'at the start of any scrapedatshi conversation' and explains why (to understand tool selection and pre-flight sequence). It does not mention alternatives, but the context is clear enough for an agent to know when to invoke it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ingest_fileA

Full RAG pipeline for local files: upload a file, embed the chunks using your embedding provider, and inject the vectors into your vector database.

Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB

Use this when the user wants to ADD a local document (PDF, markdown, etc.) to their vector DB. This is the file-based equivalent of sync_to_vectordb.

Provide the ABSOLUTE path to the file on the user's local machine.

PRE-FLIGHT REQUIRED — before calling:

  1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

  2. Present models to user, ask them to choose one

  3. Call list_vector_db_providers if user is unsure what config fields are needed

  4. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

  5. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables.

ParametersJSON Schema
NameRequiredDescriptionDefault
overlapNoToken overlap between consecutive chunks. Default: 50.
file_pathYesAbsolute path to the local file to ingest. Supported: .pdf, .md, .txt, .yaml, .yml, .json. Example: 'C:/Users/user/Documents/report.pdf'
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode.
vector_dbYesVector DB provider. Call list_vector_db_providers to see required config fields for each.
chunk_sizeNoTarget token count per chunk. Default: 512.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first.
embedding_modelNoEmbedding model name from verify_provider_key. Do not guess or hardcode.
vector_db_configYesProvider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars.
embedding_api_keyNoAPI key for the embedding provider. Can be omitted if set as env var.
embedding_endpointNoPublic HTTPS endpoint for Ollama only (e.g. from ngrok). Not needed for cloud providers.
embedding_providerYesEmbedding provider. Call verify_provider_key(provider, 'embedding') first to get available models.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model.

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries full burden. Describes the full pipeline, supported formats, max file size, and pre-flight requirements. Mentions cost for contextual retrieval and that keys can be omitted as env vars. Does not detail error handling or edge cases, but provides substantial behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well-structured: purpose first, then supported formats/size, then usage instruction, then numbered pre-flight list. Sentences are informative and not redundant. Slightly long but each part earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers purpose, usage, parameters, pre-flight dependencies. Missing description of return value/output (no output schema provided). Does not explain error scenarios or what happens on failure. Adequate for a complex tool but incomplete in capturing all aspects.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 100% description coverage. Description adds some value by reinforcing that file_path must be absolute, embedding_model should come from verify_provider_key, and pre-flight steps relate to parameters. However, the schema already covers these details, so the description provides marginal additional meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Full RAG pipeline for local files: upload a file, embed the chunks... and inject the vectors into your vector database.' It distinguishes from sibling tools like sync_to_vectordb by calling itself the file-based equivalent, and from chunk_file by implying it does the full pipeline.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use: 'Use this when the user wants to ADD a local document... to their vector DB.' Provides a detailed pre-flight checklist. Does not explicitly say when not to use or list alternatives beyond the sibling, but context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_embedding_providersA

Returns all supported embedding providers with labels and notes. Call this to help the user choose an embedding provider before sync_to_vectordb. After the user chooses, call verify_provider_key to get the live model list.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description explains it returns labels and notes, which is sufficient for a read-only listing tool. No annotations are present, so the description carries the burden; it is brief but adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences, front-loaded with the primary action, no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and no parameters, the description covers the return values and workflow steps. It is reasonably complete for a helper tool, though could mention if the list is static.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist, so the description does not need to add parameter info. The schema coverage is 100%, meeting the baseline for zero parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns all supported embedding providers with labels and notes. It distinguishes itself from siblings by specifying its role before sync_to_vectordb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises when to call (before sync_to_vectordb) and the next step (call verify_provider_key after user choice), providing a clear workflow.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_vector_db_providersA

Returns all supported vector database providers with required config fields, optional fields, and setup notes. Call this before sync_to_vectordb to help the user understand what vector_db_config fields they need to provide.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; description doesn't add behavioral traits beyond purpose. For a read-only list, minimal but adequate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences front-loading the purpose and usage, with no unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with no parameters and no output schema, the description adequately covers what the tool returns and why it should be called.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters exist; schema coverage is 100%. Description adds value by explaining what the output contains (config fields, optional fields, setup notes).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states it returns supported vector database providers with config fields and setup notes, and distinguishes itself from sibling tools like list_embedding_providers.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Clearly says to call this before sync_to_vectordb to understand required fields, providing explicit when-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

scrape_urlA

Scrape a single web URL, chunk its content into RAG-ready text segments, and return the structured chunks. No embedding or vector DB required — this is the fastest and cheapest operation.

Use this when the user wants to read, summarize, or process the content of a specific web page WITHOUT extracting structured fields.

If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:

  1. Call verify_provider_key(provider, 'llm') → get live model list

  2. Ask user to choose a model from the list

  3. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

  4. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe web URL to scrape and chunk.
overlapNoToken overlap between consecutive chunks. Default: 50.
selectorNoOptional CSS selector to target a specific element (e.g. 'article', '.content', 'main').
js_renderNoUse headless Chromium to render JavaScript before scraping. Required for SPAs and JS-heavy pages. Ask the user before enabling. Adds a small surcharge.
llm_modelNoLLM model name. MUST be chosen from the list returned by verify_provider_key — do not guess or hardcode.
chunk_sizeNoTarget token count per chunk. Default: 512. Range: 64–4096.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. One of: 'openai', 'anthropic', 'gemini'. Verify with verify_provider_key first.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment. An LLM generates a unique context string for each chunk, boosting retrieval accuracy by 35–50%. Present this as a recommended upgrade. Requires llm_provider and llm_model (from verify_provider_key).

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses chunking behavior, cost (surcharge for js_render, ~$0.001/chunk for contextual retrieval), prerequisite steps (verify_provider_key), and important flags (js_render for SPAs). However, it omits output format details, error handling, rate limits, or caching behavior, which would increase transparency for a tool with 9 parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise in the first paragraph but includes a detailed multi-step pre-flight sequence that, while informative, makes it longer than necessary. Every sentence serves a purpose, but the pre-flight instructions could be more succinct. Overall structured logically: purpose, usage, special case.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has 9 parameters, no output schema, and moderate complexity. The description covers the main use case and the contextual retrieval flow. However, it does not describe the return format (e.g., array of chunks with text, metadata) or error handling. Without output schema, the agent needs more detail on what to expect. Given the richness of the description for the main flow, it's adequate but not complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description adds value by explaining when to use js_render (JavaScript-heavy pages, SPAs), that llm_model must come from verify_provider_key, and that llm_api_key can be an env var. The pre-flight sequence for contextual_retrieval adds operational context beyond schema. However, some schema descriptions already suffice, limiting extra contribution.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool scrapes a single URL, chunks it into RAG-ready segments, and returns structured chunks. It distinguishes from sibling tools like extract_data (which extracts structured fields) and crawl_site (which does multiple pages). The verb 'scrape' and resource 'single web URL' are specific and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when to use: 'when the user wants to read, summarize, or process content of a specific web page WITHOUT extracting structured fields.' Provides a detailed pre-flight sequence for contextual_retrieval. Does not explicitly list when not to use, but the positive criteria imply exclusion of structured extraction. Sibling names provide context, but description could be more explicit about alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

sync_to_vectordbA

Full RAG pipeline: scrape a URL, embed the chunks using your embedding provider, and inject the vectors into your vector database — all in one call.

Use this when the user wants to ADD web content to their vector DB for later retrieval. The user brings their own embedding provider and vector DB.

PRE-FLIGHT REQUIRED — before calling:

  1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

  2. Present models to user, ask them to choose one

  3. Call list_vector_db_providers if user is unsure what config fields are needed

  4. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

  5. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra. If yes, I'll also need your LLM provider and model.'

  6. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables (OPENAI_API_KEY, PINECONE_API_KEY, etc.).

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe web URL to scrape, embed, and inject into the vector DB.
overlapNoToken overlap between consecutive chunks. Default: 50.
selectorNoOptional CSS selector to target a specific page section.
js_renderNoUse headless browser to render JS before scraping. Ask the user before enabling.
llm_modelNoLLM model name from verify_provider_key. Do not guess or hardcode.
vector_dbYesVector DB provider. Call list_vector_db_providers to see required config fields for each.
chunk_sizeNoTarget token count per chunk. Default: 512.
llm_api_keyNoAPI key for the LLM provider. Can be omitted if set as env var.
llm_providerNoLLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first.
embedding_modelNoEmbedding model name from verify_provider_key. Do not guess or hardcode.
vector_db_configYesProvider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. Examples: pinecone: {"index_host": "https://my-index.svc.pinecone.io"} | qdrant: {"url": "https://cluster.qdrant.io", "collection_name": "docs"} | supabase: {"connection_string": "postgresql://...", "table_name": "documents"} | chroma: {"collection_name": "docs"}
embedding_api_keyNoAPI key for the embedding provider. Can be omitted if set as env var.
embedding_endpointNoPublic HTTPS endpoint for Ollama only (e.g. from ngrok). Not needed for cloud providers.
embedding_providerYesEmbedding provider. Call verify_provider_key(provider, 'embedding') first to get available models.
contextual_retrievalNoEnable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model.

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It discloses the composite operation, pre-flight requirements, headless browser for js_render, and extra cost for contextual_retrieval. It does not cover rate limits, error handling, or idempotency, but provides sufficient transparency for a complex tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is lengthy but well-structured with numbered steps and bullet points. It front-loads the purpose and systematically guides the user. Slightly verbose but efficient for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given high complexity (15 params, nested objects, no output schema), the description covers the workflow and pre-flight thoroughly. It lacks explicit return value description, but the process is clear enough for an agent to proceed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, baseline 3. The description adds value by explaining workflow context (e.g., embedding_endpoint only for Ollama, env var fallbacks, and pre-flight steps). It does not repeat schema but enhances usability beyond schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool is a full RAG pipeline that scrapes, embeds, and injects into a vector DB. It explicitly says 'Use this when the user wants to ADD web content to their vector DB for later retrieval,' distinguishing it from sibling tools like chunk_file or ingest_file that handle only parts of the pipeline.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage context ('ADD web content') and a detailed pre-flight checklist with steps. However, it does not explicitly mention when NOT to use this tool or compare directly with siblings like autorag, leaving some ambiguity about alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

verify_provider_keyA

Verify an LLM or embedding API key and return the live list of models available for that key. Call this BEFORE any operation that requires an LLM or embedding provider — never assume or hardcode model names.

Returns: key validity, list of available model names (live from the provider's API), and an error message if the key is invalid.

Supported LLM providers: openai, anthropic, gemini Supported embedding providers: openai, cohere, gemini, mistral, voyage

The API key can be omitted if the corresponding env var is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, COHERE_API_KEY, MISTRAL_API_KEY, VOYAGE_API_KEY).

ParametersJSON Schema
NameRequiredDescriptionDefault
api_keyNoThe API key to verify. Can be omitted if the corresponding env var is set.
providerYesProvider to verify. LLM: 'openai', 'anthropic', 'gemini'. Embedding: 'openai', 'cohere', 'gemini', 'mistral', 'voyage'.
provider_typeYes'llm' for text generation models (used in extract_data, extract_crawl, contextual_retrieval). 'embedding' for vector embedding models (used in sync_to_vectordb).

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses return values (key validity, model list, error message) and mentions that API keys can be omitted. It does not mention side effects, rate limits, or idempotency, but for a verification tool, the behavioral traits are adequately covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, comprising three clear sentences that front-load the primary purpose. It states the action, return values, and supported providers without any extraneous information. Every sentence adds value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given there is no output schema, the description fully describes return values (key validity, model list, error). It lists all supported providers and explains how the key can be omitted. The tool is straightforward, and the description covers everything needed for an agent to use it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds value by explaining that 'provider_type' indicates whether models are used for LLM or embedding, with examples of where they are applied (extract_data, sync_to_vectordb). It also elaborates on the 'api_key' parameter's optionality with env var names, going beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: verifying an API key and returning live available models. It uses specific verbs ('verify', 'return') and identifies the resource (LLM/embedding API key). This clearly distinguishes it from sibling tools like 'extract_data' or 'sync_to_vectordb' which perform different operations.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly states when to use the tool: 'Call this BEFORE any operation that requires an LLM or embedding provider' and advises against hardcoding model names. It lists supported providers and notes that the API key can be omitted if env vars are set. However, it does not explicitly discuss scenarios where the tool should not be used or contrast it with alternatives, though the context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 12 tool updatesv0.2.1
    • First observedautorag
    • First observedchunk_file
    • First observedcrawl_site
    • First observedextract_crawl
    • First observedextract_data
    • First observedget_usage_guide
    • First observedingest_file
    • First observedlist_embedding_providers
    • First observedlist_vector_db_providers
    • First observedscrape_url
    • First observedsync_to_vectordb
    • First observedverify_provider_key

TDQS

A4.2/5.0
Disambiguation4/5

Tools are mostly distinct, with clear purposes like scrape_url vs crawl_site vs extract_data. However, some overlap exists: autorag combines crawl_site and sync_to_vectordb, and ingest_file is similar to the file-based equivalent. Descriptions help disambiguate.

Naming Consistency4/5

Naming follows a consistent verb_noun pattern (scrape_url, list_embedding_providers, verify_provider_key). Minor deviations: autorag is a brand name, get_usage_guide uses 'get' instead of a verb like 'show', and sync_to_vectordb has a preposition. Overall mostly consistent.

Tool Count5/5

12 tools is an appropriate number for a web scraping and RAG pipeline server. Each tool covers a distinct step or combination of steps, and the count is neither too few nor too many for the scope.

Completeness5/5

The tool set covers the full workflow: provider discovery, key verification, scraping (single/multi page, structured extraction), file processing, and full RAG pipelines (chunk, embed, inject). Includes a usage guide. No obvious gaps for the stated purpose.

Maintenance

ActivitySlowing
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/scrapedatshi/scrapedatshi-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server