Verify an LLM or embedding API key and return the live list of models available for that key. Call this BEFORE any operation that requires an LLM or embedding provider — never assume or hardcode model names.

Returns: key validity, list of available model names (live from the provider's API), and an error message if the key is invalid.

Supported LLM providers: openai, anthropic, gemini Supported embedding providers: openai, cohere, gemini, mistral, voyage

The API key can be omitted if the corresponding env var is set (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, COHERE_API_KEY, MISTRAL_API_KEY, VOYAGE_API_KEY).

Returns the complete guided workflow for using scrapedatshi tools. Call this at the start of any scrapedatshi conversation to understand which tool to use for each task and the required pre-flight sequence.

Scrape a single web URL, chunk its content into RAG-ready text segments, and return the structured chunks. No embedding or vector DB required — this is the fastest and cheapest operation.

Use this when the user wants to read, summarize, or process the content of a specific web page WITHOUT extracting structured fields.

If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:

1. Call verify_provider_key(provider, 'llm') → get live model list

2. Ask user to choose a model from the list

3. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

4. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.

Crawl an entire website, chunk all pages, and return structured JSON chunks. Two modes: 'sitemap' (reads sitemap.xml — best for docs/blogs) and 'spider' (follows links — works on any site).

Use this when the user wants chunks from MULTIPLE pages WITHOUT extracting structured fields. For structured field extraction across pages, use extract_crawl.

⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 10 pages. For large sites, warn about credit usage first.

If contextual_retrieval is requested, follow the PRE-FLIGHT sequence:

1. Call verify_provider_key(provider, 'llm') → get live model list

2. Ask user to choose a model

3. Ask about JS rendering

4. Present Contextual Retrieval as a recommended upgrade

LLM keys can be omitted if set as environment variables.

Scrape a URL and extract structured data matching a user-defined schema using an LLM. Returns a JSON object (or array if extract_as_list=true).

Use this when the user wants specific FIELDS from a page (e.g. product name, price, stock status; article author, date, summary).

PRE-FLIGHT REQUIRED — before calling:

1. Call verify_provider_key(provider, 'llm') → get live model list

2. Present models to user, ask them to choose one

3. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

4. Present Contextual Retrieval is NOT applicable here (extraction only)

LLM keys can be omitted if OPENAI_API_KEY, ANTHROPIC_API_KEY, or GEMINI_API_KEY is set in the MCP environment config.

Crawl a domain and extract structured data from every page using your LLM. Each page is processed independently — failed pages return an error without aborting the batch. Only successfully extracted pages are billed.

Use this when the user wants structured FIELDS from MULTIPLE pages (e.g. extract title + price from every product page on a site).

⚠️ Each page takes 5–15 seconds. Default is 5 pages. For more than 20 pages, warn the user about wait times and credit usage before proceeding.

PRE-FLIGHT REQUIRED — before calling:

1. Call verify_provider_key(provider, 'llm') → get live model list

2. Present models to user, ask them to choose one

3. Ask: 'Is this a JavaScript-heavy site?' → js_render (not available for extract_crawl, note this)

4. Confirm max_pages with the user

LLM keys can be omitted if set as environment variables.

Full RAG pipeline: scrape a URL, embed the chunks using your embedding provider, and inject the vectors into your vector database — all in one call.

Use this when the user wants to ADD web content to their vector DB for later retrieval. The user brings their own embedding provider and vector DB.

PRE-FLIGHT REQUIRED — before calling:

1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

2. Present models to user, ask them to choose one

3. Call list_vector_db_providers if user is unsure what config fields are needed

4. Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render

5. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra. If yes, I'll also need your LLM provider and model.'

6. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables (OPENAI_API_KEY, PINECONE_API_KEY, etc.).

Upload a local file, chunk its content into RAG-ready text segments, and return the structured chunks as JSON. No embedding or vector DB required.

Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB

Use this when the user says 'chunk this PDF', 'process this document', 'read this file', or wants to extract text from a local file.

Provide the ABSOLUTE path to the file on the user's local machine (e.g. 'C:/Users/user/Documents/report.pdf' or '/home/user/docs/manual.pdf').

If contextual_retrieval=true is requested, follow the PRE-FLIGHT sequence:

1. Call verify_provider_key(provider, 'llm') → get live model list

2. Ask user to choose a model

3. Present Contextual Retrieval as a recommended upgrade

LLM keys can be omitted if set as environment variables.

Full RAG pipeline for local files: upload a file, embed the chunks using your embedding provider, and inject the vectors into your vector database.

Supported file formats: .pdf, .md, .txt, .yaml, .yml, .json Maximum file size: 50 MB

Use this when the user wants to ADD a local document (PDF, markdown, etc.) to their vector DB. This is the file-based equivalent of sync_to_vectordb.

Provide the ABSOLUTE path to the file on the user's local machine.

PRE-FLIGHT REQUIRED — before calling:

1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

2. Present models to user, ask them to choose one

3. Call list_vector_db_providers if user is unsure what config fields are needed

4. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

5. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables.

Full AutoRAG pipeline: crawl an entire domain, chunk every page, embed all chunks, and inject into your vector database — all in a single call.

Use this when the user wants to bulk-ingest an entire website into their vector DB. This combines crawl_site + sync_to_vectordb into one operation.

⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 5 pages. Each page is fetched, chunked, embedded, and injected. For large sites, warn about credit usage and wait times first.

PRE-FLIGHT REQUIRED — before calling:

1. Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list

2. Present models to user, ask them to choose one

3. Call list_vector_db_providers if user is unsure what config fields are needed

4. Confirm max_pages with the user

5. Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'

6. If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too

Keys can be omitted if set as environment variables.

Returns all supported embedding providers with labels and notes. Call this to help the user choose an embedding provider before sync_to_vectordb. After the user chooses, call verify_provider_key to get the live model list.

Returns all supported vector database providers with required config fields, optional fields, and setup notes. Call this before sync_to_vectordb to help the user understand what vector_db_config fields they need to provide.