sync_to_vectordb
Scrape a web page, generate embeddings, and sync the vectors to your vector database for retrieval. Supports multiple embedding and vector DB providers.
Instructions
Full RAG pipeline: scrape a URL, embed the chunks using your embedding provider, and inject the vectors into your vector database — all in one call.
Use this when the user wants to ADD web content to their vector DB for later retrieval. The user brings their own embedding provider and vector DB.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list
Present models to user, ask them to choose one
Call list_vector_db_providers if user is unsure what config fields are needed
Ask: 'Is this a JavaScript-heavy page or SPA?' → js_render
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra. If yes, I'll also need your LLM provider and model.'
If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too
Keys can be omitted if set as environment variables (OPENAI_API_KEY, PINECONE_API_KEY, etc.).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The web URL to scrape, embed, and inject into the vector DB. | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector to target a specific page section. | |
| js_render | No | Use headless browser to render JS before scraping. Ask the user before enabling. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db | Yes | Vector DB provider. Call list_vector_db_providers to see required config fields for each. | |
| chunk_size | No | Target token count per chunk. Default: 512. | |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first. | |
| embedding_model | No | Embedding model name from verify_provider_key. Do not guess or hardcode. | |
| vector_db_config | Yes | Provider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. Examples: pinecone: {"index_host": "https://my-index.svc.pinecone.io"} | qdrant: {"url": "https://cluster.qdrant.io", "collection_name": "docs"} | supabase: {"connection_string": "postgresql://...", "table_name": "documents"} | chroma: {"collection_name": "docs"} | |
| embedding_api_key | No | API key for the embedding provider. Can be omitted if set as env var. | |
| embedding_endpoint | No | Public HTTPS endpoint for Ollama only (e.g. from ngrok). Not needed for cloud providers. | |
| embedding_provider | Yes | Embedding provider. Call verify_provider_key(provider, 'embedding') first to get available models. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model. |