autorag
Crawl an entire domain, chunk every page, embed all chunks, and inject into your vector database in a single call.
Instructions
Full AutoRAG pipeline: crawl an entire domain, chunk every page, embed all chunks, and inject into your vector database — all in a single call.
Use this when the user wants to bulk-ingest an entire website into their vector DB. This combines crawl_site + sync_to_vectordb into one operation.
⚠️ ALWAYS confirm the max_pages limit with the user before calling. Default is 5 pages. Each page is fetched, chunked, embedded, and injected. For large sites, warn about credit usage and wait times first.
PRE-FLIGHT REQUIRED — before calling:
Call verify_provider_key(embedding_provider, 'embedding') → get live embedding model list
Present models to user, ask them to choose one
Call list_vector_db_providers if user is unsure what config fields are needed
Confirm max_pages with the user
Present Contextual Retrieval as a recommended upgrade: 'Would you like Contextual Retrieval (RAG 2.0)? It enriches each chunk with LLM-generated context before embedding, improving retrieval accuracy by 35–50%. Costs ~$0.001/chunk extra.'
If contextual_retrieval=yes: call verify_provider_key(llm_provider, 'llm') too
Keys can be omitted if set as environment variables.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The root domain to crawl (e.g. 'https://docs.example.com'). | |
| overlap | No | Token overlap between consecutive chunks. Default: 50. | |
| selector | No | Optional CSS selector applied to every page before chunking. | |
| llm_model | No | LLM model name from verify_provider_key. Do not guess or hardcode. | |
| max_pages | No | Maximum pages to crawl and inject. Default: 5. Maximum: 200. Always confirm with user for large sites. | |
| vector_db | Yes | Vector DB provider. Call list_vector_db_providers to see required config fields for each. | |
| chunk_size | No | Target token count per chunk. Default: 512. | |
| crawl_mode | No | 'sitemap': reads sitemap.xml (best for docs/blogs). 'spider': follows links from root URL (works on any site). | sitemap |
| llm_api_key | No | API key for the LLM provider. Can be omitted if set as env var. | |
| llm_provider | No | LLM provider for contextual retrieval. Verify with verify_provider_key(provider, 'llm') first. | |
| embedding_model | No | Embedding model name from verify_provider_key. Do not guess or hardcode. | |
| exclude_pattern | No | Skip URLs containing this substring (e.g. '/blog/'). | |
| include_pattern | No | Only crawl URLs containing this substring (e.g. '/docs/'). | |
| vector_db_config | Yes | Provider-specific config. Call list_vector_db_providers for required fields. API keys within this config can be omitted if set as env vars. | |
| embedding_api_key | No | API key for the embedding provider. Can be omitted if set as env var. | |
| embedding_provider | Yes | Embedding provider. Call verify_provider_key(provider, 'embedding') first to get available models. | |
| contextual_retrieval | No | Enable RAG 2.0 contextual enrichment before embedding. Present as a recommended upgrade. Requires llm_provider and llm_model. |