crawl_website_to_rag_markdown
Crawl websites or sitemaps into clean, LLM-ready Markdown and token-bounded RAG chunks without headless browser overhead, preparing site content for retrieval-augmented generation pipelines.
Instructions
Crawl any website or sitemap into clean LLM-ready Markdown and token-bounded RAG chunks without headless browser overhead.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"startUrls": "<startUrls>"
},
"name": "crawl_website_to_rag_markdown"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| startUrls | Yes | List of URLs or bare domains to crawl. | |
| useSitemap | No | Discover internal pages via /sitemap.xml. | |
| chunkSizeWords | No | Target word count per RAG chunk. | |
| maxPagesPerSite | No | Max pages to extract per starting domain. | |
| includeRagChunks | No | Split Markdown into token-bounded RAG chunks. |