ChunkTuner
chunktuner
用于 RAG 流水线的自动分块调优器和 MCP 服务器。
提供您的文档,它会尝试多种分块策略,衡量哪种策略能让 AI 最准确地回答问题,并告诉您胜出者。
功能概述
在构建 RAG 流水线时,如何将文档拆分为块直接影响检索质量。chunktuner 自动化了为您的特定语料库、嵌入模型和用例寻找最佳分块策略的过程。
它会对固定标记窗口、递归字符拆分、语义拆分、PDF 结构化分块和基于 AST 的代码分块等策略进行基准测试,然后根据实际检索指标(标记召回率、MRR、NDCG)和可选的生成指标(RAGAS 忠实度、答案相关性)对每种策略进行评分。
Related MCP server: golden-dataset-mcp
接口
Python 库 — 以编程方式集成到您的流水线中
CLI (
chunk-tune) — 从终端进行人工驱动的调优MCP 服务器 — 直接从 Claude Desktop 或任何 MCP 主机使用
快速入门
# Install
uv tool install chunktuner
# Initialize workspace
chunk-tune init --provider openai
# See cost estimate before running anything
chunk-tune estimate ./my_docs --use-case rag_qa
# Get a recommendation
chunk-tune recommend ./my_docs --use-case rag_qaPython API:
from pathlib import Path
from chunktuner import FileIngestor, LiteLLMEmbeddingFunction, AutoTuner
from chunktuner import default_registry, Evaluator, ScoreCalculator
docs = FileIngestor().ingest_dir(Path("./my_docs"))
embedding_fn = LiteLLMEmbeddingFunction("text-embedding-3-small")
tuner = AutoTuner(
strategies=default_registry,
evaluator=Evaluator(embedding_fn),
scorer=ScoreCalculator(use_case="rag_qa"),
)
result = tuner.recommend(docs, use_case="rag_qa")
print(result.best.config)支持的策略
策略 | 最适合 |
| 基准;统一标记窗口 |
| 一般散文和文档 |
| 主题性强的文章 |
| 结构化 Markdown 文档 |
| 带有布局区域和表格的 PDF |
| 混合布局和文本的 PDF/DOCX |
| 带有密集交叉引用的长文档 |
| 高价值叙事文档 |
| 代码仓库(Python, JavaScript) |
| 代码基准(滑动窗口) |
MCP 服务器 (Claude Desktop)
Python FastMCP (chunk-tune-mcp, stdio)。无需 Node.js 构建。请参阅 docs/mcp_setup.md。
添加到您的 .mcp.json 中:
{
"mcpServers": {
"chunktuner": {
"command": "uvx",
"args": ["--from", "chunktuner[mcp]", "chunk-tune-mcp"],
"env": {
"CHUNK_TUNER_BASE_DIR": "/path/to/your/corpus"
}
}
}
}可用工具:list_strategies, preview_chunks, evaluate_chunking, recommend_config。
CLI 参考
chunk-tune init Bootstrap workspace config
chunk-tune analyze Quick structural scan (no API cost)
chunk-tune estimate Dry-run cost/token estimate
chunk-tune evaluate Full evaluation across strategies
chunk-tune recommend Evaluation + best config recommendation
chunk-tune compare Side-by-side comparison of specific strategies
chunk-tune preview Inspect how a strategy splits a document
chunk-tune cache Manage embedding and chunk cache安装选项
uv add chunktuner # library
uv tool install chunktuner # global CLI
uvx chunktuner # ephemeral, no install
# With optional extras
uv add "chunktuner[docling]" # PDF/DOCX support
uv add "chunktuner[ragas]" # generation metrics
uv add "chunktuner[semantic]" # semantic chunking
uv add "chunktuner[code]" # AST code chunking
uv add "chunktuner[all]" # everything贡献
请参阅 CONTRIBUTING.md。
👨🏻💻 作者
全栈开发人员,拥有构建端到端 AI 应用的经验。
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceExposes queryable GPU inference benchmark data (quantization, throughput, VRAM, concurrent users) as tools for LLM clients.MIT
- AlicenseAqualityBmaintenanceEnables version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines using TF-IDF cosine similarity, without requiring an LLM API key.9MIT
- AlicenseNot gradedqualityDmaintenanceEnables LLMs to search and retrieve information from large technical documentation (OpenAPI specs, markdown) via intelligent chunking and semantic search.MIT
- AlicenseNot gradedqualityCmaintenanceEvaluates RAG outputs on faithfulness, answer relevancy, and context precision using an LLM-as-a-Judge backend. Exposes tools for running evaluations, scoring individual samples, and checking thresholds, enabling CI gating and on-demand assessment via MCP.MIT
Related MCP Connectors
Multi-LLM entity enrichment: schemas, single/batch enrichment, fusion, model benchmarks.
Bounded tools for rendering, extraction, RAG, enrichment, local discovery and review analysis.
Machine-readable utilities and datasets for AI agents.
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/shantanu-deshmukh/chunktuner'
If you have feedback or need assistance with the MCP directory API, please join our Discord server