Skip to main content
Glama

chunktuner

PyPI version Python versions License: MIT CI Docs

用于 RAG 流水线的自动分块调优器和 MCP 服务器。

提供您的文档,它会尝试多种分块策略,衡量哪种策略能让 AI 最准确地回答问题,并告诉您胜出者。

chunktuner 项目流程:文档经过策略、评估,最终得到推荐配置


功能概述

在构建 RAG 流水线时,如何将文档拆分为块直接影响检索质量。chunktuner 自动化了为您的特定语料库、嵌入模型和用例寻找最佳分块策略的过程。

它会对固定标记窗口、递归字符拆分、语义拆分、PDF 结构化分块和基于 AST 的代码分块等策略进行基准测试,然后根据实际检索指标(标记召回率、MRR、NDCG)和可选的生成指标(RAGAS 忠实度、答案相关性)对每种策略进行评分。


Related MCP server: golden-dataset-mcp

接口

  • Python 库 — 以编程方式集成到您的流水线中

  • CLI (chunk-tune) — 从终端进行人工驱动的调优

  • MCP 服务器 — 直接从 Claude Desktop 或任何 MCP 主机使用


快速入门

# Install
uv tool install chunktuner

# Initialize workspace
chunk-tune init --provider openai

# See cost estimate before running anything
chunk-tune estimate ./my_docs --use-case rag_qa

# Get a recommendation
chunk-tune recommend ./my_docs --use-case rag_qa

Python API:

from pathlib import Path
from chunktuner import FileIngestor, LiteLLMEmbeddingFunction, AutoTuner
from chunktuner import default_registry, Evaluator, ScoreCalculator

docs = FileIngestor().ingest_dir(Path("./my_docs"))
embedding_fn = LiteLLMEmbeddingFunction("text-embedding-3-small")
tuner = AutoTuner(
    strategies=default_registry,
    evaluator=Evaluator(embedding_fn),
    scorer=ScoreCalculator(use_case="rag_qa"),
)
result = tuner.recommend(docs, use_case="rag_qa")
print(result.best.config)

支持的策略

策略

最适合

fixed_tokens

基准;统一标记窗口

recursive_character

一般散文和文档

semantic

主题性强的文章

markdown_semantic

结构化 Markdown 文档

pdf_structural

带有布局区域和表格的 PDF

structural_semantic

混合布局和文本的 PDF/DOCX

late_chunking

带有密集交叉引用的长文档

agentic

高价值叙事文档

code_ast

代码仓库(Python, JavaScript)

code_window

代码基准(滑动窗口)


MCP 服务器 (Claude Desktop)

Python FastMCP (chunk-tune-mcp, stdio)。无需 Node.js 构建。请参阅 docs/mcp_setup.md

添加到您的 .mcp.json 中:

{
  "mcpServers": {
    "chunktuner": {
      "command": "uvx",
      "args": ["--from", "chunktuner[mcp]", "chunk-tune-mcp"],
      "env": {
        "CHUNK_TUNER_BASE_DIR": "/path/to/your/corpus"
      }
    }
  }
}

可用工具:list_strategies, preview_chunks, evaluate_chunking, recommend_config


CLI 参考

chunk-tune init       Bootstrap workspace config
chunk-tune analyze    Quick structural scan (no API cost)
chunk-tune estimate   Dry-run cost/token estimate
chunk-tune evaluate   Full evaluation across strategies
chunk-tune recommend  Evaluation + best config recommendation
chunk-tune compare    Side-by-side comparison of specific strategies
chunk-tune preview    Inspect how a strategy splits a document
chunk-tune cache      Manage embedding and chunk cache

安装选项

uv add chunktuner                    # library
uv tool install chunktuner           # global CLI
uvx chunktuner                       # ephemeral, no install

# With optional extras
uv add "chunktuner[docling]"         # PDF/DOCX support
uv add "chunktuner[ragas]"           # generation metrics
uv add "chunktuner[semantic]"        # semantic chunking
uv add "chunktuner[code]"            # AST code chunking
uv add "chunktuner[all]"             # everything

贡献

请参阅 CONTRIBUTING.md


👨🏻💻 作者

Shantanu Deshmukh

全栈开发人员,拥有构建端到端 AI 应用的经验。

Linkedin / Twitter / AngelList

Install Server
A
license - permissive license
B
quality
B
maintenance

Maintenance

Maintainers
Response time
Release cycle
1Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Exposes queryable GPU inference benchmark data (quantization, throughput, VRAM, concurrent users) as tools for LLM clients.
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Enables version-controlled golden dataset management and semantic evaluation for RAG/LLM pipelines using TF-IDF cosine similarity, without requiring an LLM API key.
    9
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Evaluates RAG outputs on faithfulness, answer relevancy, and context precision using an LLM-as-a-Judge backend. Exposes tools for running evaluations, scoring individual samples, and checking thresholds, enabling CI gating and on-demand assessment via MCP.
    MIT

View all related MCP servers

Related MCP Connectors

  • Multi-LLM entity enrichment: schemas, single/batch enrichment, fusion, model benchmarks.

  • Bounded tools for rendering, extraction, RAG, enrichment, local discovery and review analysis.

  • Machine-readable utilities and datasets for AI agents.

View all MCP Connectors

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/shantanu-deshmukh/chunktuner'

If you have feedback or need assistance with the MCP directory API, please join our Discord server