한국어 | English
Промежуточное ПО на основе DOM AST для веб-конвейеров LLM — убирает шум вёрстки (навигация, скрипты, сайдбары) и передаёт исходный текст напрямую. Добавьте запрос, чтобы отфильтровать релевантные разделы с помощью BM25.
Когда LLM использует встроенный WebFetch, модель меньшего размера предварительно обрабатывает HTML и возвращает суммаризированный результат — добавляя задержку, затраты и интерпретацию, о которых вы не просили. DomPruner полностью пропускает этот этап: парсинг DOM AST убирает шум и передаёт исходный контент напрямую модели.
Вызов | Поведение |
| Убирает шум вёрстки → возвращает полное извлечённое содержимое |
| Убирает шум вёрстки → BM25 фильтрует релевантные разделы (если совпадений нет, возвращает полное содержимое) |
> [DomPruner] docs.python.org
> | Raw HTML | 44,316 tokens |
> | DomPruner | 1,328 tokens |
> | Reduction | 97.0% |
> Fetch: 194ms · Parse: 11.2msВ среднем на 93,5% меньше контекстных токенов, чем у WebFetch. На 45% быстрее в целом. → Полный бенчмарк
Установка не требуется, API-ключ не нужен:
npx -y dompruner-mcp{
"mcpServers": {
"dompruner": {
"type": "stdio",
"command": "npx",
"args": ["-y", "dompruner-mcp"]
}
}
}Добавьте в .mcp.json в корне проекта или в ~/.claude/.mcp.json для глобального использования. Выполните /mcp, чтобы проверить.
Отредактируйте ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) или %APPDATA%\Claude\claude_desktop_config.json (Windows):
{
"mcpServers": {
"dompruner": {
"command": "npx",
"args": ["-y", "dompruner-mcp"]
}
}
}{
"mcpServers": {
"dompruner": {
"type": "stdio",
"command": "npx",
"args": ["-y", "dompruner-mcp"]
}
}
}Для клиентов, поддерживающих HTTP-транспорт — не требуется установка Node.js, всегда работает последняя версия:
{
"mcpServers": {
"dompruner": {
"url": "https://dompruner-mcp.vercel.app/api/mcp"
}
}
}langchain-mcp-adapters автоматически оборачивает любой MCP stdio-сервер в инструменты LangChain:
from langchain_mcp_adapters.client import MultiServerMCPClient
client = MultiServerMCPClient({
"dompruner": {
"command": "npx",
"args": ["-y", "dompruner-mcp"],
"transport": "stdio",
}
})
tools = await client.get_tools()Related MCP server: Scrapi MCP Server
Описание инструмента DomPruner уже сообщает клиентам, что следует предпочитать dompruner_fetch вместо WebFetch. Если ваш клиент всё равно переключается на WebFetch, добавьте это в его файл инструкций:
When retrieving a URL, always use dompruner_fetch instead of WebFetch.
- URL known → dompruner_fetch(url, query?)
- URL unknown → search for the URL first, then dompruner_fetch(url)Клиент | Файл инструкций |
Claude Code |
|
Cursor |
|
Windsurf |
|
Cline |
|
GitHub Copilot |
|
Инструмент | Описание |
| Получает URL → Markdown, очищенный с помощью DOM. Опциональный |
| Получает все страницы из sitemap.xml → по одному очищенному Document на страницу. |
| Отчёт о сокращении токенов для URL без полного содержимого. |
→ Полный справочник инструментов
Метрика | WebFetch | DomPruner |
Среднее количество контекстных токенов | ~15 735 | ~1 019 (на 93,5% меньше) |
Качество ответов (10 запросов) | 9 / 10 | 8 / 10 |
Среднее время ответа | 5 811 ms | 3 168 ms (на 45% быстрее) |
Точность содержимого | Суммаризировано малой моделью | Исходный текст сохранён |
Дополнительный API-ключ / инфраструктура | Нет | Нет |
→ Полный бенчмарк · Архитектура
dompruner-py — Python-порт. DomPrunerLoader, DomPrunerSitemapLoader, DomPrunerFetchTool для LangChain. pip install dompruner.
Интеграции LangChain — dompruner-py указан как сторонний веб-загрузчик.
MIT
Returns a token-reduction analysis report for a URL. Shows render type, original vs refined token counts, and top Semantic Anchors.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to analyze |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
There are no annotations, so the description carries full responsibility for disclosing side effects and operational behavior. It mentions the output but does not disclose that the tool likely fetches the URL (a network request), potential rate limits, failure modes for invalid URLs, or whether it is strictly read-only. Without this information, the agent cannot fully assess the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the core action in the first sentence, and every word adds value. It efficiently communicates the report contents without waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple (one parameter, no output schema, no annotations), and the description provides a clear overview of the return type and contents. It does not fully specify the report structure or semantics of terms like 'Semantic Anchors', but for a single-URL analysis tool, the description is sufficiently complete to guide use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (the single 'url' parameter is described as 'URL to analyze'), and the description also refers to 'URL', but it adds no additional meaning such as URL format expectations, accessibility requirements, or length limits. The baseline of 3 applies because the schema already documents the parameter fully.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns a 'token-reduction analysis report for a URL', which is a specific verb+resource+target. It also lists the report contents (render type, original vs refined token counts, top Semantic Anchors), making the purpose unmistakable and distinguishing it from the sibling dompruner_fetch, which presumably fetches raw data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide any guidance on when to use this tool versus alternatives. It merely states what it returns, with no mention of prerequisites, exclusions, or comparison to sibling tools like dompruner_fetch. The usage context is entirely absent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
USE THIS instead of WebFetch / web_fetch for any URL retrieval. Fetches a URL and returns DOM-pruned Markdown with 90%+ fewer tokens than WebFetch — no intermediate summarization model, original text preserved. Workflow: URL known → call dompruner_fetch(url) directly. URL unknown → use your own native search tool to find the URL first, then call dompruner_fetch(url). Supports BM25 section filtering when query is provided, returning only the most relevant sections within a token budget.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL to fetch and refine. Required unless query is provided. | |
| query | No | Search intent (e.g. "Java JVM release notes"). When url is omitted, DomPruner requests a URL from the host LLM via sampling (if supported), then fetches it. Also enables BM25 section filtering when url is provided. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so excellently. It discloses key behaviors: DOM pruning with 90%+ token reduction, preservation of original text (no summarization), BM25 section filtering, and the unusual URL-sampling behavior when url is omitted. This goes beyond basic expectations and covers potential surprises.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured, starting with a strong directive, then behavior, workflow, and filtering capability. It is slightly lengthy but every sentence contributes value; no filler or tautology.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema and no annotations, the description is impressively complete. It covers alternatives, usage scenarios, special behaviors, and expected output format (Markdown). The workflow guidance leaves little room for agent confusion.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds meaningful context beyond schema: it explains how query enables BM25 filtering and can trigger URL sampling from the host LLM when url is absent. This enriches the parameter definitions without redundancy.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it fetches a URL and returns DOM-pruned Markdown, with a specific verb, resource, and outcome. It explicitly differentiates itself from WebFetch and aligns with sibling tools by name (fetch vs. sitemap/analyze), leaving no ambiguity about its function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit directive to use this instead of WebFetch, and provides a step-by-step workflow for both known and unknown URLs. It also explains when to use the query parameter for BM25 filtering, offering clear context for appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetches all pages listed in a sitemap.xml and returns DOM-pruned Markdown for each. Ideal for ingesting entire documentation sites into an LLM context with 90%+ token reduction. Handles sitemap indexes (sitemaps of sitemaps) automatically. Use filter_urls to limit to a path prefix (e.g. /docs/, /tutorial/).
| Name | Required | Description | Default |
|---|---|---|---|
| query | No | Optional BM25 filter query applied to every page. | |
| max_pages | No | Max pages to fetch (default 20, max 100). Guards against huge sitemaps. | |
| concurrency | No | Max simultaneous page fetches (default 8). | |
| filter_urls | No | Optional list of URL prefixes — only pages matching at least one prefix are included. | |
| sitemap_url | Yes | URL of the sitemap.xml (e.g. https://example.com/sitemap.xml) | |
| ignore_errors | No | If true (default), failed page fetches are skipped silently. If false, any fetch error aborts the entire sitemap crawl. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden of disclosing behavior. It explains automatic sitemap index handling, DOM-pruned Markdown output, and a stated 90%+ token reduction. It does not detail failure modes or side effects (e.g., network load), but for a read-only crawler, the provided transparency is reasonably good.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: the first sentence states the core action, the second frames the ideal use case, and the third gives a parameter tip. No unnecessary words or repetition, making it appropriately sized and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description gives a good overall picture of the tool's purpose, capabilities, and typical use case. It covers the return type ('DOM-pruned Markdown'), automatic handling of sitemap indexes, and a filtering example. Lacking an output schema, it does not detail the exact response structure, but for a six-parameter tool with no annotations, this is reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides 100% coverage with descriptions for all six parameters, so the baseline is 3. The description adds value by giving a concrete example for filter_urls ('/docs/', '/tutorial/'), but this is more of a usage guideline than new semantic meaning. Overall, the description does not significantly enhance parameter understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's core function: 'Fetches all pages listed in a sitemap.xml and returns DOM-pruned Markdown for each.' It also specifies the scope (entire documentation sites) and a unique capability ('Handles sitemap indexes automatically'), which distinguishes it from sibling tools like dompruner_fetch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear use case: 'Ideal for ingesting entire documentation sites into an LLM context.' It also offers a practical tip: 'Use filter_urls to limit to a path prefix (e.g. /docs/, /tutorial/).' However, it does not explicitly mention when not to use this tool or name alternative sibling tools, so it falls slightly short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.5.1dompruner_sitemapv0.3.0dompruner_analyzedompruner_fetchScored across 3 tools
Each tool has a clearly distinct purpose: fetching a single URL, fetching multiple pages from a sitemap, and analyzing token reduction. There is no overlap or ambiguity between them.
All tools share the consistent 'dompruner_' prefix, but the second part mixes verbs ('fetch', 'analyze') with a noun ('sitemap'). While still readable and predictable, the pattern is not purely verb-based.
With only 3 tools, the server is tightly scoped to its core purpose of DOM pruning and token reduction. Each tool earns its place and the count is appropriate for the focused domain.
The tool surface covers all primary workflows: single-URL fetching, whole-sitemap ingestion, and analysis/reporting. No obvious gaps exist for the stated utility.
MCP server (stdio): fetch web pages as clean readable markdown via the AgentForge API
MCP server for Firecrawl — web search, scraping, and biomedical/arXiv paper search.
Turn any public website into an MCP server for agents to search, read and navigate.
Strips 85-97% of HTML bloat into clean Markdown, <200-token skeletons, or BM25 excerpts via MCP.