# dompruner-mcp

[한국어](./README.ko.md) | English

![DOM Tree Pruning for DomPruner](assets/banner.png)

> DOM AST middleware for LLM web pipelines — strips layout noise (nav, scripts, sidebars) and passes original text directly. Add a query to filter to relevant sections with BM25.

When an LLM uses a web fetch tool, the raw page HTML is converted to text and passed into the model's context — often tens of thousands of tokens of navigation, sidebars, and boilerplate the model doesn't need. DomPruner strips that at the DOM level and passes only the **original content directly to the model**.

| Call | Behavior |
|---|---|
| `dompruner_fetch(url)` | Strips layout noise → returns full extracted content |
| `dompruner_fetch(url, query)` | Strips layout noise → BM25 filters to relevant sections (falls back to full content if no match) |

```
> [DomPruner] docs.python.org
> | Raw HTML  | 44,316 tokens |
> | DomPruner |  1,328 tokens |
> | Reduction |        97.0%  |
> Fetch: 194ms · Parse: 11.2ms
```

**93.5% fewer context tokens than WebFetch on average. 45% faster end-to-end.**
→ [Full benchmark](https://github.com/dong7812/dompruner-mcp/blob/main/docs/benchmark.md)

---

## Quick Start

No installation, no API key:

```bash
npx -y dompruner-mcp
```

### Claude Code

```json
{
  "mcpServers": {
    "dompruner": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "dompruner-mcp"]
    }
  }
}
```

Add to `.mcp.json` in your project root, or `~/.claude/.mcp.json` for global. Run `/mcp` to verify.

### Claude Desktop

Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "dompruner": {
      "command": "npx",
      "args": ["-y", "dompruner-mcp"]
    }
  }
}
```

### Cursor / Windsurf / other MCP clients

```json
{
  "mcpServers": {
    "dompruner": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "dompruner-mcp"]
    }
  }
}
```

### Remote HTTP (no install, always up to date)

For clients that support HTTP transport — no Node.js install required, always runs the latest version:

```json
{
  "mcpServers": {
    "dompruner": {
      "url": "https://dompruner-mcp.vercel.app/api/mcp"
    }
  }
}
```

### LangChain / LangGraph

[`langchain-mcp-adapters`](https://pypi.org/project/langchain-mcp-adapters/) wraps any MCP stdio server as LangChain tools automatically:

```python
from langchain_mcp_adapters.client import MultiServerMCPClient

client = MultiServerMCPClient({
    "dompruner": {
        "command": "npx",
        "args": ["-y", "dompruner-mcp"],
        "transport": "stdio",
    }
})
tools = await client.get_tools()
```

### Ensuring Your AI Always Uses DomPruner

DomPruner's tool description already tells clients to prefer `dompruner_fetch` over WebFetch. If your client still falls back, add this to its instruction file:

```markdown
When retrieving a URL, always use dompruner_fetch instead of WebFetch.
- URL known → dompruner_fetch(url, query?)
- URL unknown → search for the URL first, then dompruner_fetch(url)
```

| Client | Instruction file |
|--------|-----------------|
| Claude Code | `CLAUDE.md` (project) or `~/.claude/CLAUDE.md` (global) |
| Cursor | `.cursorrules` |
| Windsurf | `.windsurfrules` |
| Cline | `.clinerules` |
| GitHub Copilot | `.github/copilot-instructions.md` |

---

## Tools

| Tool | Description |
|------|-------------|
| `dompruner_fetch` | Fetch a URL → DOM-refined Markdown. Optional `query` enables BM25+ section filtering. |
| `dompruner_sitemap` | Fetch all pages in a sitemap.xml → one refined Document per page. |
| `dompruner_analyze` | Token-reduction report for a URL without full content. |

→ **[Full tool reference](https://github.com/dong7812/dompruner-mcp/blob/main/docs/tools.md)**

---

## Benchmark Summary

### vs WebFetch (live comparison, Claude Haiku)

Measured using Anthropic's `web_fetch_20260209` server tool under identical conditions (same model, system prompt, tool definitions). → [Full methodology](https://github.com/dong7812/dompruner-mcp/blob/main/docs/benchmark.md)

| Metric | `web_fetch` tool | **DomPruner** |
|---|:---:|:---:|
| Avg context tokens | ~15,735 | **~1,019 (93.5% less)** |
| Answer quality (10 queries) | 9 / 10 | **8 / 10** |
| Avg response time | 5,811 ms | **3,168 ms (45% faster)** |
| Content fidelity | Raw page content (full HTML → text) | **DOM-extracted original text** |
| Extra API key / infra | No | **No** |

### BM25 Section Accuracy (snapshot-based, deterministic)

| Set | Pages | Avg reduction | Section preservation |
|---|:---:|:---:|:---:|
| Tuning (MDN, docs.python.org) | 10 | 98.0% | **98.0%** |
| **Validation (held-out sites)** | 5 | 98.7% | **96.0%** |

headingBoost=2.0 was selected by running a parameter sweep (1.0–3.0) on the tuning set. The validation set uses five sites not present in the tuning set (react.dev, typescriptlang.org, expressjs.com, doc.rust-lang.org, nextjs.org) to confirm the result is not overfit. The 2 %p gap is within the known heading-bias limitation.

→ **[Full benchmark](https://github.com/dong7812/dompruner-mcp/blob/main/docs/benchmark.md)** · **[Architecture](https://github.com/dong7812/dompruner-mcp/blob/main/docs/architecture.md)**

---

## vs Crawl4AI

Crawl4AI is a crawler framework — browser automation, JS rendering, click/scroll interaction, JSON extraction, deep crawl. DomPruner is LLM context middleware — a single-purpose layer that reduces token cost when an agent already knows the URL.

**They solve different problems.** But both can be used to read a web page inside an MCP agent, so the comparison is real.

### Setup

| | Crawl4AI | **DomPruner** |
|---|:---:|:---:|
| Install | `pip install crawl4ai && crawl4ai-setup` (Chromium ~300 MB) | **`npx -y dompruner-mcp`** |
| Browser required | Default install includes Chromium; HTTP-only mode available | **No** (SSR; Playwright only on CSR detection) |
| MCP server | Requires running the Crawl4AI Docker server | **`npx -y dompruner-mcp` — no server** |
| Config to filter tokens | Pass a filter object in code (`BM25ContentFilter(user_query=...)`) | **Zero — just pass `query=`** |

### Held-out validation: 5 sites not in DomPruner's tuning set

The most defensible comparison: five sites that were **not** used to tune DomPruner's parameters (react.dev, typescriptlang.org, expressjs.com, doc.rust-lang.org, nextjs.org). Both tools use the same query.

| Page | C4AI BM25ContentFilter | **DomPruner** |
|------|:---:|:---:|
| React useEffect | 4,275 | **1,257** |
| TypeScript narrowing | 2,782 | **1,214** |
| Express routing | 1,938 | **1,799** |
| Rust ownership | 1,832 | **718** |
| Next.js layouts | 1,505 | **1,379** |
| **Total** | **12,332** | **6,367** |
| **Accuracy** | **96%** | **96%** |

**On sites DomPruner was not tuned on: 1.9× fewer tokens with the same accuracy (96%).**

### Additional reference: 10-page tuning set (MDN, docs.python.org)

On DomPruner's own tuning pages, vs Crawl4AI `PruningContentFilter`: **13.2× smaller, 8 pp more accurate** (90% → 98%). Crawl4AI `BM25ContentFilter` with the same query returns near-empty output on 5 of 10 pages — a structural issue with `<dl>/<dt>/<dd>` definition-list pages where a threshold sweep (0.0–0.10) shows a hard cliff at 0.01 with no intermediate optimum.

→ **[Full comparison with all tables and threshold sweep data](https://github.com/dong7812/dompruner-mcp/blob/main/docs/benchmark.md#5-vs-crawl4ai)**

---

## Related

- **[dompruner-py](https://github.com/dong7812/dompruner-py)** — Python port. `DomPrunerLoader`, `DomPrunerSitemapLoader`, `DomPrunerFetchTool` for LangChain. `pip install dompruner`.
- **[LangChain integrations](https://docs.langchain.com/oss/python/integrations/document_loaders)** — dompruner-py listed as a third-party web loader.

---

## Glama Score
[![dompruner-mcp MCP server](https://glama.ai/mcp/servers/dong7812/dompruner-mcp/badges/card.svg)](https://glama.ai/mcp/servers/dong7812/dompruner-mcp)

---
## License

MIT