# mcp-dom-extract

MCP server that downloads a web page, parses its DOM and extracts precise data
via CSS selectors, then returns the structured result (JSON) to the client.

## Requirements

- Node.js ≥ 20.18.1

## Installation and development

```bash
npm install
npm run build   # compiles src/ → dist/
npm run dev     # runs the server in watch mode (tsx)
npm run smoke   # end-to-end test against a local page
```

## Exposed tools

### `extract_data_from_url`

Downloads a page and extracts values via CSS selectors.

| Parameter | Type | Description |
|---|---|---|
| `url` | string | Absolute URL of the page |
| `selectors` | array | List of `{ name, css, attribute?, multiple? }` |
| `max_value_length` | number | Maximum length of each extracted value; longer values are truncated (default: 2000, cap: 100000) |

- `attribute`: if set (e.g. `href`, `src`), returns the attribute value
  instead of the text.
- `multiple`: if `true`, returns an array of all matches.

### `extract_page_text`

Downloads a page and returns its readable text, optionally scoped to a CSS
selector and truncated to `max_length` characters.

| Parameter | Default | Description |
|---|---|---|
| `url` | — | Absolute URL of the page |
| `selector` | — | CSS selector scoping the text; defaults to the whole document |
| `max_length` | 20000 | Maximum number of characters returned (cap: 100000) |

### `inspect_page_structure`

Returns a **condensed DOM outline** — tags, ids, classes, truncated text
samples, repeated identical siblings aggregated (`count` + `samples`) — to pick
relevant selectors **without loading the full page into the context**.

| Parameter | Default | Description |
|---|---|---|
| `max_nodes` | 200 | Maximum number of outline entries (cap: 500) |
| `max_text_length` | 40 | Maximum length of each sample (cap: 200) |

### `discover_selectors`

The model describes what it is looking for with **keywords**; the server scans
the DOM and returns candidate selectors whose text matches, with a sample. No
full page enters the context.

```json
{
  "url": "https://example.com/product",
  "targets": [
    { "name": "price", "keywords": ["19.99", "$", "price"] },
    { "name": "name", "keywords": ["Widget", "product"] }
  ]
}
```

| Parameter | Default | Description |
|---|---|---|
| `url` | — | Absolute URL of the page |
| `targets` | — | List of `{ name, keywords }`; keywords are matched case-insensitively against element text (1–10 targets, up to 10 keywords each) |
| `max_candidates` | 5 | Maximum number of candidate selectors returned per target (cap: 10) |

### `probe_selectors`

Checks selectors at low cost: for each, returns the **number of matches** and a
sample of the first one, to refine before the final extraction.

| Parameter | Type | Description |
|---|---|---|
| `url` | string | Absolute URL of the page |
| `selectors` | array | List of `{ name, css, attribute? }` (`multiple` is not supported) |

## Recommended flow

1. **`discover_selectors`** (or `inspect_page_structure`) → candidate selectors
2. **`probe_selectors`** → verification and refinement
3. **`extract_data_from_url`** → final structured data

## robots.txt compliance (RFC 9309)

Before each fetch, the server checks `/robots.txt` of the target origin and
applies the rules of the `mcp-dom-extract` group (or `*` if no specific group
exists). The cache is 24 h per origin, per the RFC.

- **404 / 4xx errors** → access allowed (the file is "unavailable").
- **5xx / network / timeout errors** → extraction blocked: the file is
  "unreachable", RFC 9309 requires a complete disallow. Unreachable states are
  re-checked after 5 minutes instead of being cached for 24 h.
- **Parsing limit** of 512 KiB (the RFC requires at least 500 KiB).
- **`crawl-delay`** is not supported (non-standard extension outside RFC 9309).

To disable the check (e.g. for internal test sites):

```bash
MCP_DOM_EXTRACT_IGNORE_ROBOTS=1 node dist/index.js
```

## Page caching

Fetched pages are kept in an in-memory LRU cache, keyed by URL, so a
multi-step workflow (`discover_selectors` → `probe_selectors` →
`extract_data_from_url`) fetches each page **once** instead of once per tool
call. Cache entries expire after a TTL, the total size is bounded (oldest
entries are evicted first), errors are never cached, and concurrent requests
for the same URL share a single fetch.

| Env var | Default | Description |
|---|---|---|
| `MCP_DOM_EXTRACT_CACHE_TTL_MS` | `300000` (5 min) | How long a fetched page is reused before re-fetching |
| `MCP_DOM_EXTRACT_CACHE_MAX_BYTES` | `52428800` (50 MiB) | Total cache size limit; set to `0` to disable caching |

## Client configuration

### Claude Desktop standard (`mcpServers`)

This is the format used by Claude Desktop, Cherry Studio and most MCP clients
(the emerging standard):

```json
{
  "mcpServers": {
    "mcp-dom-extract": {
      "command": "npx",
      "args": ["-y", "--allow-git=all", "github:Llamatron2112/mcp-dom-extract"]
    }
  }
}
```

> Zed uses the same entry under the `mcp_servers` key instead.

## Distribution from GitHub

For `npx github:user/mcp-dom-extract` to work without a build step on the
client side, the **`dist/` folder** generated by `npm run build` must be
committed. The `bin` field of `package.json` points to `dist/index.js`.

Since npm 11.10+ (and by default in npm 12), installing packages directly from
Git is disabled for security reasons (`EALLOWGIT`). Pass `--allow-git=all` to
opt in:

```bash
npx -y --allow-git=all github:Llamatron2112/mcp-dom-extract
```

## Known limitations

- JavaScript-rendered pages (SPAs): client-side generated content is not
  visible without a headless browser (Playwright would be a future addition).
- The sites' terms of use remain your responsibility: the server applies
  robots.txt, not ToS.
- Page encoding is detected by cheerio; pages with exotic charsets may be
  decoded incorrectly.