# Yet another web scraper MCP

A minimal MCP server that lets an LLM drive a real Chromium browser to scrape
websites, collect structured rows, and export them to CSV.

## Setup

```bash
uv sync
uv run playwright install chromium
```

## Running

```bash
uv run main.py
```

This starts the MCP server over stdio. Point your MCP client (e.g. Claude
Desktop / Claude Code) at `uv run --directory <this folder> main.py`.

## Installing via npm (recommended for end users)

No need to clone this repo. The npm package still runs the Python
implementation under the hood via [uv](https://astral.sh/uv) (which it will
tell you how to install if it's missing), but gives a familiar
one-line install/run command that works the same on Windows, macOS, and
Linux:

```bash
npx -y yet-another-web-scraper-mcp
```

Add it to your MCP client config, e.g. Claude Desktop's
`claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}
```

On first run, `npm install` triggers a postinstall step that runs
`uv sync` and downloads the Chromium browser for Playwright. This can
take a minute the very first time.

Requires [uv](https://astral.sh/uv) and Python 3.13+ to be resolvable on
the machine; npm alone cannot provide the Python runtime.

### Adding it to CLI coding agents

**Claude Code**

```bash
claude mcp add web-scraper -- npx -y yet-another-web-scraper-mcp
```

**Codex CLI**: add to `~/.codex/config.toml`:

```toml
[mcp_servers.web-scraper]
command = "npx"
args = ["-y", "yet-another-web-scraper-mcp"]
```

**Gemini CLI**: add to `~/.gemini/settings.json` (or a project's
`.gemini/settings.json`):

```json
{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}
```

**Cursor**: add to `.cursor/mcp.json` (project-level) or
`~/.cursor/mcp.json` (global):

```json
{
  "mcpServers": {
    "web-scraper": {
      "command": "npx",
      "args": ["-y", "yet-another-web-scraper-mcp"]
    }
  }
}
```

**Windsurf**: same `mcpServers` JSON shape as Cursor above, in
Windsurf's MCP config panel or `~/.codeium/windsurf/mcp_config.json`.

Any other MCP-compatible client that takes a raw `command`/`args` pair
(stdio transport) can point at the same `npx -y yet-another-web-scraper-mcp` command;
the JSON snippets above are all equivalent to that one line.

## Tools

- `open_browser(headless=True)` / `close_browser()`: start/stop the browser.
- `navigate(url)`: go to a URL, returns cleaned page content.
- `click(selector)`: click a CSS-selected element (e.g. a "next page" button),
  returns the resulting cleaned content.
- `get_page_content(max_length)`: re-read the current page's HTML, with
  `<script>` and `<svg>` tags stripped out since they're noise for an LLM.
- `get_page_links()`: list `{text, href}` for links on the page.
- `add_record(record)`: append one scraped row (dict of column -> value).
- `get_records()`: review everything collected so far.
- `export_csv(filename)`: write the collected rows to a CSV via pandas.

Typical flow: `open_browser` -> `navigate` -> `get_page_content` /
`get_page_links` -> `click` to page through results -> `add_record` per item
-> `export_csv` when done -> `close_browser`.