# websearch-mcp

An MCP server that hooks into Claude Code and answers queries with **full page content**, not just snippets.

Flow when Claude calls the `websearch` tool:

```
Claude: websearch("PHP send email class")
  → 1. SearXNG search   → top N result URLs (title + url + snippet)
  → 2. BrowserOS        → open each URL as a background tab, read body text (no HTML)
  → 3. fallback         → Playwright (headless Chromium) if present, else plain HTTP fetch
  → 4. return           → [{title, url, text}] back to Claude
```

"10 tabs at a time": BrowserOS opens up to `WEBSEARCH_RESULTS` background tabs in one batch and reads them in parallel. Any page that fails (blocked, DNS, 404) is skipped gracefully and reported, with a per-page fallback to Playwright/curl.

## Setup

```bash
npm install
npm run build
```

Node 18+. Playwright is optional — if globally installed (`npm root -g`) the fallback picks it up automatically; otherwise it silently uses plain HTTP fetch.

## Register in Claude Code

In the project you want to use it from, copy the template and set your **absolute** path:

```sh
cp .mcp.json.example .mcp.json   # then edit <YOUR_CLONE_DIR>
```

```json
{
  "mcpServers": {
    "websearch": {
      "command": "node",
      "args": ["<YOUR_CLONE_DIR>/dist/index.js"],
      "env": { "WEBSEARCH_SEARXNG_URL": "http://100.77.7.3:8890/search", "WEBSEARCH_BROWSEROS_URL": "http://127.0.0.1:9002/mcp" }
    }
  }
}
```

> `.mcp.json` holds a machine-specific absolute path, so it's gitignored here — don't commit it. Configure per-machine from the `.mcp.json.example` template.

Then in a Claude Code session ask anything, e.g. *"Search PHP send email class and summarize the top 3 results."*

## Config (env vars)

All config is via env vars in the MCP server entry — no code changes to move endpoints.

| Var | Default | Meaning |
|---|---|---|
| `WEBSEARCH_SEARXNG_URL` | `http://100.77.7.3:8890/search` | SearXNG endpoint (JSON format auto-appended) |
| `WEBSEARCH_BROWSEROS_URL` | `http://127.0.0.1:9002/mcp` | BrowserOS MCP (streamable HTTP) |
| `WEBSEARCH_RESULTS` | `10` | Top N results to fetch full text for (max 12) |
| `WEBSEARCH_MAX_PAGE_CHARS` | `8000` | Truncate each page's text to this many chars (context budget) |
| `WEBSEARCH_PAGE_TIMEOUT_MS` | `25000` | Per-page read/navigation timeout |
| `WEBSEARCH_USE_BROWSEROS` | `1` | Use BrowserOS batch extraction |
| `WEBSEARCH_USE_PLAYWRIGHT` | `1` | Fall back to headless Chromium |
| `WEBSEARCH_USE_CURL` | `1` | Last fallback: plain HTTP fetch |

Set `WEBSEARCH_USE_BROWSEROS=0` to force the Playwright/curl path.

## Tool

- **`websearch(query, n?)`** — `query` string, `n` = how many top results to read (default `WEBSEARCH_RESULTS`). Returns JSON: `{ query, results: [{ title, url, text, source, error? }] }`. `source` is `browseros` / `playwright` / `curl`; `error` present on failures (e.g. bot-blocked).

## Layout

```
src/
  index.ts               MCP stdio server + websearch tool
  config.ts              env-var config
  searxng.ts             SearXNG JSON search
  text.ts                html-strip / squash / bot-block heuristics
  extraction/
    extractor.ts         orchestrates BrowserOS → playwright → curl
    browseros.ts         MCP *client* to BrowserOS (tabs new/list, read, close)
    playwright.ts        headless Chromium fallback (auto-resolves global install)
    curl.ts              realistic-browser HTTP fetch + cheerio text extraction
```