# webharvest

A local Firecrawl alternative: scrape and search over MCP. Query search engines, fetch and parse web pages, all via Claude's MCP protocol.

## What It Does

**webharvest** is a daemon that:
- Runs a local [Model Context Protocol](https://modelcontextprotocol.io/) server
- Exposes search (via SearXNG) and web scraping (via Playwright)
- Integrates with Claude Code as an MCP server
- Enforces politeness: per-domain request queueing (2 concurrent, 500 ms apart)
- Caches results in SQLite for 24 hours by default

You control the browser automation — no opaque API calls, no subscription, all on your machine.

## Installation

### Prerequisites

- **Node 20+**
- **Docker & Docker Compose** (for SearXNG search)
- **Chromium** (installed via Playwright, see below)

### Steps

1. **Clone and build:**
   ```bash
   git clone <repo-url>
   cd webharvest
   npm install
   npx playwright install chromium
   npm run build
   ```

2. **Install the CLI globally:**
   ```bash
   npm link
   ```
   This makes `webharvest` available as a command in your shell. Alternatively, use `npx webharvest` to run without installing globally.

3. **Start SearXNG (search backend):**
   ```bash
   # Create your local config from the template (one-time).
   # searxng/settings.yml is gitignored: secret_key is per-install and must not be committed.
   cp searxng/settings.yml.example searxng/settings.yml
   sed -i '' "s/CHANGE_ME_TO_A_RANDOM_STRING/$(openssl rand -hex 32)/" searxng/settings.yml
   
   # Start the container
   docker compose up -d
   
   # Verify it's up (wait 10 seconds, then):
   curl -s 'http://127.0.0.1:8080/search?q=hello&format=json' | head -c 100
   ```
   The last command must return JSON. If it returns HTML or a 403, `formats` in
   `searxng/settings.yml` is missing `json` — fix it and restart the container.
   Note that `limiter` must stay `false`, or SearXNG rate-limits the daemon querying it.

4. **Register the daemon with launchd (macOS only for now):**
   ```bash
   webharvest install
   ```
   This registers the daemon with launchd, so it starts automatically and restarts if it crashes. Use `webharvest start` to start it immediately, or it will start at next login.

5. **Connect to Claude Code:**
   ```bash
   claude mcp add webharvest -- node /absolute/path/to/webharvest/dist/mcp/index.js
   ```
   Replace the path with the actual path to your clone. You can find it with `pwd` in the webharvest directory.

### Automated Setup (hand this prompt to an AI agent)

If you'd rather not run the steps above by hand, paste the prompt below into an
agent that can execute shell commands and edit files (Claude Code, Codex,
Cursor, Gemini CLI, …). It performs the same installation and then registers
webharvest as an MCP server in whichever agents you have installed.

Read it before you run it: it installs a launchd job, starts a Docker
container, and edits your agents' MCP config files.

````text
You are setting up "webharvest" — a local MCP server that gives you web search
and web scraping. Work through the steps in order, run every verification
command, and stop and report if one fails instead of continuing.

Repository: the directory you were pointed at, or clone it if I gave you a URL.
Let REPO be its absolute path (get it with `pwd`, never guess, never use `~`).

1. Prerequisites. Check `node --version` (must be 20+), `docker --version`, and
   `docker compose version`. If Docker is not installed or its daemon is not
   running, tell me and stop — search will not work without it.

2. Build.
     cd REPO && npm install && npx playwright install chromium && npm run build
   Verify REPO/dist/mcp/index.js, REPO/dist/cli/index.js and
   REPO/dist/daemon/index.js now exist.

3. Search backend (SearXNG). If REPO/searxng/settings.yml does not exist:
     cp searxng/settings.yml.example searxng/settings.yml
   then replace the literal CHANGE_ME_TO_A_RANDOM_STRING in it with the output
   of `openssl rand -hex 32`. Never commit this file — it is gitignored on
   purpose. In that file confirm `formats` contains `json` and `limiter` is
   `false`; fix them if not. Then:
     docker compose up -d
   Wait ~10 s and verify search actually answers with JSON:
     curl -s 'http://127.0.0.1:8080/search?q=hello&format=json' | head -c 100
   HTML or a 403 back means the settings above are wrong. Fix and restart the
   container; do not move on.

4. Daemon. Run `npm link` in REPO so the `webharvest` command exists (if that
   needs sudo on this machine, tell me instead of running sudo yourself).
   On macOS: `webharvest install` — this already loads the launchd job and
   starts the daemon, so run `webharvest status` next and only call
   `webharvest start` if it reports the daemon is not running.
   On Linux/Windows there is no launchd integration — start the daemon in the
   background yourself with `node REPO/dist/daemon/index.js` and tell me it
   will not survive a reboot.
   Verify: `curl -s http://127.0.0.1:8787/health` returns JSON.

5. Register as an MCP server. The MCP entry point is always:
     command: node
     args:    ["REPO/dist/mcp/index.js"]
   It talks to the daemon over http://127.0.0.1:8787 and does not start it, so
   step 4 must be working first. If the daemon runs on another port, also set
   the env var WEBHARVEST_URL for the server.

   Do this for each agent that is actually installed on this machine — check
   with `command -v claude`, `command -v codex`, … and not by the existence of
   a ~/.something directory, which is often left behind by an uninstall. Skip
   the rest, and never overwrite an existing "webharvest" entry without
   telling me:

   - Claude Code:
       claude mcp add -s user webharvest -- node REPO/dist/mcp/index.js
     `-s user` matters: without it the server is registered only for the
     directory you happen to be in.
   - Codex: add to ~/.codex/config.toml
       [mcp_servers.webharvest]
       command = "node"
       args = ["REPO/dist/mcp/index.js"]
   - Cursor (~/.cursor/mcp.json), Windsurf, Claude Desktop, and most others
     use the same JSON shape — merge into the existing "mcpServers" object,
     do not replace the file:
       { "mcpServers": { "webharvest": {
           "command": "node", "args": ["REPO/dist/mcp/index.js"] } } }
   - Gemini CLI: same JSON shape in ~/.gemini/settings.json.

   If an agent you find is not in this list, look up its MCP config location in
   its own docs and use the same command/args.

6. Verify end to end. Talk to the MCP server directly over stdio, so you do
   not have to restart anything to know it works:
     printf '%s\n' \
       '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}' \
       '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
       '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"model context protocol","limit":3}}}' \
       | node REPO/dist/mcp/index.js 2>/dev/null | tail -1
   That must come back with real search results, not "Не удалось" — the latter
   means the daemon or SearXNG is not reachable, so go back to step 3 or 4.
   Then repeat with a `scrape` call on one of the returned URLs.
   Also confirm the registration itself: `claude mcp list` for Claude Code
   (expect "webharvest … ✔ Connected"), `codex mcp list` for Codex. For the
   others, tell me to restart the agent and check that the tools `scrape` and
   `search` appear.

7. Report: what you installed, the exact paths and config files you touched,
   what still needs a restart, and anything you skipped and why.
````

### Common Commands

- `webharvest start` — Start the daemon manually
- `webharvest stop` — Stop the daemon
- `webharvest status` — Check if it's running
- `webharvest logs` — Follow daemon logs (Ctrl+C to exit)
- `webharvest uninstall` — Remove launchd registration

## What It Can't Do

**webharvest** has clear limitations. Before opening an issue, check this list:

### ❌ No PDF Support

PDFs are not parsed. The scraper detects the non-HTML content type and returns a clean `not_html` error — it never hands you raw binary data. Use a dedicated PDF parser if you need full-text extraction from PDFs.

### ❌ No Authentication

Pages that require login are not supported. The scraper has no credential storage, session management, or form-filling logic for authentication. If a page redirects to a login screen, the scraper will return the login page HTML, not the authenticated content.

### ⚠️ Bot-Protected Sites — Recognised, Not Bypassed

Sites behind a bot wall (Cloudflare, DataDome, PerimeterX, Akamai) are often
unreadable, and webharvest does not try to defeat the wall. What it does:

- **Says so honestly.** When a protection is recognised, the response is
  `blocked` and names the protection. A challenge page is never handed over as
  content, and an empty body is never handed over as an article.
- **Uses your browser.** With `browserChannel: "chrome"` the scraper drives your
  system Google Chrome instead of bundled chromium (falling back to chromium
  when Chrome is not installed); with `browserProfileDir` set, the browser keeps
  a persistent profile, so cookies survive daemon restarts and a site you have
  already visited does not see a stranger on every run. Both options are about
  making the request look like an ordinary returning visitor — they are not a
  way past a check, and on the protected sites I tested neither the channel
  choice nor a warmed-up profile made a wall go away.
- **Gives up rather than guesses.** No captcha-solving services, no proxy
  rotation, nothing that needs a human (image selection, hCaptcha, DataDome and
  PerimeterX widgets), and no attempt at servers that block by IP reputation.

The line between a wall and a form: Turnstile widget markup counts as a wall
only when the page has almost no readable content of its own after extraction
(footers and sidebars do not count towards that measure — they cannot promote a
wall into an article; a wall measured tens of characters, an article carrying the
same widget markup measured ~11 000) and offers nothing for a human to fill in —
a sign-in or sign-up page that keeps only its form and the widget is handed over
as a page, not reported as `blocked`.

Protections also come and go. `bazaraki.com`, which needed the browser when this
section was written, now answers plain HTTP requests.

### ❌ robots.txt Is Not Enforced as a Prohibition

The scraper does **not** read or honour `robots.txt` as a legal prohibition. Instead, politeness is enforced through rate limiting:
- **Per-domain request queue**: at most 2 concurrent requests per domain, 500 ms apart
- **This is respectful by default**: delays stack, so successive requests to the same domain are naturally throttled

The brief specifies why `robots.txt` as a prohibition was deliberately omitted: it is a machine-readable convention, not a legal boundary, and enforcing it in code would create false confidence. Rate limiting is more honest: you are making fewer, slower requests, which is measurably less disruptive than hitting a site at full speed.

A `respectRobots` config flag was considered and removed. It is not coming back. Rate limiting is the mechanism.

### ❌ SSRF Protection Doesn't Close DNS Rebinding

The private-address check (`assertPublicHost`) resolves the hostname and validates the addresses it gets, but that's a check-then-use, not a closed gate: undici and Playwright each resolve the hostname again, independently, when they actually connect. A host with a short DNS TTL that answers with a public address during the check and a private one moments later (classic DNS rebinding) can walk past the guard. Properly closing this needs pinned-IP dispatch — resolve once, force every subsequent connection onto that exact address, and validate the connecting socket's remote address before reading any bytes — which is real engineering, not a quick patch, and is out of scope for a personal tool. Known and accepted, not silently assumed away.

## Configuration

Settings live in `~/.webharvest/config.json`. The daemon only ever reads this
file — it is never created for you, so if you want to override defaults,
create it yourself:

```json
{
  "port": 8787,
  "cachePath": "/Users/you/.webharvest/cache.db",
  "cacheTtlMs": 86400000,
  "searxngUrl": "http://127.0.0.1:8080",
  "braveApiKey": null,
  "idleTimeoutMs": 300000
}
```

Note: `cachePath` is used as-is, with no `~` expansion — write an absolute
path (as above), not `~/.webharvest/cache.db`, or you'll get a literal `./~`
directory relative to wherever the daemon happens to be started from.

Environment variables override config file settings:
- `WEBHARVEST_PORT` — daemon listening port
- `WEBHARVEST_SEARXNG_URL` — search backend URL
- `BRAVE_API_KEY` — optional Brave Search API key (for fallback search)
- `WEBHARVEST_BROWSER_CHANNEL` — `"chromium"` (default) | `"chrome"` — use your
  system Google Chrome instead of the bundled chromium. If Chrome is not
  installed, the daemon automatically falls back to bundled chromium.
- `WEBHARVEST_BROWSER_PROFILE_DIR` — path to a persistent browser profile.
  Cookies survive daemon restarts, so a site you have already visited does not
  see a stranger on every run. Disabled by default. Note: the profile is shared
  between scrape and browser-use via two subdirectories (`<dir>/scrape` and
  `<dir>/sessions`) — two Chromium processes cannot share one `userDataDir`,
  hence the split.
- `WEBHARVEST_SCRAPE_ENGINE` — `"chromium"` (default) | `"lightpanda"` — render
  engine for `scrape`. With `"lightpanda"` pages are rendered by
  [Lightpanda](https://github.com/lightpanda-io/browser) (a headless browser
  without graphics, several times lighter on memory than Chromium). If
  Lightpanda fails, hits a challenge, or returns less than 1000 characters of
  text, the page is rendered again with Chromium, which only starts when needed.
  Browser-use tools always run on Chromium. Measurements:
  [docs/bench-lightpanda.md](docs/bench-lightpanda.md).
- `WEBHARVEST_LIGHTPANDA_BIN` — path to the Lightpanda binary (default:
  `lightpanda` in `PATH`). Install it with `brew tap lightpanda-io/browser &&
  brew install lightpanda`, or download it from the
  [releases page](https://github.com/lightpanda-io/browser/releases).

The same keys can be set in `~/.webharvest/config.json` as `browserChannel`,
`browserProfileDir`, `scrapeEngine` and `lightpandaBin`.

Note: `host` and `allowPrivate` cannot be set from config.json for security reasons. The daemon always binds to `127.0.0.1` only.

There is no LLM configuration, and no LLM API key is ever needed. The daemon
doesn't run its own model — browser-use is a set of deterministic tools
(`browser_open`, `browser_snapshot`, `browser_click`, `browser_fill`, …) that
directly manipulate the page; the reasoning about *what* to do with the page
is left entirely to whichever agent is calling webharvest (e.g. Claude Code),
which already sees the page tree in its own context.

## Browser Use Tools

Beyond `scrape` and `search`, webharvest exposes a small set of deterministic
browser-automation tools over MCP. There is no inference step in the daemon:
every tool call does exactly the one thing it's named for, and returns a
diff of what changed on the page (or the fresh page tree itself) so the
calling agent can decide the next step.

- `browser_open(url)` — opens a page in a real (headless) browser, returns a
  `sessionId` and the page's accessibility tree. The page stays open between
  calls.
- `browser_snapshot(sessionId)` — returns a fresh tree of the same page,
  without performing any action.
- `browser_click(sessionId, elementId)`
- `browser_hover(sessionId, elementId)`
- `browser_fill(sessionId, elementId, text, variables?)` — clears the field
  and types `text` into it.
- `browser_type(sessionId, elementId, text, variables?)` — types `text`
  character by character (real keydown events), for fields that need it
  (autocompletes, input masks).
- `browser_press(sessionId, elementId, key)`
- `browser_select(sessionId, elementId, value)` — picks an option in a
  native `<select>` by its label.
- `browser_scroll(sessionId, elementId, percent)`
- `browser_close(sessionId)`

`elementId` is a `frame-node` address (e.g. `0-18372`) copied verbatim from
the tree returned by the last `browser_open`/`browser_snapshot` call, or from
the diff returned by the previous action — addresses can change after every
action, so always use the freshest one.

`browser_fill`/`browser_type` still support the same `variables` +
`%name%`-placeholder mechanism as before. Its purpose changed along with the
architecture: it used to keep secrets out of the daemon's own model call;
now, with no model inside the daemon, it keeps secrets out of the calling
agent's own context instead — write `%password%` in `text`, pass the real
value as `variables: { password: "..." }`, and the daemon substitutes it
right before typing into the browser. The value never has to appear in the
agent's conversation history to be used. The substituted value is still
redacted back out of the page tree the daemon returns (see
`redactSecrets`/`registerSecrets` in `src/core/a11y/format.ts` and
`src/daemon/service.ts`), so it doesn't leak back to the agent through a
later snapshot either.

## Architecture

- **src/core/** — Scraping logic: browser pool, SSRF protection, cache
- **src/daemon/** — HTTP server (localhost only) with health check
- **src/mcp/** — Model Context Protocol server (stdio-based)
- **src/cli/** — Command-line interface for daemon management
- **searxng/** — Docker Compose config and SearXNG settings

## Development

```bash
npm test          # Run unit tests
npm run typecheck # TypeScript strict mode check
npm run build     # Compile to dist/
```

Tests are excluded from the live-test suite unless you run `npm run test:live`.

## License

MIT — see [LICENSE](./LICENSE).

This is a personal tool, and running it is a local decision: the requests come
from your machine under your IP, and staying within the terms of the sites you
point it at is on you. The daemon deliberately does not read `robots.txt` (see
above) — it enforces politeness with rate limiting instead.

## Attribution

webharvest переиспользует код из [Stagehand](https://github.com/browserbase/stagehand)
(MIT, © Browserbase, Inc.) — слой представления accessibility-дерева и контракт
промптов act/observe/extract. Подробности в [NOTICE](./NOTICE).

HTML-фикстуры в `test/fixtures/` — снимки публичных страниц, права остаются у их
авторов. Происхождение и условия по каждой — в
[test/fixtures/README.md](./test/fixtures/README.md).