# Department Web-Search MCP Gateway

A self-hosted **web-search service** the whole department can share. It reuses a
**single logged-in browser session** (a shared service account), so intranet /
SSO / consent-wall logins are handled **once** — every client just calls a
`web_search` tool, no per-user login or API key.

Any MCP client connects to **one URL**:

- Chatbox (≥1.14)
- OpenCode — local, on a shared server, or via vscode-remote
- Claude Code (and other coding agents that speak MCP)

It is the **T1 “centralized search gateway”** from the research notes:
one internal machine + one shared Chrome profile + one HTTP MCP endpoint.

---

## How it works

```
Chatbox / OpenCode(local|server|vscode-remote) / Claude Code
        │  remote MCP (Streamable HTTP, /mcp) — same URL for everyone
        ▼
┌──────────────────────────────────────────────┐
│  Gateway (this service, Node + Express)       │
│   • Bearer token (optional) + Host validation │
│   • MCP tools: web_search / read_webpage      │
└──────────────────────────────────────────────┘
        │  connectOverCDP / launchPersistentContext
        ▼
┌──────────────────────────────────────────────┐
│  Chrome (persistent profile, shared account)  │  ← logged in ONCE via `npm run login`
│   • per-request new tab (isolation)           │
│   • concurrency cap + timeouts                │
└──────────────────────────────────────────────┘
        │  optional fallback
        ▼
   SearXNG (if SEARXNG_URL set) — public-search fallback when browser returns nothing
```

`createMcpHandler` serves **both** 2025-era and 2026-era MCP clients on the same
`/mcp` endpoint, so client transport compatibility is not a concern.

---

## Headless Linux servers + a Windows PC for login

Servers have no GUI, but a human can log in on a Windows PC. Pick a mode in
`.env` (`BROWSER_MODE`) — the code is identical, only config differs.

> ⚠️ **Do NOT copy a Windows Chrome *profile* directory to Linux.** Chromium
> encrypts cookies with OS-bound keys (DPAPI on Windows, keyring/”peanuts” on
> Linux), so a copied profile loses the login silently. Use one of the
> cross-OS-safe modes below.

### Mode C — `BROWSER_MODE=cdp` (recommended): Linux gateway attaches to the Windows browser
- **Windows PC** (stays on): log in once with the shared account, then keep
  Chrome running with a local-only debug port:
  ```
  chrome --remote-debugging-port=9222 --remote-debugging-address=127.0.0.1 ^
         --user-data-dir=C:\dept-search-profile
  ```
- Carry that port safely to the Linux server with an **SSH reverse tunnel**
  (run on the Windows PC; Win10/11 ships OpenSSH):
  ```
  ssh -R 9222:127.0.0.1:9222 linuxuser@gateway.server
  ```
- **Linux server**: `.env` → `BROWSER_MODE=cdp`,
  `CDP_ENDPOINT=http://127.0.0.1:9222` (local on the server, tunneled back to
  the Windows browser). Then `npm start`.
- Login stays **live** (cookies refresh as the browser is used); no profile copy;
  the unauthenticated CDP port is never on the network. Downside: Windows PC off
  → searches fail until it’s back (use Mode B if that’s unacceptable).

### Mode B — `BROWSER_MODE=storagestate`: snapshot, Linux self-sufficient
- **Windows PC**: `npm run login` (headed), log in, press Enter → writes
  `auth.json` (OS-agnostic JSON of cookies + localStorage).
- Copy `auth.json` to the Linux server, set `BROWSER_MODE=storagestate`,
  `STORAGE_STATE_FILE=./auth.json`, run `npm start`. Linux runs its own headless
  browser loading the snapshot — no tunnel, survives the Windows PC being off.
- Trade-off: a **frozen snapshot** — re-export when the SSO cookie expires;
  carries only cookies + localStorage (not IndexedDB/client certs) — fine for
  most SSO.

### Mode A — `BROWSER_MODE=persistent`: Windows PC runs everything
- If a spare Windows PC can be the always-on service host: `npm run login`
  there (seeds the profile), then `npm start` with `BROWSER_MODE=persistent`.
- Linux servers are pure clients pointing at `http://<windows-pc>:8787/mcp`.
- Simplest of all — no tunnel, no snapshot ceremony.

Client onboarding is identical in every mode: clients point at the gateway’s
MCP URL; the gateway talks to whichever browser mode is configured.

---

## Bring-up runbook — Mode C (Linux gateway + Windows browser)

The confirmed setup: a **Linux server** runs the gateway; an **always-on
Windows PC** runs a real Chrome (logged in once) and an SSH reverse tunnel.
No browser is downloaded to the Linux server (`playwright-core` only).

### Windows PC (once, then leave running) — see `windows/README.md`
1. `windows\start-browser.ps1` → dedicated Chrome on `127.0.0.1:9222`, profile
   `C:\dept-search-profile`. Sign in with the shared account (SSO/2FA). Keep it open.
2. `$env:GATEWAY_SSH = "linuxuser@gateway.server"; windows\start-tunnel.ps1`
   → maintains `ssh -R 9222:127.0.0.1:9222 gateway`, auto-reconnects.
3. Make both **Scheduled Tasks** (At startup / On logon, run whether logged on
   or not) so the PC is a self-healing browser appliance.

### Linux gateway server (this machine)
```bash
cd dept-web-search-gateway
cp .env.example .env
# edit .env:
#   BROWSER_MODE=cdp                       (default)
#   CDP_ENDPOINT=http://127.0.0.1:9222     (the tunneled port, local on this server)
#   HOST=0.0.0.0
#   ALLOWED_HOSTS=search.internal,localhost   # hostnames clients will use
#   GATEWAY_TOKEN=...                      (optional; else rely on network ACL)
npm install                 # lean — playwright-core, no Chromium download
npm run build               # typecheck
npm start                   # dev (tsx); or `npm run build && npm run start:prod`
curl http://127.0.0.1:8787/health         # {"ok":true,...}
```
Point clients at `http://<this-server>:8787/mcp` (see Client onboarding).

### Sanity-check the tunnel
On the Linux server:
```bash
curl -s http://127.0.0.1:9222/json/version   # Chrome's JSON → tunnel + Chrome are up
```
Empty / connection refused → the Windows Chrome or the reverse tunnel isn’t
running yet; `web_search` will fail until it is.

---

## Setup (one-time)

```bash
cd dept-web-search-gateway
npm install                 # also runs `playwright install chromium`
cp .env.example .env       # then edit .env (see knobs below)
```

### 1) Seed the shared login (the crux)

Run **once** on a machine with a display (or under `xvfb-run -a`):

```bash
npm run login
# or, for an internal portal:
LOGIN_START_URL=https://wiki.internal npm run login
```

A real Chrome window opens. Sign in with the **shared service account**
(SSO / 2FA), confirm you’re logged in to the search engine / portal, then close
the window. The session is persisted to `BROWSER_PROFILE_DIR` (default `./.profile`)
and reused by the headless gateway from now on.

> Servers are headless? Do the `npm run login` step on the Windows PC, then
> choose Mode B (copy `auth.json` to Linux) or Mode C (SSH-tunnel CDP to Linux)
> as described in **“Headless Linux servers + a Windows PC for login”** above.
> Renewal when the SSO session expires: Mode A/B → re-run `npm run login` (and
> re-copy `auth.json` for B); Mode C → just re-login in the Windows Chrome.

### 2) Run the gateway

```bash
npm start                   # dev (tsx)
# or production:
npm run build && npm run start:prod
```

You should see:

```
[server] MCP gateway on http://0.0.0.0:8787/mcp  (engine=bing)
[server] profile=./.profile
```

---

## Client onboarding (give these to your colleagues)

Replace `search.internal` / `8787` with your gateway host/port. Everyone uses
the **same URL**.

### Chatbox (≥1.14)
Settings → MCP → **Add Server** → choose **Remote / URL**:
- URL: `http://search.internal:8787/mcp`
- (if `GATEWAY_TOKEN` is set) add a header `Authorization: Bearer <TOKEN>` where the client supports it; otherwise protect with network ACL.

One-click deep link (put on your intranet page):
```
chatbox://mcp/install?server=<base64 of {"name":"websearch","url":"http://search.internal:8787/mcp"}>
```

### OpenCode — all three flavors
Add to `opencode.json` (project) or `~/.config/opencode/opencode.json` (global):
```json
{
  "mcp": {
    "websearch": {
      "type": "remote",
      "url": "http://search.internal:8787/mcp",
      "enabled": true
    }
  }
}
```
- **Local opencode**: same snippet, host = `127.0.0.1` or the gateway host.
- **Server opencode**: the process runs on the server → point at the gateway’s
  internal URL directly (the server must reach it over the internal network).
- **vscode-remote opencode**: the process runs on the remote host → point at the
  gateway’s internal URL (reachable from that host). No tunneling needed because
  the gateway is on the internal network.
- Verify: `opencode mcp list`.

### Claude Code
```bash
claude mcp add --transport http websearch http://search.internal:8787/mcp
# with a token:
claude mcp add --transport http --header "Authorization: Bearer <TOKEN>" \
  websearch http://search.internal:8787/mcp
```

### Cline / Cursor / others
If they support remote MCP, point at the same URL. If they only do **stdio**,
run a tiny local shim that calls the HTTP gateway (a 20-line wrapper) — not
included here, but trivial to add.

---

## Tools exposed

| Tool | Args | Returns |
|---|---|---|
| `web_search` | `query` (str, required), `engine` (bing\|google\|duck\|custom, optional) | list of `{title, url, snippet}` as text + JSON |
| `read_webpage` | `url` (str, required) | `# title` + main text (≤20k chars), login/SSO handled |

The agent in Chatbox/OpenCode/Claude Code will call `web_search` when it needs
fresh info, and `read_webpage` to read a specific page — no extra wiring.

---

## Config knobs (`.env`)

| Var | Default | Meaning |
|---|---|---|
| `HOST` | `0.0.0.0` | bind address. `127.0.0.1` = localhost-only (+auto DNS-rebinding protection) |
| `ALLOWED_HOSTS` | — | comma list of hostnames clients use (enables Host-header validation). **Set when binding 0.0.0.0** |
| `PORT` | `8787` | listen port |
| `GATEWAY_TOKEN` | — | if set, require `Authorization: Bearer <token>`. Empty = no auth (network-ACL only) |
| `BROWSER_MODE` | `cdp` | `persistent` / `storagestate` / `cdp` — see topology section |
| `CDP_ENDPOINT` | `http://127.0.0.1:9222` | cdp mode: the attached browser’s CDP URL (usually a tunneled port) |
| `STORAGE_STATE_FILE` | `./auth.json` | storagestate mode: login snapshot exported on Windows, copied here |
| `BROWSER_PROFILE_DIR` | `./.profile` | persistent mode: Chrome profile holding the shared login |
| `HEADLESS` | `true` | `false` only for debugging |
| `MAX_CONCURRENT_PAGES` | `4` | concurrency cap (one Chrome, isolated tabs) |
| `PAGE_TIMEOUT_MS` | `20000` | per-page hard timeout |
| `SEARCH_ENGINE` | `bing` | `bing` (tuned extractor) / `google` / `duck` / `custom` |
| `SEARCH_URL_TEMPLATE` | — | custom URL with `{q}` placeholder, e.g. `https://wiki.internal/search?q={q}` (overrides engine URL) |
| `RESULT_COUNT` | `10` | results per query |
| `SEARXNG_URL` | — | optional public-search fallback (needs outbound internet), e.g. `http://127.0.0.1:8080` |

---

## Adding a custom internal-portal extractor

`extractBing` in `src/tools.ts` is tuned for Bing’s DOM. For an internal portal,
add `extractPortal(page, count)` and select it on the engine name in
`searchWithBrowser`. The generic `extractGeneric` already returns anchor links +
nearby text as a passable fallback for unknown DOMs.

---

## Security & ops notes

- **Bind & expose**: prefer keeping the gateway on the internal network. If you
  bind `0.0.0.0`, **set `ALLOWED_HOSTS`** and use a firewall / network ACL, or
  set `GATEWAY_TOKEN`, or put it behind an SSO reverse proxy.
- **Shared profile = shared identity**: every search is attributed to the shared
  account. Fine for a department service account; review if the target audits
  per-user or has quota.
- **Session renewal**: re-run `npm run login` when SSO expires. Consider a
  weekly cron that emails a reminder, or a health probe that detects a login
  wall (`read_webpage` on a known-login-required URL returns the login page text).
- **Concurrency / scale**: one Chrome with isolated tabs handles a small
  department. Grow to a browser pool (N persistent contexts) if it saturates —
  the `withPage` seam is the only place to change.
- **Headless Chrome on Linux**: `--no-sandbox --disable-dev-shm-usage` are
  already set (container-friendly).

---

## Development & testing

- **Probes** live in `scripts/` and import from `../dist/`, so build first:
  `npm run build`.
  - `scripts/probe-search.mjs "<query>"` — drives the shared browser directly
    (bypasses MCP); validates CDP attach + the Bing extractor.
  - `scripts/probe-mcp.mjs <url> "<query>"` — connects to a **running** gateway
    over Streamable HTTP (the real client path), lists tools, calls `web_search`.
    Start the gateway first: `node --env-file=.env dist/server.js`.
- **Dev mode (`npm start` → tsx)**: under npm 11, `tsx`'s transitive `esbuild`
  postinstall is blocked by `allow-scripts` by default. Approve it once
  (`npm approve-scripts`) or just use the compiled path everywhere:
  `npm run build && node --env-file=.env dist/server.js`.

---

## Status

This is a **reviewable PoC / skeleton** — verified against the v2 MCP SDK API
(`@modelcontextprotocol/server` 2.x, `createMcpHandler` / `createMcpExpressApp`
/ `requireBearerAuth` / `toNodeHandler`) and Playwright’s persistent-context
API. Before production: pin exact dependency versions, add tests, and harden the
auth layer (JWT / introspection instead of a static token) if you expose it
beyond a trusted internal network.

Design context (Mode A/B/C topologies, the cross-OS cookie-encryption gotcha,
SearXNG boundaries) is in the “Headless Linux servers + a Windows PC for login”
section above.