# SDW_Search

`SDW_Search` is a provider-neutral Web Search MCP server and CLI. It combines live search, scholarly discovery, verified PDF downloads, URL normalization, multi-provider ranking, secure page fetching, caching, and citation-ready research evidence.

## Current Tools

| MCP tool | Purpose |
| --- | --- |
| `web_search` | Search configured providers and return normalized, deduplicated candidate sources |
| `web_fetch` | Fetch a public URL and extract readable text with redirect and SSRF checks |
| `academic_search` | Discover papers and download verified PDFs into `reference`, with manual fallback reasons |
| `browser_auth` | Start, inspect, or close the browser session used for authenticated PDF downloads |
| `browser_open` | Open a page or download target in the system default browser with its existing SSO session |
| `research` | Run multiple discovery queries, fetch diverse pages, and return evidence for client-side synthesis |
| `search_status` | Show provider configuration, cache state, version, proxy state, and safety limits |

Search snippets are discovery data. `research` only promotes successfully fetched page text into evidence.

## Requirements

- Node.js 20.18.1 or newer
- Google Chrome or Microsoft Edge for browser-authenticated downloads
- Network access, optionally through `SDW_PROXY_URL`
- No API key is required for the default Bing and DuckDuckGo providers

## Install And Run

```powershell
npm install
npm run build
node dist/cli.js status
node dist/cli.js search "OpenAI Responses API web search" --max-results 5
node dist/cli.js academic "metasurface antenna" --journal "IEEE Antennas and Wireless Propagation Letters" --max-papers 3
node dist/cli.js auth "https://ieeexplore.ieee.org" --auth-mode persistent
node dist/cli.js fetch "https://example.com"
node dist/cli.js open "https://ieeexplore.ieee.org/document/11536863/"
node dist/cli.js download "https://publisher.example/paper.pdf"
node dist/cli.js research "Compare two web search APIs" --depth standard
```

Start the stdio MCP server:

```powershell
node dist/mcp.js
```

Generic MCP client configuration:

```json
{
  "mcpServers": {
    "SDW_Search": {
      "command": "node",
      "args": ["F:/absolute/path/to/SDW_Search/dist/mcp.js"],
      "env": {
        "SDW_SEARCH_PROVIDERS": "auto"
      }
    }
  }
}
```

Use an absolute path because MCP clients may start the server from another working directory.

## Providers

`SDW_SEARCH_PROVIDERS` accepts a comma-separated list:

- `duckduckgo`: keyless DuckDuckGo Lite search
- `bing`: keyless Bing HTML search with RSS fallback
- `brave`: official Brave Search API, configured by `SDW_BRAVE_API_KEY`
- `searxng`: one or more comma-separated instances in `SDW_SEARXNG_URL`
- `auto`: enables configured API providers plus Bing and DuckDuckGo

Example:

```powershell
$env:SDW_SEARCH_PROVIDERS = "brave,bing,duckduckgo"
$env:SDW_BRAVE_API_KEY = "..."
node dist/cli.js search "query"
```

## Configuration

| Variable | Default | Meaning |
| --- | --- | --- |
| `SDW_SEARCH_PROVIDERS` | `auto` | Enabled provider list |
| `SDW_BRAVE_API_KEY` | empty | Brave Search API key |
| `SDW_SEARXNG_URL` | empty | Comma-separated SearXNG base URLs |
| `SDW_PROXY_URL` | auto-detected | Explicit HTTP(S) proxy URL |
| `SDW_HTTP_TIMEOUT_MS` | `15000` | Per-request timeout |
| `SDW_MAX_FETCH_BYTES` | `5242880` | Maximum downloaded page size |
| `SDW_MAX_PDF_BYTES` | `52428800` | Maximum PDF download size |
| `SDW_SEARCH_CACHE_TTL_SECONDS` | `900` | Search cache lifetime |
| `SDW_FETCH_CACHE_TTL_SECONDS` | `21600` | Fetched page cache lifetime |
| `SDW_CACHE_PATH` | `~/.sdw-search/cache.sqlite` | SQLite cache path |
| `SDW_ALLOW_PRIVATE_NETWORK` | `false` | Allow fetching private network targets |
| `SDW_BROWSER_HANDOFF_ENABLED` | `true` | Allow explicit handoff of public URLs to the system default browser |
| `SDW_AUTH_MODE` | `persistent` | Authenticated download mode: `none`, `persistent`, `cdp`, or experimental `extension` |
| `SDW_BROWSER_CHANNEL` | `chrome` | Persistent browser channel: `chrome`, `msedge`, or bundled `chromium` |
| `SDW_BROWSER_PROFILE_DIR` | `%LOCALAPPDATA%/SDW_Search/browser-profile` | Dedicated persistent profile; never set this to the daily Chrome/Edge data directory |
| `SDW_BROWSER_BRING_TO_FRONT` | `false` | Keep automated browser navigation in the background; on Windows, persistent mode starts minimized |
| `SDW_BROWSER_HEADLESS` | `false` | Run the dedicated browser headless; keep false for interactive SSO/MFA |
| `SDW_BROWSER_STARTUP_TIMEOUT_MS` | `30000` | Browser launch, connection, and initial navigation timeout |
| `SDW_CDP_ENDPOINT` | empty | Explicit HTTP(S) or WebSocket CDP endpoint required by `cdp` mode |
| `SDW_REFERENCE_DIR` | `<current directory>/reference` | Academic PDF and per-run report directory |

On Windows, the enabled system proxy is detected from Internet Settings when no proxy environment variable is present. `NO_PROXY` is honored. Keep `SDW_ALLOW_PRIVATE_NETWORK=false` for normal MCP use.

## Institutional SSO

The default `persistent` mode uses a dedicated browser profile. The profile starts lazily, so ordinary searches do not open a browser. By default it starts minimized on Windows and automated navigation does not take foreground focus. Click its taskbar icon when manual SSO/MFA is required, or set `SDW_BROWSER_BRING_TO_FRONT=true` to restore foreground behavior.

1. Call `browser_auth` with `action: "start"`, `auth_mode: "persistent"`, and optionally an institution or publisher URL.
2. Complete the school SSO and MFA in the dedicated Chrome/Edge window.
3. Leave the MCP server running and call `academic_search`. Cookies and browser storage persist across later browser and MCP restarts. Authenticated HTTP retries use the profile's cookie jar; browser storage remains available to publisher pages and is never exported.

If `academic_search` discovers that the profile still needs authentication, it opens the paper page and returns `manual_required` with instructions to complete SSO and run the search again. SDW_Search does not return, log, or copy browser cookies. Treat the profile directory as sensitive credential storage and do not sync or commit it.

For CLI use, run `sdw-search auth <url>` first. It keeps the dedicated browser open until Ctrl+C so the interactive login can finish and the profile can close cleanly.

Set `SDW_BROWSER_CHANNEL=msedge` to use Edge. `chromium` requires the matching Playwright browser binary to be installed. Do not point `SDW_BROWSER_PROFILE_DIR` at the normal Chrome or Edge User Data directory; Playwright does not support automating the regular profile this way.

`cdp` is an advanced alternative for a browser that explicitly exposes a debugging endpoint:

```powershell
& "$env:ProgramFiles\Google\Chrome\Application\chrome.exe" `
  --remote-debugging-port=9222 `
  --user-data-dir="$env:LOCALAPPDATA\SDW_Search\cdp-profile"
$env:SDW_AUTH_MODE = "cdp"
$env:SDW_CDP_ENDPOINT = "http://127.0.0.1:9222"
```

Chrome 136 and newer ignore command-line remote-debugging switches for the default Chrome data directory. SDW_Search only connects to the configured endpoint and disconnects without closing the attached browser. It does not yet auto-discover the Chrome 144+ `chrome://inspect/#remote-debugging` consent endpoint.

`extension` is an interactive mode backed by the pinned official Playwright MCP sidecar. Install the [Playwright Extension](https://github.com/microsoft/playwright/tree/main/packages/extension#readme) in the Chrome/Edge profile you want to reuse, then call `browser_auth` with `action: "start"` and `auth_mode: "extension"`. The browser asks you to approve a tab; an optional URL is opened in a new controlled tab. SDW_Search never exports browser cookies.

Extension mode can reuse the daily browser's selected tabs and login state. For supported publishers such as IEEE Xplore, SDW_Search performs a same-origin fetch inside the approved tab, relays the PDF in bounded chunks, verifies it, and writes it into `reference`. PDF bytes are removed from page memory after the relay, and sidecar snapshots/logs use a temporary directory that is deleted when the bridge closes. Unsupported publishers still return `extension_download_path_uncontrolled` with a manual URL. Leave `PLAYWRIGHT_MCP_EXTENSION_TOKEN` empty for interactive approval; if configured, treat it as a secret.

The existing `browser_open` tool remains available as a simple system-browser handoff. It can use the daily browser session, but SDW_Search cannot verify that download or its final path.

## Academic Downloads

`academic_search` combines OpenAlex metadata with the configured web providers, filters obvious non-paper pages, resolves common PDF targets such as IEEE Xplore document pages and arXiv abstracts, and verifies the `%PDF` file signature before writing anything. It first tries public HTTP and only starts the configured browser authentication mode after an authentication or publisher-block response.

For IEEE Xplore browser fallback, persistent/CDP mode visibly opens the article and IEEE PDF wrapper, then captures the raw PDF response before Chrome's built-in PDF Viewer transforms it. Provider discovery, signature/hash verification, and the final `reference` write remain background operations.

The known IEEE persistent flow has an opt-in live regression test. Close any SDW_Search dedicated Chrome window first, then run this from PowerShell:

```powershell
$env:SDW_RUN_IEEE_TESTS="1"
npx tsx --test tests/ieee-persistent.integration.test.ts
Remove-Item Env:SDW_RUN_IEEE_TESTS
```

The test uses the configured persistent profile, visibly opens IEEE Xplore, forces the public-request failure path, captures the raw PDF response, verifies the `%PDF` signature, and writes only to a temporary reference directory that is removed afterward.

The equivalent real CDP regression test expects an authenticated Chrome/Edge instance already running with a non-default `--user-data-dir` and the configured `SDW_CDP_ENDPOINT`:

```powershell
$env:SDW_RUN_IEEE_CDP_TESTS="1"
npx tsx --test tests/ieee-cdp.integration.test.ts
Remove-Item Env:SDW_RUN_IEEE_CDP_TESTS
```

It also forces the public-request failure path, so a pass proves the PDF came through the attached browser session. The test disconnects from CDP without taking ownership of the external browser and removes its temporary download directory.

The Extension relay has a separate opt-in live regression test. Keep Chrome open with the official extension installed, run the following command, and approve the connection once:

```powershell
$env:SDW_RUN_EXTENSION_TESTS="1"
npx tsx --test tests/ieee-extension.integration.test.ts
Remove-Item Env:SDW_RUN_EXTENSION_TESTS
```

The default output is the `reference` folder under the MCP process working directory. Set `SDW_REFERENCE_DIR` to an absolute path in the MCP configuration when the client uses an unpredictable working directory. Filenames are sanitized and include a stable URL hash; existing files are reported as `already_exists` and never overwritten. Every run also creates an `academic-search-*.json` report in the same folder.

Failed papers are returned as `manual_required` with a reason code and `manualUrl`. Typical reasons are `authentication_required`, `publisher_blocked`, `browser_unavailable`, `extension_download_path_uncontrolled`, `not_pdf`, `network_error`, and `file_too_large`. Successful results identify whether the file came from `public_http`, `browser_authenticated`, or an `existing_file`.

## Ranking And Safety

Search processing includes:

1. Tracking-link decoding and URL canonicalization.
2. Cross-provider duplicate fusion using reciprocal-rank fusion.
3. Query relevance scoring.
4. MMR-style diversity selection to reduce same-domain repetition.

Page fetching allows only HTTP(S), rejects credentials and private/reserved destinations, resolves DNS before every request, validates every redirect, limits redirect count and response size, and labels returned page content as untrusted.

## Development

```powershell
npm run check
npm test
npm run build
```

Inspect the MCP server with any MCP inspector by launching `node dist/mcp.js` over stdio.