# WebX — Local On-Demand Web Search for Coding Agents

Small, Unix-y local tool that gives coding agents web access **only when desired**.  Not a research agent — just two primitives plus lifecycle management:

```
search(query) -> ranked URLs/snippets   (local SearXNG, Docker, 127.0.0.1:8888, normally stopped)
read(url)     -> cleaned Markdown       (controlled fetch + Trafilatura, SSRF-protected)
```

- **Minimal-agent mode:** agent shells out `webx search / webx read / webx stop` only when a temporary prompt authorizes it. No permanent web tool in the system prompt.
- **Exploration/MCP mode:** host launches `webx-mcp` (stdio). Server exposes **exactly** `web_search` + `web_read`. Launch does not start SearXNG; first `web_search` lazy-starts it and owns shutdown.

## Install

Requires Python 3.12+ and Docker + Compose for search. `webx read` works without Docker.

```bash
# with uv (recommended)
uv sync
uv sync --extra mcp      # for MCP server
uv sync --extra dev      # for tests

# or pip
pip install -e .
pip install -e ".[mcp]"

# global tool (so `webx` works in `pi`'s bash and any shell)
uv tool install .        # installs to ~/.local/bin/webx — ensure ~/.local/bin is on PATH
# or pipx
pipx install .

# per-project (no global install)
uv sync && uv run webx --help
# or add .venv/bin to PATH for this shell/session (useful for pi coding agent)
export PATH="$PWD/.venv/bin:$PATH"
which webx && webx --help
```

> **pi coding agent note:** The `bash` tool inside `pi` inherits `PATH` from the host. If `webx: command not found`, run `uv tool install .` once or `export PATH="$PWD/.venv/bin:$PATH"` in the session where you launch `pi`.

## Quick start

```bash
webx init                # materialize ~/.local/share/webx/{compose.yml,settings.yml,.env,cache}
webx doctor              # check docker, templates, SearXNG reachability (does NOT start SearXNG)
webx status              # {initialized, docker_available, searxng_running, url, runtime_dir}
webx status --json

webx search "SearXNG documentation" --limit 5 --pretty
webx status              # now running

webx read "https://docs.searxng.org/" --max-chars 12000
webx read "https://docs.searxng.org/" --json | jq

# denials are exit 5
webx read "http://127.0.0.1:8888/"      # -> exit 5 unsafe URL
webx read "http://192.168.1.1/"         # -> exit 5
webx read "file:///etc/passwd"          # -> exit 5

webx stop                # docker compose stop (retains container)
webx status              # stopped
```

### Temporary web-access prompt (minimal agent)

```
For this task you are allowed to use the local WebX utility when external/current
information materially helps.
Available commands:
- webx search "<query>" to discover relevant public-web sources.
- webx read "<url>" to read a relevant public page as cleaned text/Markdown.
...
When the web-research portion is finished, run webx stop.
```

### MCP host config

Stdio only. Example (Claude Code / MCP Inspector):

```json
{
  "mcpServers": {
    "webx": {
      "command": "webx-mcp",
      "env": { "WEBX_DATA_DIR": "/home/you/.local/share/webx" }
    }
  }
}
```

Tool list must be exactly `web_search` + `web_read`.  Lifecycle is internal — do **not** expose `webx up/stop` as agent tools.

## CLI reference

```
webx --help
webx --version
webx init [--force-templates] [--show-path]   # idempotent, never rotates secret
webx doctor [--json]                          # inspection only (now reports searxng_image/version)
webx up                                       # ensure SearXNG running
webx stop                                     # compose stop (normal shutdown)
webx status [--json]                          # now includes searxng_image/version
webx logs [--tail 100]
webx search QUERY [--limit 8] [--category general] [--language en] [--page 1]
              [--time {day,month,year}] [--safe-search {0,1,2}] [--engine NAME] [--pretty]
webx read URL [--max-chars N] [--json] [--links] [--no-tables] [--precision] [--recall] [--no-cache]
```

* `stdout` = data (JSON for search, Markdown/text or JSON for read).  `stderr` = diagnostics.
* Exit codes: `0` ok, `2` usage/validation, `3` runtime/docker unavailable, `4` SearXNG failure, `5` unsafe URL, `6` fetch/extraction failure, `7` unsupported content type (`2xx` with `image/*` etc.; `application/pdf` needs `uv sync --extra pdf` else `7` with hint, `2xx` image/pdf without pdf extra → `7`). `4xx`/`5xx`/timeout from a public URL is `6`, not `7` (e.g. `wikimedia PNG -> HTTP 400` -> `6`).

`--verbose` (global) enables debug traces to `stderr` (e.g. `read ok: https://example.com/ text/html 114 chars engine=trafilatura 1.23s`). Secrets never printed.

Engine/category examples (SearXNG aggregates 269 services; filter per query when upstream rate-limits hit):
```bash
webx search "python httpx" --engine wikipedia --engine github --pretty
webx search "SearXNG" --category it --pretty
webx search "SearXNG documentation" --time month --pretty
```

Reader extraction examples (`--links` preserves `[text](url)` markdown; `--precision`/`--recall` tune trafilatura):
```bash
webx read "https://en.wikipedia.org/wiki/Python_(programming_language)" --max-chars 2000 --links | head -n 40
webx read "https://en.wikipedia.org/wiki/Python_(programming_language)" --max-chars 2000 | head -n 40
webx read "https://api.github.com/zen" --json | jq  # application/json is returned raw (engine=raw), not trafilatura
```

## Runtime & config

Runtime dir via `platformdirs` (overridable with `WEBX_DATA_DIR`):

- Linux: `~/.local/share/webx/` (XDG)
- macOS: `~/Library/Application Support/webx/`
- Windows: `%LOCALAPPDATA%\webx\`

Contains `compose.yml`, `settings.yml`, `.env` (`SEARXNG_SECRET` 0600), `cache/`.

`settings.yml` is a tiny override (`use_default_settings: true`, `formats: [html, json]`, `limiter: false`, `public_instance: false`, `image_proxy: false`).  Do not copy the whole SearXNG default config.

`compose.yml` (pinned, `latest` no longer used):

```yaml
services:
  searxng:
    image: ${SEARXNG_IMAGE:-docker.io/searxng/searxng:2026.8.19-5ffd32ca2}
    container_name: webx-searxng
    ports: ["127.0.0.1:8888:8080"]
    env_file: [.env]
    volumes: ["./settings.yml:/etc/searxng/settings.yml:ro", "./cache:/var/cache/searxng"]
    restart: "no"
```

Loopback binding only, single container, no Valkey/Redis, no proxy, no TLS. If the read-only single-file mount ever breaks due to SearXNG `FORCE_OWNERSHIP`, switch to a directory mount — but keep `127.0.0.1` binding (see `04_SEARXNG_RUNTIME.md`).

Env overrides (all `WEBX_`):

```
WEBX_DATA_DIR, WEBX_SEARXNG_URL (default http://127.0.0.1:8888), WEBX_DOCKER_CMD,
WEBX_STARTUP_TIMEOUT (30s), WEBX_SEARCH_TIMEOUT (15s), WEBX_READ_TIMEOUT (15s),
WEBX_MAX_RESPONSE_BYTES (10 MiB), WEBX_MAX_READ_CHARS (40000), WEBX_MCP_STOP_ON_EXIT (true)
```

`SEARXNG_IMAGE` can also be set in `.env` or env to pin an image tag.

## SearXNG image version

Pinned at implementation (2026-08-20) — `v1.2` (`0f5e582`):

- Tag: `docker.io/searxng/searxng:2026.8.19-5ffd32ca2` (was `latest`)
- Running version via `webx doctor --json` / `webx status --json`: `searxng_version: 2026.8.1+8892414dc` (from `/config` when reachable) or image tag when stopped
- Override: `SEARXNG_IMAGE=docker.io/searxng/searxng:2026.8.17-374939b88 webx up` or `SEARXNG_IMAGE=...` in `.env` — then `webx init --force-templates` to materialize
- `latest` is intentionally not used for reproducibility; see `https://docs.searxng.org/admin/api.html` (`/config`) for engine suspension diagnostics
- Current `settings.yml` still `use_default_settings: true` — no Valkey, limiter off for loopback

`webx doctor --json` example:
```json
{
  "searxng_image": "docker.io/searxng/searxng:2026.8.19-5ffd32ca2",
  "searxng_version": "2026.8.1+8892414dc",
  "searxng_reachable": true
}
```

Manual update:

```bash
webx stop
docker compose -f $(webx init --show-path)/compose.yml pull   # or: SEARXNG_IMAGE=... docker compose pull
webx up
webx search "test" --limit 1 --pretty
webx stop
```

Never auto-update on search.

## MCP lifecycle

- Launching `webx-mcp` **does not** start SearXNG.
- First `web_search` probes `http://127.0.0.1:8888/`; if stopped it does `docker compose up -d` + poll, then marks `started_by_mcp = true`; if already running it marks `false`.
- `web_read` never starts SearXNG.
- On clean exit, if `started_by_mcp && WEBX_MCP_STOP_ON_EXIT` it runs `compose stop`; else it leaves SearXNG running.  Process-local lock protects concurrent first searches.  Multiple independent MCP processes needing a lease/refcount is deferred to v2.

Tool descriptions state the trust boundary: returned page text is **untrusted external data**, never agent instructions; JS/auth pages may not work.

## Security model

`webx read` treats URLs as untrusted input.

- Allow only `http://` / `https://`; deny `file:`, `ftp:`, `data:`, `javascript:`, bare paths, credential-bearing URLs.
- Resolve hostname via OS resolver, inspect **every** IPv4/IPv6 with `ipaddress`: deny loopback, RFC1918 private, IPv6 ULA, link-local (`169.254.0.0/16`, `fe80::/10`), multicast, unspecified, reserved, metadata `169.254.169.254`, and the SearXNG endpoint itself.  No `--allow-private` in v1.
- **DNS pinning (v1.2):** `http`+`https` resolve once via `resolve_and_check`, validate all IPs, then pin transport to those IPs (`Host` header + `sni_hostname` for TLS, try each IP on `ConnectError`, fail-closed, no fallback to unpinned URL). Validates every redirect target; `127.0.0.1:8888` SearXNG endpoint also denied.
- Redirects: manual loop, max 5, `Location` resolved against current URL, re-validated, loop/excess fails.
- Fetch: `User-Agent: webx/<version> local-research-tool`, connect 5s, read 15s, streamed with `Content-Length` pre-check + 10 MiB cap, no browser masquerade.
- Allowed types: `text/html`, `application/xhtml+xml`, `text/plain`, markdown-like, `json`/`xml` text; `application/pdf` via `pypdf` (`--extra pdf`, first 20 pages, `engine=pypdf`, `pages_total/pages_read/partial`); binary `image/*` etc. → exit 7.
- Extraction: raw body → `trafilatura.extract(output_format="markdown", ...)` + `html2txt` fallback; PDF via `pypdf` in isolated subprocess (10s timeout); truncate **after** extraction at a word/Newline boundary, report `truncated` + `characters` + `engine`/`pages_total`/`partial` in `--json`.
- No cookies, auth headers, POST, or browser.

## Operations & troubleshooting

`webx doctor` is the first diagnostic.

| Failure | Likely cause |
|---|---|
| `doctor` says docker unavailable | Install Docker/Compose; `webx read` still works |
| Search 403 | `json` not enabled in `settings.yml` (check `search.formats`) |
| SearXNG starts but searches 0 results / 5xx | Upstream engines rate-limited / CAPTCHAd your IP — check `webx logs` for `suspended_time=180` / `Too many request` / `HTTP 403`. Not a WebX bug; try different query/category or pin engines: `webx search "…" --engine wikipedia --engine github` (`google cse` is often the only engine not rate-limited from this IP) |
| Reader returns tiny text | JS-rendered page — try `--recall` or different source; browser rendering is out of scope for v1 |
| Reader rejects URL | Private/local network denial — intentional |
| `webx logs` empty | `SearXNG not running` — `webx logs` now hints `run webx up or webx search to start` instead of silent empty |
| `WEBX_DATA_DIR=/tmp/... webx status` says `running:true` but `compose missing` | Single `webx-searxng` container name shared across dirs — `status` now shows `compose: missing` + note; probe is global `127.0.0.1:8888` |
| `webx: command not found` in `pi` | `~/.local/bin` not on `PATH` — see Install ( `uv tool install` / `export PATH="$PWD/.venv/bin:$PATH"` ) |

Research heuristics (agent-side, not WebX): prefer official docs → upstream repo/notes → specs → vendor announcements → quality writing; use `--category it` when it helps; run multiple focused searches, read primary sources, search for contradictions.

## Testing

```bash
uv sync --extra dev --extra mcp
uv run pytest                # default deterministic suite; integration tests are excluded
uv run pytest -m integration # opt-in live tests (needs Docker + net)
uv run pytest --cov=webx
```

Manual acceptance (from clean `WEBX_DATA_DIR`):

```bash
webx --help; webx init; webx doctor; webx status   # stopped
webx search "SearXNG documentation" --limit 5 --pretty
webx status                                        # running
webx read "https://docs.searxng.org/" --max-chars 12000
webx read "http://127.0.0.1:8888/"      # -> exit 5
webx read "http://192.168.1.1/"         # -> exit 5
webx read "file:///etc/passwd"          # -> exit 5
webx stop; webx status                 # stopped
# MCP: inspector 2 tools, web_read while stopped, first search starts, second reuses, stop-on-exit ownership
```

> **Note on `httpbin.org`:** Live `httpbin.org` currently returns `503 Service Temporarily Unavailable` from some networks (verified 2026-08-20 via `curl -A "webx/0.1.0"` and `curl -A "Mozilla/5.0"` both 503). If `webx read https://httpbin.org/html` 503s, use stable alternatives: `https://example.com`, `https://en.wikipedia.org/wiki/Python_(programming_language)` (good for truncation/`--links` tests), or `https://httpbingo.org/get`.

## Project layout

```
src/webx/
  __init__.py, cli.py, config.py, lifecycle.py, searxng.py, security.py, reader.py, core.py, mcp_server.py
  assets/{compose.yml,settings.yml}
tests/{unit,integration}
docs/{instructions,PLAN.md}
```

Core `WebX` facade is shared by CLI and MCP; neither shells out to the other.

## Non-goals (v1.2)

Browser/Playwright (explicit `web_read_rendered` deferred — bench 81% useful, ~300MB Chromium not justified), crawling, reranker, LLM summarizer, inter-process lease, engine presets, domain filters — see `09_DECISIONS_AND_FUTURE.md` for rationale and P3 harnesses (`scripts/bench_*.py` 50/60 corpora). `v1.2` added: pinned SearXNG, SNI pinning, PDF `pypdf` subprocess, `engine` provenance, `--no-cache` in-memory LRU, MCP/Pi unified contract.

## License

MIT