# JedSearch MCP Server

MCP server that gives agents a web search tool and a web reader, with two search
engines fused per query and a tiered page reader. Plain streamable-HTTP MCP,
API-key protected.

## Tools

| Tool | What it does | Measured cost |
|---|---|---|
| `search` | Queries both engines in parallel, merges the rankings, returns title / URL / domain / query-focused snippet / age. No page downloads. | 0.5–1.9 s, 4–6 KB |
| `read` | Readable markdown text of specific URLs, tiered (see below). | 0.5–3 s per page, budgeted |
| `search_read` | `search` plus the full text of the top `max_pages` results in one call. | ~5 s, budgeted |

Split by cost on purpose: an agent that only needs a fact pays ~5 KB, not the
17–64 KB per call that the previous single `web_search` tool always returned.

## Search legs

| Leg | Role | Notes |
|---|---|---|
| Yandex Cloud Search API v2 | Russian intents, `region=ru` | Keyed. `passages` are query-focused excerpts, so snippets carry real answers rather than page titles. |
| SearXNG (self-hosted) | Global coverage, key-free | Engine list is measured on this host — google/duckduckgo/brave are blocked or challenged from here, `google cse`/bing/duckduckgo web/gmx/mwmbl answer. |

Legs run concurrently and are merged with reciprocal-rank fusion
(`1/(60+rank)`, tie-break prefers Yandex). A result with `"agree": 2` was ranked
by both engines — the strongest signal available. Each leg reports
`{ok, ms, n}` and, for SearXNG, `engines_failed`, so a degraded leg is visible
to the caller instead of silently shrinking the result set.

Two Yandex API quirks are handled in `yandex_client.py`: `FORMAT_JSON` is
rejected (only `FORMAT_XML` works, base64 in `rawData`), and `sites[]`,
`pages[]`, `maxpassages`, `window`/`timeInterval` are accepted but silently
ignored — hence `site:` in the query text and post-filtering by `modtime` for
freshness.

## Reader tiers

Per URL, cheapest first, stopping as soon as the text is usable:

1. `httpx` with SSRF validation on every redirect hop + `trafilatura` extraction
   (markdown, keeps the page date).
2. The Yandex cached copy of that page (`yandexwebcache.net`), learned from the
   search leg's `saved-copy-url` — gets past login walls, bot checks and
   geo-blocked origins.
3. Playwright sidecar (`ws://127.0.0.1:3000`) for JS-only sites. Images, fonts,
   media and stylesheets are aborted; service workers blocked.

`render: "always" | "never"` forces tier 3 on or off. Every page reports
`reliable` plus the reason it failed, so an unreadable page is not mistaken for
a missing fact. Pages are cached (TTL 1 h), searches cached for 10 min.

## SSRF

`net_guard.py` validates scheme, host and **every resolved IP** before any
request, for both the HTTP client and the browser's initial navigation, and
re-validates each redirect target. Blocked: RFC1918/loopback/link-local
(including `169.254.169.254`), CGNAT, IPv6 private ranges, IPv4-mapped v6, and
single-label hostnames — which is what stops the reader from reaching a container
service name (`http://<service>:9000`) from inside the docker network.
`SSRF_ALLOW_PRIVATE=true` or `SSRF_ALLOWED_HOSTS=` relax it deliberately.

## Layout

```
config.py         every environment knob in one place
models.py         tool argument models (also the JSON schema agents see)
server.py         MCP wiring: three tools, auth middleware, logging
yandex_client.py  Yandex v2 XML client
searxng_client.py SearXNG JSON client
fusion.py         URL normalisation, dedupe, reciprocal-rank fusion, budgets
reader.py         tiered page reader (httpx → Yandex copy → Playwright)
net_guard.py      SSRF validation + redirect-safe GET
page_cache.py     TTL/LRU caches + counters
entrypoint.sh     Playwright sidecar + server
```

## Configuration

See `.env.example`. Everything is read in `config.py`; nothing is hardcoded.
`SEARXNG_SECRET` is for SearXNG itself (`openssl rand -hex 32`), not for this server.

## Running

Both legs — `compose.yaml` starts SearXNG and jedsearch on one private network:

```bash
cp .env.example .env    # set YANDEX_SEARCH_API_KEY, JEDSEARCH_API_KEY, SEARXNG_SECRET
docker compose up -d    # jedsearch on :3001, SearXNG internal (no published port)
```

Single container, Yandex leg only:

```bash
docker build -t jedsearch .
docker run -d --name jedsearch --env-file .env -p 3001:3001 jedsearch
```

`SEARXNG_URL` empty disables the second leg, and `backend: "yandex"` skips it per
call. `searxng/settings.yml` is the engine list that was measured on one network —
re-measure on yours: `GET /config` lists every engine, `GET /search?q=x&format=json&engines=<name>`
shows whether one actually answers from your egress. The container runs as uid 977
and chowns `/etc/searxng` at startup, so expect that directory to come back owned
by it.

Smoke test — 401 without the key, 200 with it:

```bash
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:3001/mcp \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -H "x-api-key: $JEDSEARCH_API_KEY" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"c","version":"1"}}}'
```

The container runs two processes: the Playwright browser server (port 3000,
internal) and the MCP server (port 3001, `/mcp`). `entrypoint.sh` waits for the
browser and, if it never comes up, logs a warning and starts anyway — tier 3 is
lost, the rest keeps working. Healthcheck is a TCP probe on 3001.

`mcp` is pinned `<2` because the agents' MCP client (and this code) target the
1.x `FastMCP` API; SDK 2.x renamed it to `MCPServer`.