# research-mcp

An **MCP facade** that hides a pyramid of search/read providers behind a
single streamable-http MCP endpoint and exposes just **4 clean tools** with good
help texts. An LLM gets a simple "search → read" toolset (or both at once);
behind it, several providers are tried, merged, and failed over automatically.

The app does **no authentication** — it is published through Traefik + basicAuth
on the host. The only things it persists are a log file and a small quota store
(`data/quota.sqlite3`: per-instance quota usage and the cumulative billed
counters), both under `data/` (kept on a volume).

## Works with zero keys

`make run` on an empty `.env` is enough — **no key, no SearXNG, nothing to deploy
first.** Two instances need no configuration at all: **`duckduckgo`** (search —
the no-JS `html.duckduckgo.com` SERP, one request per query, no token handshake)
and **`trafilatura`** (read — local HTML→Markdown extraction). Together they are
the floor: search and read both work out of the box.

Everything else is an upgrade on top of that floor. A self-hosted **SearXNG**
(`SEARXNG_URL`) is *optional*; when it is configured it **replaces** `duckduckgo`:
ours queries the same DuckDuckGo from the same IP, so running both only doubled
the hits that earn a block. Every paid vendor lights up the moment its key
appears.

The floor is deliberately modest: DuckDuckGo is a scraped SERP, not an API, so it
paces itself to one query per 45s (skipping, never waiting, when the slot is
taken) and reports a block or a captcha as a failure rather than as an empty
result set. That pace is the one measured for this upstream: DuckDuckGo blocks
by IP for 7-8 minutes after a burst.

## Tools

| Tool | What it does |
|------|--------------|
| `web_search(query, num_results=8, language=None)` | Search across all enabled providers, merge + dedup → ranked list (title, URL, snippet). Search only. |
| `read_page(url)` | One page or PDF → clean Markdown. Auto-detects type, walks the read pipeline (light → heavy) until one succeeds. |
| `read_pages(urls)` | Up to 20 urls concurrently → `{summary, pages}`, where each page is `{url, ok, markdown}` or `{url, ok, error, reason}`. |
| `search_and_read(query, num_results=5, language=None)` | Search **and** read the top hits in one call → `{summary, results}`, each result a hit (`title`, `url`, `snippet`) plus its `markdown` (or `error` + `reason`). Over-fetches candidates and reads them in waves, so failed urls do not eat the quota. |

The tool descriptions cross-reference each other (when to take this one, when to
take another), so the model gets a routing graph instead of four independent
texts. `search_and_read` is the default for research; `web_search` is for links
only; `read_pages` / `read_page` are for urls that are already known.

The batch tools (`read_pages`, `search_and_read`) cap each page at
`READ_BATCH_MAX_CHARS` and mark the cut with `[содержимое обрезано на N
символах]`; `read_page` always returns the page in full.

## Architecture: types + instances

Providers are **plugins**. We separate:

- **type** — an implementation class (e.g. the `searxng` search provider), one
  per module in `src/providers/`, registered with `@register("type")`.
- **instance** — a configured copy of a type with its secrets/URL resolved from
  **named environment variables** (multiple instances of one type are allowed,
  e.g. `tavily-1` / `tavily-2` / `tavily-3` with different keys).

Which instances exist and the order each pipeline tries them is configured **in
code** (`src/pipeline_config.py`); keys/URLs come **from ENV by variable name**.

- **Search pipeline** (`searxng → duckduckgo → brave → tavily-search →
  firecrawl-search → jina-search → xmlriver → parallel → octen → linkup → youcom
  → serper → exa → openalex → semantic-scholar`):
  enabled instances run concurrently; results are merged and deduplicated by
  normalized URL (earlier pipeline position wins). Position is therefore a
  **dedup preference, not a cost gate** — every enabled instance is called on
  every query, so cost scales with how many keys are set. When `JINA_API_KEY` is set (and
  `SEARCH_RERANK_ENABLED` is not turned off), the full merged list is then
  reranked by `jina-reranker-v3.5` so the trim to `num_results` keeps the most
  relevant hits instead of a blind pipeline-order prefix; any rerank failure
  falls back to the merge order.
  `searxng`, `duckduckgo` and `brave` additionally throttle themselves locally
  (one query per 45s, 45s and 1.1s respectively, each matching a measured
  upstream limit — DuckDuckGo shares SearXNG's, since both reach the same engine
  from the same address); when the slot is taken they **skip** the current
  search instead of waiting for it. `duckduckgo` is the only search instance
  that needs no configuration, which is why it sits directly behind
  `searxng`: it is on everywhere, but a deployment that runs SearXNG must
  keep SearXNG's copy of every shared url.
  `openalex` and `semantic-scholar` (free keys `OPENALEX_API_KEY`,
  `SEMANTIC_SCHOLAR_API_KEY`) mix scholarly papers into every search, linked by
  their DOI or arXiv url where one exists so the `scholarly` reader can open them.
- **Read pipeline** (`youtube → supadata → instagram → instagram-profile →
  scholarly → trafilatura → jina → crawl4ai → tavily-1 → tavily-2 → tavily-3 → firecrawl →
  brightdata`):
  here the order IS a cost gate — it stops at the first sufficient answer, and
  `brightdata` (the anti-bot unlocker) sits last so it only ever sees pages
  everything cheaper already bounced off. The first five are url-specific
  readers: each is offered only the urls it recognises, ahead of the probe, and
  its answer is final (see below). A single probe
  GET classifies the url. PDFs (Content-Type /
  `.pdf` / `%PDF` magic) are extracted with pypdf — except a PDF with **no text
  layer** (a scan), which falls through into the chain so the remote readers get
  a shot at it with their own parsers, with pypdf's notice kept as the last
  resort. jina's OCR tier joins that attempt only when the url path ends in
  `.pdf` *and* jina is keyed; for HTML, that same body is
  handed to `trafilatura` so the hot path never GETs twice (a probe the host never
  answered skips `trafilatura`, and `crawl4ai` when no connection or TLS handshake
  completed, instead of waiting out the same timeout again), then the remaining
  instances are tried in order and the first to return content
  `>= FALLBACK_MIN_CHARS` wins — except a bot-challenge page (that reader counts
  as failed) or a paywall teaser (kept only as the thin fallback, which the
  answer then flags as a teaser rather than as the whole article).
  A **YouTube video url** (`youtube.com/watch?v=…`, `/shorts/…`, `/live/…`,
  `/embed/…`, `youtu.be/…`) is answered before the probe with the video's
  **transcript** — title, channel, description and the captions as timestamped
  paragraphs — fetched from YouTube's own player API (the unofficial ANDROID
  client, the same one `youtube-transcript-api` uses). The track in the spoken
  language wins — a manual one when it exists, the auto-generated one otherwise.
  The spoken language comes from the audio track marked "original" on an
  auto-dubbed video (which carries auto-generated captions for every dub), and
  from the single auto-generated track on a plain one.
  When that fails — YouTube answers "Sign in to confirm you're not a bot" to IPs
  it flags — and `SUPADATA_API_KEY` is set, the paid Supadata API is asked for
  the same video's existing captions (`mode=native`: 1 credit per video, charged
  also when the video has none; no title or description in its answer).
  When the YouTube readers fail (a video without captions, a fetch that fails),
  the read fails with their reasons and is not passed to the normal chain above.
  An **Instagram video url** (`instagram.com/reel/…`, `/reels/…`, `/p/…`,
  `/tv/…`, and `/<username>/reel/…`, `/<username>/p/…`) is answered the same way, with a
  **transcript of its audio** — author, caption and the speech as timestamped
  paragraphs. The post comes from one anonymous request to Instagram's web
  GraphQL API (the logged-out query `yt-dlp` uses); Groq Whisper
  (`whisper-large-v3-turbo`) fetches the post's audio-only DASH track by url and
  transcribes it, so the server never downloads the media itself. Enabled by
  `GROQ_API_KEY`; without it, or when the post is private, login-gated or has no
  video, the url falls through to the normal chain.
  An **Instagram profile url** (`instagram.com/<username>/`) is answered with the
  profile's **posts**, 12 at a time, from the same anonymous GraphQL API (the
  logged-out profile posts query): per post its date, kind (reel / video /
  carousel / photo), link and caption; a reel or video link can be read again
  for its transcript (with `GROQ_API_KEY`). A last line `Next page: …/<username>/?after=<cursor>` points to
  the next 12 — reading that url continues the list. Needs no key. Hashtag pages
  are not covered: Instagram serves them only to a logged-in account.
  A **DOI url** (`doi.org/…`) or **arXiv url** (`arxiv.org/abs/…`, `/pdf/…`) is
  answered with the paper's legal **open-access full text**: arXiv (an arXiv url
  or an arXiv DOI `10.48550/arXiv.…`) serves its own PDF, while any other DOI is
  resolved through Unpaywall to a repository PDF or Europe PMC's full-text XML
  (abstract + body) — repository copies first, the publisher landing page never
  fetched. DOI resolution needs `UNPAYWALL_EMAIL` (a free, non-secret identifier
  Unpaywall requires; without it DOI urls go straight to the normal chain); a
  DOI with no OA copy falls through to the normal chain. arXiv needs no key.

Cross-cutting: one transient retry (5xx / transport errors) with a short backoff;
**402 (out of credits) / 429 (rate limited) are treated as a provider failure →
next instance** (this is what makes `tavily-1 → tavily-2 → tavily-3` fail over). Vendors
that report an empty balance with some other 4xx — serper answers `400 {"message":
"Not enough credits"}`, octen `403 "Insufficient balance"` — are recognised by
the body and logged as `out of credits` too, so an unpaid account never reads as
a broken API. Tavily's non-standard HTTP 432 (a spent monthly plan) is read as
`out of credits` as well. The `tavily-search` instance walks its keys on those
same statuses — `TAVILY_1_API_KEY` first, then the reader failover keys — so
search and the readers together drain all three Tavily accounts before anything
stops working.

An instance is **enabled** only if its required env var(s) are set; otherwise it
is skipped with a log line. `duckduckgo` and `trafilatura` need no config
(`duckduckgo` steps aside for `SEARXNG_URL`); `jina` works keyless (its key is
optional). At startup the server requires at least one search and one read
instance — a condition always satisfied (by searxng or duckduckgo, and by
trafilatura), so the check now only catches a broken `pipeline_config.py`.

## Adding a provider

1. Write `src/providers/<type>.py` with a class decorated `@register("<type>")`
   implementing `SearchProvider.search(...)` or `ReadProvider.read(...)`.
2. Import the module in `src/providers/__init__.py` (so the decorator runs).
3. Add an `Instance("name", "<type>", api_key_env="YOUR_ENV_NAME")` line in
   `src/pipeline_config.py` and reference its `name` in `SEARCH_PIPELINE` /
   `READ_PIPELINE`. **Use the ENV var NAME, never a value.**
4. Document the env var in `.env.example`.

## Quick start

```bash
make install                # create .venv + install dev/test deps
cp .env.example .env        # fill in the keys you have  (shortcut: make env)
make test                   # run tests
make run                    # run the server (streamable-http on MCP_HOST:MCP_PORT, endpoint /mcp)
```

## Configuration

All config comes from ENV / `.env` (see `.env.example`). Provider secrets/URLs
are read by **name** in the instance loader, not declared as Settings fields. The
non-secret knobs (all defaulted): `MCP_HOST`, `MCP_PORT`, `LOG_LEVEL`,
`LOG_FILE`, `LOG_ROTATION`, `LOG_RETENTION`, `REQUEST_TIMEOUT`,
`FALLBACK_MIN_CHARS`, `READ_PAGES_CONCURRENCY`, `READ_BATCH_MAX_CHARS` (per-page
content budget of the batch tools — `read_pages` and `search_and_read`; beyond it
the markdown is cut and marked, `read_page` is never truncated, `0` disables the
cut), `RETRIES`,
`SEARCH_RERANK_ENABLED`, `JINA_TOKEN_BUDGET`, `ALLOW_PRIVATE_NETWORK` (escape
hatch for the SSRF guard: `true` lets `read_page` fetch private/loopback
addresses, which are blocked by default). The `read_pages`
per-call url cap is a fixed `20` (hard constant, matching the tool description) —
not configurable.

Provider env vars — `duckduckgo` and `trafilatura` take none and are on by
default (`duckduckgo` steps aside when `SEARXNG_URL` is set: SearXNG queries the
same DuckDuckGo, and running both only doubles the hits that get us blocked);
everything below is optional on top of them: `SEARXNG_URL`, `BRAVE_API_KEY`,
`SERPER_API_KEY`, `EXA_API_KEY`, `JINA_API_KEY`
(one key enables the `jina` reader in keyed mode, the `jina-search` provider and
the search reranker; the reader alone also works keyless), `CRAWL4AI_URL` +
`CRAWL4AI_TOKEN`, `TAVILY_1_API_KEY`, `TAVILY_2_API_KEY`, `TAVILY_3_API_KEY`, `FIRECRAWL_API_KEY`.
The Tavily and Firecrawl keys each enable **two** instances — the reader and the
search provider — because both vendors sell search and extract off one key, out
of one shared monthly pool. Search runs on every query and will drain that pool
well before the readers do; when a Tavily key runs out, the search fails over to
the readers' spare keys (`TAVILY_2_API_KEY`, `TAVILY_3_API_KEY`). Firecrawl has
one key and no spare: once its pool is empty, both `firecrawl-search` and the
`firecrawl` reader stop until the plan resets.
Keyless until registered: `XMLRIVER_USER_ID` + `XMLRIVER_API_KEY` (Yandex SERP),
`PARALLEL_API_KEY`, `OCTEN_API_KEY`, `LINKUP_API_KEY`, `YOUCOM_API_KEY`, and
`BRIGHTDATA_API_KEY` + `BRIGHTDATA_ZONE`. `OPENALEX_API_KEY` and
`SEMANTIC_SCHOLAR_API_KEY` (both free) enable scholarly search, `UNPAYWALL_EMAIL`
the DOI half of the `scholarly` reader. `GROQ_API_KEY` enables the `instagram`
reader (Instagram transcripts), `SUPADATA_API_KEY` the `supadata` reader (paid
fallback for YouTube transcripts); the `youtube` and `instagram-profile` readers
take no key and are always on.

## Proxy

Any external instance can be routed through its own **SOCKS5/HTTP proxy** by
setting `<INSTANCE>_PROXY` — useful for clean egress past IP-based blocks (e.g.
Cloudflare in front of Exa). Supported per instance: `EXA_PROXY`, `BRAVE_PROXY`, `SERPER_PROXY`,
`JINA_PROXY`, `TAVILY_1_PROXY`, `TAVILY_2_PROXY`, `TAVILY_3_PROXY`, `FIRECRAWL_PROXY`,
`XMLRIVER_PROXY`, `PARALLEL_PROXY`, `OCTEN_PROXY`, `LINKUP_PROXY`,
`YOUCOM_PROXY`, `BRIGHTDATA_PROXY`, `YOUTUBE_PROXY`, `SUPADATA_PROXY`,
`INSTAGRAM_PROXY`, `SCHOLARLY_PROXY`, `OPENALEX_PROXY`,
`SEMANTIC_SCHOLAR_PROXY`. The
instances that do not need clean egress
have no proxy: the internal `searxng` / `crawl4ai` / `trafilatura` (which still
take their own url/token vars) and the keyless `duckduckgo`.

`YOUTUBE_PROXY` routes the `youtube` reader. Where youtube.com is blocked — or
the egress IP is flagged as a
bot, which YouTube answers with "Sign in to confirm you're not a bot" — transcripts
work only through it.

`INSTAGRAM_PROXY` routes every request to instagram.com — the `instagram`
transcript reader and the `instagram-profile` post list alike (the latter needs
no `GROQ_API_KEY`). `GROQ_PROXY` routes the `instagram` reader's transcription
call to api.groq.com — an option of that instance, not a `<INSTANCE>_PROXY`:
Groq is its second upstream. Where Instagram is
blocked, or Groq answers `Forbidden` for the egress country, that leg works only
through its proxy.

The value is passed straight to httpx; `socks5://host:port` does **proxy-side
DNS** (the target hostname is resolved by the proxy, like `curl
--socks5-hostname`), and `socks5h://` / `http://host:port` are also accepted.
Unset → that instance goes direct. The pipeline keeps one pooled httpx client
per distinct proxy URL (and one direct client), selected per instance, so
proxied and direct providers run side by side. Needs the `socks` extra
(`httpx[socks]`, already pinned).

## Logging

Besides stderr (captured by Docker's rotation-capped json-file driver), the
server writes a **persistent log file** to `data/research-mcp.log` (default;
`LOG_ROTATION=20 MB`, `LOG_RETENTION=14 days`). It lives on the `data/` volume,
so it survives container restarts and image updates. The file carries one
**per-request line** per tool call — search (`query`, which provider instances
actually ran, result count, latency) and read (`url`, the winning provider/tier
or `pdf`, `ok`, latency), plus a `read_pages count=N ok=K` summary — making it
useful for analyzing how requests distribute across provider tiers. No request
bodies or secrets are logged, only urls/queries, provider names, counts, timings.

## Deployment

Gitea Actions builds the image and pushes it to the Gitea registry
`gitea.vvzvlad.xyz/projects/research-mcp` (`test` → `build`, tags `latest` +
`sha`). On prod we pull the prebuilt image via `docker-compose.yml` (behind
Traefik + basicAuth, watchtower auto-updates `latest`; the `data/` volume keeps
the log file across updates) — we never build on prod.

## Layout

| Path | Purpose |
|------|---------|
| `src/providers/base.py` | Provider interfaces + `SearchResult` / `ProviderError`. |
| `src/providers/registry.py` | `@register` decorator → `REGISTRY`. |
| `src/providers/<type>.py` | One module per provider type. |
| `src/providers/pdf.py` | PDF detection + pypdf text extraction (used by the pipeline). |
| `src/providers/youtube.py` | YouTube video-url detection + transcript fetch (the `youtube` url-specific reader). |
| `src/providers/instagram.py` | Instagram url detection: post → audio transcript via Groq Whisper, profile → its posts with paging (the `instagram` / `instagram_profile` url-specific readers). |
| `src/pipeline_config.py` | In-code instances + pipeline order. |
| `src/pipeline.py` | Instance loader + search/read logic (and `search_and_read`, their composition). |
| `src/rerank.py` | `JinaReranker` — post-merge rerank of search results. |
| `src/settings.py` | Non-secret knobs (pydantic-settings). |
| `src/server.py` | `build_server()` with the 4 `@mcp.tool` definitions. |
| `main.py` | Thin entry point: build server, run streamable-http. |
| `tests/` | pytest suite (network mocked with respx). |