research-mcp

An MCP facade that hides a pyramid of search/read providers behind a single streamable-http MCP endpoint and exposes just 4 clean tools with good help texts. An LLM gets a simple "search → read" toolset (or both at once); behind it, several providers are tried, merged, and failed over automatically.

The app does no authentication — it is published through Traefik + basicAuth on the host. The only things it persists are a log file and a small quota store (data/quota.sqlite3: per-instance quota usage and the cumulative billed counters), both under data/ (kept on a volume).

Works with zero keys

make run on an empty .env is enough — no key, no SearXNG, nothing to deploy first. Two instances need no configuration at all: duckduckgo (search — the no-JS html.duckduckgo.com SERP, one request per query, no token handshake) and trafilatura (read — local HTML→Markdown extraction). Together they are the floor: search and read both work out of the box.

Everything else is an upgrade on top of that floor. A self-hosted SearXNG (SEARXNG_URL) is optional; when it is configured it replaces duckduckgo: ours queries the same DuckDuckGo from the same IP, so running both only doubled the hits that earn a block. Every paid vendor lights up the moment its key appears.

The floor is deliberately modest: DuckDuckGo is a scraped SERP, not an API, so it paces itself to one query per 45s (skipping, never waiting, when the slot is taken) and reports a block or a captcha as a failure rather than as an empty result set. That pace is the one measured for this upstream: DuckDuckGo blocks by IP for 7-8 minutes after a burst.

Related MCP server: Web Search MCP

Tools

The tool descriptions cross-reference each other (when to take this one, when to take another), so the model gets a routing graph instead of four independent texts. search_and_read is the default for research; web_search is for links only; read_pages / read_page are for urls that are already known.

The batch tools (read_pages, search_and_read) cap each page at READ_BATCH_MAX_CHARS and mark the cut with [содержимое обрезано на N символах]; read_page always returns the page in full.

Architecture: types + instances

Providers are plugins. We separate:

• type — an implementation class (e.g. the searxng search provider), one per module in src/providers/, registered with @register("type").

• instance — a configured copy of a type with its secrets/URL resolved from named environment variables (multiple instances of one type are allowed, e.g. tavily-1 / tavily-2 / tavily-3 with different keys).

Which instances exist and the order each pipeline tries them is configured in code (src/pipeline_config.py); keys/URLs come from ENV by variable name.

• Search pipeline (searxng → duckduckgo → brave → tavily-search → firecrawl-search → jina-search → xmlriver → parallel → octen → linkup → youcom → serper → exa → openalex → semantic-scholar): enabled instances run concurrently; results are merged and deduplicated by normalized URL (earlier pipeline position wins). Position is therefore a dedup preference, not a cost gate — every enabled instance is called on every query, so cost scales with how many keys are set. When JINA_API_KEY is set (and SEARCH_RERANK_ENABLED is not turned off), the full merged list is then reranked by jina-reranker-v3.5 so the trim to num_results keeps the most relevant hits instead of a blind pipeline-order prefix; any rerank failure falls back to the merge order. searxng, duckduckgo and brave additionally throttle themselves locally (one query per 45s, 45s and 1.1s respectively, each matching a measured upstream limit — DuckDuckGo shares SearXNG's, since both reach the same engine from the same address); when the slot is taken they skip the current search instead of waiting for it. duckduckgo is the only search instance that needs no configuration, which is why it sits directly behind searxng: it is on everywhere, but a deployment that runs SearXNG must keep SearXNG's copy of every shared url. openalex and semantic-scholar (free keys OPENALEX_API_KEY, SEMANTIC_SCHOLAR_API_KEY) mix scholarly papers into every search, linked by their DOI or arXiv url where one exists so the scholarly reader can open them.

• Read pipeline (youtube → supadata → instagram → instagram-profile → scholarly → trafilatura → jina → crawl4ai → tavily-1 → tavily-2 → tavily-3 → firecrawl → brightdata): here the order IS a cost gate — it stops at the first sufficient answer, and brightdata (the anti-bot unlocker) sits last so it only ever sees pages everything cheaper already bounced off. The first five are url-specific readers: each is offered only the urls it recognises, ahead of the probe, and its answer is final (see below). A single probe GET classifies the url. PDFs (Content-Type / .pdf / %PDF magic) are extracted with pypdf — except a PDF with no text layer (a scan), which falls through into the chain so the remote readers get a shot at it with their own parsers, with pypdf's notice kept as the last resort. jina's OCR tier joins that attempt only when the url path ends in .pdf and jina is keyed; for HTML, that same body is handed to trafilatura so the hot path never GETs twice (a probe the host never answered skips trafilatura, and crawl4ai when no connection or TLS handshake completed, instead of waiting out the same timeout again), then the remaining instances are tried in order and the first to return content >= FALLBACK_MIN_CHARS wins — except a bot-challenge page (that reader counts as failed) or a paywall teaser (kept only as the thin fallback, which the answer then flags as a teaser rather than as the whole article). A YouTube video url (youtube.com/watch?v=…, /shorts/…, /live/…, /embed/…, youtu.be/…) is answered before the probe with the video's transcript — title, channel, description and the captions as timestamped paragraphs — fetched from YouTube's own player API (the unofficial ANDROID client, the same one youtube-transcript-api uses). The track in the spoken language wins — a manual one when it exists, the auto-generated one otherwise. The spoken language comes from the audio track marked "original" on an auto-dubbed video (which carries auto-generated captions for every dub), and from the single auto-generated track on a plain one. When that fails — YouTube answers "Sign in to confirm you're not a bot" to IPs it flags — and SUPADATA_API_KEY is set, the paid Supadata API is asked for the same video's existing captions (mode=native: 1 credit per video, charged also when the video has none; no title or description in its answer). When the YouTube readers fail (a video without captions, a fetch that fails), the read fails with their reasons and is not passed to the normal chain above. An Instagram video url (instagram.com/reel/…, /reels/…, /p/…, /tv/…, and /<username>/reel/…, /<username>/p/…) is answered the same way, with a transcript of its audio — author, caption and the speech as timestamped paragraphs. The post comes from one anonymous request to Instagram's web GraphQL API (the logged-out query yt-dlp uses); Groq Whisper (whisper-large-v3-turbo) fetches the post's audio-only DASH track by url and transcribes it, so the server never downloads the media itself. Enabled by GROQ_API_KEY; without it, or when the post is private, login-gated or has no video, the url falls through to the normal chain. An Instagram profile url (instagram.com/<username>/) is answered with the profile's posts, 12 at a time, from the same anonymous GraphQL API (the logged-out profile posts query): per post its date, kind (reel / video / carousel / photo), link and caption; a reel or video link can be read again for its transcript (with GROQ_API_KEY). A last line Next page: …/<username>/?after=<cursor> points to the next 12 — reading that url continues the list. Needs no key. Hashtag pages are not covered: Instagram serves them only to a logged-in account. A DOI url (doi.org/…) or arXiv url (arxiv.org/abs/…, /pdf/…) is answered with the paper's legal open-access full text: arXiv (an arXiv url or an arXiv DOI 10.48550/arXiv.…) serves its own PDF, while any other DOI is resolved through Unpaywall to a repository PDF or Europe PMC's full-text XML (abstract + body) — repository copies first, the publisher landing page never fetched. DOI resolution needs UNPAYWALL_EMAIL (a free, non-secret identifier Unpaywall requires; without it DOI urls go straight to the normal chain); a DOI with no OA copy falls through to the normal chain. arXiv needs no key.

Cross-cutting: one transient retry (5xx / transport errors) with a short backoff; 402 (out of credits) / 429 (rate limited) are treated as a provider failure → next instance (this is what makes tavily-1 → tavily-2 → tavily-3 fail over). Vendors that report an empty balance with some other 4xx — serper answers 400 {"message": "Not enough credits"}, octen 403 "Insufficient balance" — are recognised by the body and logged as out of credits too, so an unpaid account never reads as a broken API. Tavily's non-standard HTTP 432 (a spent monthly plan) is read as out of credits as well. The tavily-search instance walks its keys on those same statuses — TAVILY_1_API_KEY first, then the reader failover keys — so search and the readers together drain all three Tavily accounts before anything stops working.

An instance is enabled only if its required env var(s) are set; otherwise it is skipped with a log line. duckduckgo and trafilatura need no config (duckduckgo steps aside for SEARXNG_URL); jina works keyless (its key is optional). At startup the server requires at least one search and one read instance — a condition always satisfied (by searxng or duckduckgo, and by trafilatura), so the check now only catches a broken pipeline_config.py.

Adding a provider

1. Write src/providers/<type>.py with a class decorated @register("<type>") implementing SearchProvider.search(...) or ReadProvider.read(...).

2. Import the module in src/providers/__init__.py (so the decorator runs).

3. Add an Instance("name", "<type>", api_key_env="YOUR_ENV_NAME") line in src/pipeline_config.py and reference its name in SEARCH_PIPELINE / READ_PIPELINE. Use the ENV var NAME, never a value.

4. Document the env var in .env.example.

Quick start

make install                # create .venv + install dev/test deps
cp .env.example .env        # fill in the keys you have  (shortcut: make env)
make test                   # run tests
make run                    # run the server (streamable-http on MCP_HOST:MCP_PORT, endpoint /mcp)

Configuration

All config comes from ENV / .env (see .env.example). Provider secrets/URLs are read by name in the instance loader, not declared as Settings fields. The non-secret knobs (all defaulted): MCP_HOST, MCP_PORT, LOG_LEVEL, LOG_FILE, LOG_ROTATION, LOG_RETENTION, REQUEST_TIMEOUT, FALLBACK_MIN_CHARS, READ_PAGES_CONCURRENCY, READ_BATCH_MAX_CHARS (per-page content budget of the batch tools — read_pages and search_and_read; beyond it the markdown is cut and marked, read_page is never truncated, 0 disables the cut), RETRIES, SEARCH_RERANK_ENABLED, JINA_TOKEN_BUDGET, ALLOW_PRIVATE_NETWORK (escape hatch for the SSRF guard: true lets read_page fetch private/loopback addresses, which are blocked by default). The read_pages per-call url cap is a fixed 20 (hard constant, matching the tool description) — not configurable.

Provider env vars — duckduckgo and trafilatura take none and are on by default (duckduckgo steps aside when SEARXNG_URL is set: SearXNG queries the same DuckDuckGo, and running both only doubles the hits that get us blocked); everything below is optional on top of them: SEARXNG_URL, BRAVE_API_KEY, SERPER_API_KEY, EXA_API_KEY, JINA_API_KEY (one key enables the jina reader in keyed mode, the jina-search provider and the search reranker; the reader alone also works keyless), CRAWL4AI_URL + CRAWL4AI_TOKEN, TAVILY_1_API_KEY, TAVILY_2_API_KEY, TAVILY_3_API_KEY, FIRECRAWL_API_KEY. The Tavily and Firecrawl keys each enable two instances — the reader and the search provider — because both vendors sell search and extract off one key, out of one shared monthly pool. Search runs on every query and will drain that pool well before the readers do; when a Tavily key runs out, the search fails over to the readers' spare keys (TAVILY_2_API_KEY, TAVILY_3_API_KEY). Firecrawl has one key and no spare: once its pool is empty, both firecrawl-search and the firecrawl reader stop until the plan resets. Keyless until registered: XMLRIVER_USER_ID + XMLRIVER_API_KEY (Yandex SERP), PARALLEL_API_KEY, OCTEN_API_KEY, LINKUP_API_KEY, YOUCOM_API_KEY, and BRIGHTDATA_API_KEY + BRIGHTDATA_ZONE. OPENALEX_API_KEY and SEMANTIC_SCHOLAR_API_KEY (both free) enable scholarly search, UNPAYWALL_EMAIL the DOI half of the scholarly reader. GROQ_API_KEY enables the instagram reader (Instagram transcripts), SUPADATA_API_KEY the supadata reader (paid fallback for YouTube transcripts); the youtube and instagram-profile readers take no key and are always on.

Proxy

Any external instance can be routed through its own SOCKS5/HTTP proxy by setting <INSTANCE>_PROXY — useful for clean egress past IP-based blocks (e.g. Cloudflare in front of Exa). Supported per instance: EXA_PROXY, BRAVE_PROXY, SERPER_PROXY, JINA_PROXY, TAVILY_1_PROXY, TAVILY_2_PROXY, TAVILY_3_PROXY, FIRECRAWL_PROXY, XMLRIVER_PROXY, PARALLEL_PROXY, OCTEN_PROXY, LINKUP_PROXY, YOUCOM_PROXY, BRIGHTDATA_PROXY, YOUTUBE_PROXY, SUPADATA_PROXY, INSTAGRAM_PROXY, SCHOLARLY_PROXY, OPENALEX_PROXY, SEMANTIC_SCHOLAR_PROXY. The instances that do not need clean egress have no proxy: the internal searxng / crawl4ai / trafilatura (which still take their own url/token vars) and the keyless duckduckgo.

YOUTUBE_PROXY routes the youtube reader. Where youtube.com is blocked — or the egress IP is flagged as a bot, which YouTube answers with "Sign in to confirm you're not a bot" — transcripts work only through it.

INSTAGRAM_PROXY routes every request to instagram.com — the instagram transcript reader and the instagram-profile post list alike (the latter needs no GROQ_API_KEY). GROQ_PROXY routes the instagram reader's transcription call to api.groq.com — an option of that instance, not a <INSTANCE>_PROXY: Groq is its second upstream. Where Instagram is blocked, or Groq answers Forbidden for the egress country, that leg works only through its proxy.

The value is passed straight to httpx; socks5://host:port does proxy-side DNS (the target hostname is resolved by the proxy, like curl --socks5-hostname), and socks5h:// / http://host:port are also accepted. Unset → that instance goes direct. The pipeline keeps one pooled httpx client per distinct proxy URL (and one direct client), selected per instance, so proxied and direct providers run side by side. Needs the socks extra (httpx[socks], already pinned).

Logging

Besides stderr (captured by Docker's rotation-capped json-file driver), the server writes a persistent log file to data/research-mcp.log (default; LOG_ROTATION=20 MB, LOG_RETENTION=14 days). It lives on the data/ volume, so it survives container restarts and image updates. The file carries one per-request line per tool call — search (query, which provider instances actually ran, result count, latency) and read (url, the winning provider/tier or pdf, ok, latency), plus a read_pages count=N ok=K summary — making it useful for analyzing how requests distribute across provider tiers. No request bodies or secrets are logged, only urls/queries, provider names, counts, timings.

Deployment

Gitea Actions builds the image and pushes it to the Gitea registry gitea.vvzvlad.xyz/projects/research-mcp (test → build, tags latest + sha). On prod we pull the prebuilt image via docker-compose.yml (behind Traefik + basicAuth, watchtower auto-updates latest; the data/ volume keeps the log file across updates) — we never build on prod.

Layout