Private Search Gateway

This project is a deterministic, self-hosted search and page-retrieval backend for AI frontends. It exposes a fast SearXNG-compatible discovery endpoint, a bounded integrated search route, and Firecrawl-compatible scrape/search routes. The gateway starts with the user's exact query and a small healthy set of broad search engines. It stops when that evidence is already good, or runs one bounded fallback wave when relevance, entity coverage, or source diversity is weak. It opens pages only when the selected route asks for it, extracts their actual contents, reranks evidence locally, and returns source URLs and page-derived text.

Search results also carry deterministic evidence metadata: an inferred source type and tier, an authority score, normalized page-declared dates, version markers, a stable citation ID/URL, and exact character spans for selected passages. These are ranking and coverage aids, not claims that the gateway has proved source ownership or verified every claim.

It does not require a paid search API or an internal language model. The gateway always performs deterministic query planning first. When the first search wave has weak entity coverage, relevance, or source diversity, an optional OpenAI-compatible planner can add at most two validated alternatives. The planner is never used on a strong first pass; invalid, slow, or unavailable output falls back to deterministic planning. The frontend's model receives the retrieved evidence and writes the answer. This keeps the service usable by any frontend that accepts a custom SearXNG or JSON search provider.

What runs

• search-gateway: the only client-facing service, on internal port 8080

• searxng: web, technical, news, image, and research discovery

• reranker: local BAAI/bge-reranker-base relevance ranking

• crawl4ai: JavaScript-aware crawling for difficult pages

• web-runner: isolated Crawl4AI and Playwright control over a Unix socket

• pdf-runner: network-isolated PDF extraction

• safe-egress: blocks private-network and metadata destinations for browsers

• redis: response cache, stale-result fallback, and learned per-domain fetch strategy

The stack publishes no host ports. A frontend reaches it over a shared Docker network at:

http://search-gateway:8080/search

Every container attached to that shared network can call the gateway. Use a dedicated shared network if other unrelated containers should not have access.

Related MCP server: deep-research

Requirements

• 64-bit Linux VPS

• Docker Engine and Docker Compose v2.24.4 or newer

• About 10 GB free disk for images, Chromium, and the reranker model

• 16 GB RAM recommended for the complete stack

The supplied ceilings total about 10.5 GB, excluding shared memory and normal Docker overhead. They are limits, not reservations, but leave useful headroom on a 16 GB host. The first build is slow because it downloads Chromium, the Crawl4AI image, and the reranker model.

Clean installation

Create the Docker network once if it does not already exist:

docker network inspect docker-stacks_app-network >/dev/null 2>&1 || \
  docker network create docker-stacks_app-network

Clone and configure the project:

git clone https://github.com/ZDOSt/Research-MCP.git
cd Research-MCP
cp .env.example .env
chmod 600 .env

Generate three different secrets:

openssl rand -hex 32
openssl rand -hex 32
openssl rand -hex 32

Edit .env and replace SEARXNG_SECRET, CRAWL4AI_API_TOKEN, and FIRECRAWL_API_KEY with three different values. Change CLIENT_DOCKER_NETWORK only if your frontend uses a different external Docker network.

Validate and start the complete stack:

docker compose config --quiet
docker compose up -d --build --wait
docker compose ps

No ports: entries are needed. Do not add one unless you intentionally want to expose the gateway outside Docker.

Verify it

Run a health check from the gateway container:

docker compose exec -T search-gateway python -c \
  "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/healthz').read().decode())"

Run a real search from any container on the shared network. Replace your-frontend-container with anythingllm, librechat, or another container name:

docker exec your-frontend-container sh -lc \
  "wget -qO- 'http://search-gateway:8080/search?q=how+to+install+docker+compose&format=json' | head -c 1000"

The response should contain results, source URLs, search snippets, and diagnostics. This endpoint intentionally does not crawl pages, so it returns quickly and is suitable for standard SearXNG integrations.

Frontend setup

Use the following base URL wherever the frontend asks for a SearXNG URL:

http://search-gateway:8080

If it asks for the complete search path, use:

http://search-gateway:8080/search

AnythingLLM requires the complete discovery path even though its field is labeled SearXNG API Base URL. Configure it as:

http://search-gateway:8080/search

Use http://search-gateway:8080/integrated/search in that field instead when you specifically want AnythingLLM's search request to include a bounded crawl.

The standard request is:

GET /search?q=your+question&format=json

Supported query parameters include:

• language=auto

• time_range=day|week|month|year

• categories=general,it,news,science,images

• max_results=1..8

• mode=auto|quick|balanced|deep

When no category is supplied, the gateway infers useful SearXNG categories from the request. auto uses quick mode for simple unanchored lookups and balanced mode for technical questions, recommendations, and identifiable subjects. quick performs one search wave; balanced conditionally adds one fallback query and a small crawl budget. When an identifiable subject has no matching candidates, that fallback relaxes the request to the subject plus preserved proper names or identifiers, then to the subject alone when the mode permits another variant; deep permits two bounded fallback variants and wider evidence. The discovery route never crawls.

Broad web search remains primary. Rate-limited or CAPTCHA-blocked engines enter a temporary cooldown, duplicate URLs are combined with reciprocal-rank and multi-engine consensus signals, and weak searches may receive keyless supplemental candidates from Stack Exchange, Wikipedia, Crossref, or GitHub when the request explicitly asks for repository information. Supplemental sources never replace broad search or exclude ordinary web results.

To enable the optional planner, set GATEWAY_PLANNER_BASE_URL, GATEWAY_PLANNER_MODEL, and, when required, GATEWAY_PLANNER_API_KEY in .env. GATEWAY_PLANNER_MODES=balanced,deep is the default. The planner is called only after a weak first search wave, has a three-second default timeout, and is guarded by entity-preservation checks. Leave the URL and model empty for fully deterministic operation.

Integrated search and Firecrawl compatibility

For a frontend that has one combined search/scraper setting, use the bounded integrated route:

http://search-gateway:8080/integrated/search

It performs adaptive discovery, crawls a small initial batch, replaces failed or blocked pages from the remaining ranked candidates, and returns page-derived content. Its default timeout and crawl budget are intentionally smaller than /v1/research.

Learned browser preferences are not permanent. After the configured re-probe interval, the gateway retries direct retrieval so temporary blocks do not keep a recovered domain on the slower browser path.

For LibreChat, Open WebUI, or LobeChat Firecrawl settings, use this API base:

http://search-gateway:8080

The gateway implements POST /v2/scrape and POST /v2/search. Set the frontend's Firecrawl API key to the same value as FIRECRAWL_API_KEY in the gateway .env. The scraper accepts the common Markdown request and returns success, data.markdown, and data.metadata.sourceURL. The Firecrawl routes require a Bearer token and use the existing URL validation, direct extraction, Crawl4AI, Playwright, and PDF isolation controls.

The gateway also implements a Jina-compatible POST /v1/rerank endpoint backed by the stack's local BGE reranker. It uses the same Bearer credential as the Firecrawl routes. This adapter lets LibreChat include relevant passages from scraped pages in the model-visible Web Search result without a hosted reranking service. Frontend requests larger than the local model's 32-text client limit are split into bounded batches and their scores are merged. The integrated route uses CPU-friendly 16-text batches and caps passage reranking at 32 page-diverse passages. If the local model is unavailable or exceeds its bounded deadline, the adapter returns lexical fallback passages instead of an empty result.

Crawl4AI runs without direct public DNS or Internet access. Its redundant in-container destination precheck is disabled because it cannot resolve public targets in that topology. URL syntax is checked by web-runner, and every actual browser connection still passes through the pinning proxy and safe-egress, which resolves the hostname and rejects private, loopback, link-local, metadata, and otherwise non-public destinations.

Queries containing an explicit HTTP or HTTPS URL bypass SearXNG discovery. The supplied URL is returned as the deterministic direct discovery result and then passes through the same authenticated Firecrawl-compatible scraper. This prevents direct-page requests from depending on whether a search engine happens to index the supplied URL.

The /v1/research and /integrated/search responses include an evidence_summary with independent-domain coverage, likely primary-source coverage, date and extraction coverage, version context, and explicit warnings when evidence is thin. Each extracted result also includes an evidence list with stable passage IDs, exact source character spans, section labels when available, and passage scores. Source classification is based on transparent domain, path, and query-affinity heuristics. It never represents itself as claim-level verification or proof that two domains are organizationally independent.

Use these internal Docker URLs:

For Open WebUI select the Firecrawl web loader and set FIRECRAWL_API_KEY. For LobeChat include Firecrawl in CRAWLER_IMPLS and set the same key. For LibreChat select SearXNG as the search provider and Firecrawl as the scraper; permit the private search-gateway address in its web-search allowlist.

Examples:

POST /v2/scrape
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{"url":"https://example.com","formats":["markdown"]}

POST /v2/search
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{"query":"how to install Docker Compose","limit":5}

POST /v1/rerank
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{
  "model": "jina-reranker-v2-base-multilingual",
  "query": "Docker Engine Ubuntu installation commands",
  "documents": ["first passage", "second passage"],
  "top_n": 5,
  "return_documents": true
}

For direct integrations, the full bounded research endpoint is also available:

POST /v1/research
Content-Type: application/json

{
  "query": "What are the recommended settings for an AW3426DW?",
  "mode": "balanced",
  "max_results": 5,
  "language": "auto",
  "categories": []
}

Updating

From the repository directory on the VPS:

git pull --ff-only
docker compose config --quiet
docker compose up -d --build --remove-orphans --wait
docker compose ps

You do not need to run docker compose down for a normal update. Existing Redis cache and reranker downloads remain in named volumes.

If the existing .env file already defines reranker settings, set GATEWAY_RERANKER_TIMEOUT_SECONDS=8, GATEWAY_RERANKER_MAX_BATCH_SIZE=16, and GATEWAY_RERANKER_MAX_DOCUMENTS=32; values in .env override the Compose defaults.

Operations

Useful commands:

docker compose ps
docker compose logs --tail=200 search-gateway searxng reranker
docker compose logs --tail=200 crawl4ai web-runner safe-egress pdf-runner
docker compose restart search-gateway
docker compose down
docker compose up -d --wait

Repeatable quality baseline

The opt-in live evaluation suite covers installation guides, errors, hardware settings, recommendations, version-sensitive questions, direct URLs, multi-source comparisons, gaming, academic material, and current news. It is not run automatically and adds no latency to normal gateway requests.

From inside the running gateway container:

docker compose exec -T search-gateway python evaluate_search_quality.py \
  --base-url http://127.0.0.1:8080

Run one case while tuning with --case docker-compose-install, or retain the JSON report with --output /tmp/search-quality-report.json. See evals/README.md for the measured fields and limitations.

docker compose down preserves named volumes. docker compose down -v deletes the cache and downloaded reranker model and should be used only for a deliberate full reset.

Limitations

This can approach hosted search tools for documentation, troubleshooting, product settings, games, current information, and general research, but it cannot guarantee the same coverage as commercial providers. Keyless engines may rate-limit datacenter IPs, some sites block all automated browsers, and no single VPS has the proprietary search indexes used by Google, Brave, or paid answer engines. The gateway compensates with adaptive engine cooldowns, source-neutral query fallbacks, consensus fusion, keyless supplemental indexes, learned extraction strategies, local reranking, bounded browser fallbacks, caching, and honest partial results rather than inventing an answer.