# Private Search Gateway

This project is a deterministic, self-hosted search and page-retrieval backend
for AI frontends. It exposes a fast SearXNG-compatible discovery endpoint, a
bounded integrated search route, and Firecrawl-compatible scrape/search routes.
The gateway starts with the user's exact query and a small healthy set of broad
search engines. It stops when that evidence is already good, or runs one
bounded fallback wave when relevance, entity coverage, or source diversity is
weak. It opens pages only when the selected route asks for it, extracts their
actual contents, reranks evidence locally, and returns source URLs and
page-derived text.

Search results also carry deterministic evidence metadata: an inferred source
type and tier, an authority score, normalized page-declared dates, version
markers, a stable citation ID/URL, and exact character spans for selected
passages. These are ranking and coverage aids, not claims that the gateway has
proved source ownership or verified every claim.

It does not require a paid search API or an internal language model. The gateway
always performs deterministic query planning first. When the first search wave
has weak entity coverage, relevance, or source diversity, an optional
OpenAI-compatible planner can add at most two validated alternatives. The
planner is never used on a strong first pass; invalid, slow, or unavailable
output falls back to deterministic planning. The frontend's model receives the
retrieved evidence and writes the answer. This keeps the service usable by any
frontend that accepts a custom SearXNG or JSON search provider.

## What runs

- `search-gateway`: the only client-facing service, on internal port `8080`
- `searxng`: web, technical, news, image, and research discovery
- `reranker`: local `BAAI/bge-reranker-base` relevance ranking
- `crawl4ai`: JavaScript-aware crawling for difficult pages
- `web-runner`: isolated Crawl4AI and Playwright control over a Unix socket
- `pdf-runner`: network-isolated PDF extraction
- `safe-egress`: blocks private-network and metadata destinations for browsers
- `redis`: response cache, stale-result fallback, and learned per-domain fetch strategy

The stack publishes no host ports. A frontend reaches it over a shared Docker
network at:

```text
http://search-gateway:8080/search
```

Every container attached to that shared network can call the gateway. Use a
dedicated shared network if other unrelated containers should not have access.

## Requirements

- 64-bit Linux VPS
- Docker Engine and Docker Compose v2.24.4 or newer
- About 10 GB free disk for images, Chromium, and the reranker model
- 16 GB RAM recommended for the complete stack

The supplied ceilings total about 10.5 GB, excluding shared memory and normal
Docker overhead. They are limits, not reservations, but leave useful headroom
on a 16 GB host. The first build is slow because it downloads Chromium, the
Crawl4AI image, and the reranker model.

## Clean installation

Create the Docker network once if it does not already exist:

```console
docker network inspect docker-stacks_app-network >/dev/null 2>&1 || \
  docker network create docker-stacks_app-network
```

Clone and configure the project:

```console
git clone https://github.com/ZDOSt/Research-MCP.git
cd Research-MCP
cp .env.example .env
chmod 600 .env
```

Generate three different secrets:

```console
openssl rand -hex 32
openssl rand -hex 32
openssl rand -hex 32
```

Edit `.env` and replace `SEARXNG_SECRET`, `CRAWL4AI_API_TOKEN`, and
`FIRECRAWL_API_KEY` with three different values. Change
`CLIENT_DOCKER_NETWORK` only if your frontend uses a different external Docker
network.

Validate and start the complete stack:

```console
docker compose config --quiet
docker compose up -d --build --wait
docker compose ps
```

No `ports:` entries are needed. Do not add one unless you intentionally want to
expose the gateway outside Docker.

## Verify it

Run a health check from the gateway container:

```console
docker compose exec -T search-gateway python -c \
  "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/healthz').read().decode())"
```

Run a real search from any container on the shared network. Replace
`your-frontend-container` with `anythingllm`, `librechat`, or another container
name:

```console
docker exec your-frontend-container sh -lc \
  "wget -qO- 'http://search-gateway:8080/search?q=how+to+install+docker+compose&format=json' | head -c 1000"
```

The response should contain `results`, source URLs, search snippets, and
diagnostics. This endpoint intentionally does not crawl pages, so it returns
quickly and is suitable for standard SearXNG integrations.

## Frontend setup

Use the following base URL wherever the frontend asks for a SearXNG URL:

```text
http://search-gateway:8080
```

If it asks for the complete search path, use:

```text
http://search-gateway:8080/search
```

AnythingLLM requires the complete discovery path even though its field is
labeled `SearXNG API Base URL`. Configure it as:

```text
http://search-gateway:8080/search
```

Use `http://search-gateway:8080/integrated/search` in that field instead when
you specifically want AnythingLLM's search request to include a bounded crawl.

The standard request is:

```http
GET /search?q=your+question&format=json
```

Supported query parameters include:

- `language=auto`
- `time_range=day|week|month|year`
- `categories=general,it,news,science,images`
- `max_results=1..8`
- `mode=auto|quick|balanced|deep`

When no category is supplied, the gateway infers useful SearXNG categories from
the request. `auto` uses quick mode for simple unanchored lookups and balanced
mode for technical questions, recommendations, and identifiable subjects.
`quick` performs one search wave;
`balanced` conditionally adds one fallback query and a small crawl budget. When
an identifiable subject has no matching candidates, that fallback relaxes the
request to the subject plus preserved proper names or identifiers, then to the
subject alone when the mode permits another variant;
`deep` permits two bounded fallback variants and wider evidence. The discovery
route never crawls.

Broad web search remains primary. Rate-limited or CAPTCHA-blocked engines enter
a temporary cooldown, duplicate URLs are combined with reciprocal-rank and
multi-engine consensus signals, and weak searches may receive keyless
supplemental candidates from Stack Exchange, Wikipedia, Crossref, or GitHub
when the request explicitly asks for repository information. Supplemental
sources never replace broad search or exclude ordinary web results.

To enable the optional planner, set `GATEWAY_PLANNER_BASE_URL`,
`GATEWAY_PLANNER_MODEL`, and, when required, `GATEWAY_PLANNER_API_KEY` in
`.env`. `GATEWAY_PLANNER_MODES=balanced,deep` is the default. The planner is
called only after a weak first search wave, has a three-second default timeout,
and is guarded by entity-preservation checks. Leave the URL and model empty for
fully deterministic operation.

### Integrated search and Firecrawl compatibility

For a frontend that has one combined search/scraper setting, use the bounded
integrated route:

```text
http://search-gateway:8080/integrated/search
```

It performs adaptive discovery, crawls a small initial batch, replaces failed
or blocked pages from the remaining ranked candidates, and returns page-derived
`content`. Its default timeout and crawl budget are intentionally smaller than
`/v1/research`.

Learned browser preferences are not permanent. After the configured re-probe
interval, the gateway retries direct retrieval so temporary blocks do not keep
a recovered domain on the slower browser path.

For LibreChat, Open WebUI, or LobeChat Firecrawl settings, use this API base:

```text
http://search-gateway:8080
```

The gateway implements `POST /v2/scrape` and `POST /v2/search`. Set the
frontend's Firecrawl API key to the same value as `FIRECRAWL_API_KEY` in the
gateway `.env`. The scraper accepts the common Markdown request and returns
`success`, `data.markdown`, and `data.metadata.sourceURL`. The Firecrawl routes
require a Bearer token and use the existing URL validation, direct extraction,
Crawl4AI, Playwright, and PDF isolation controls.

The gateway also implements a Jina-compatible `POST /v1/rerank` endpoint backed
by the stack's local BGE reranker. It uses the same Bearer credential as the
Firecrawl routes. This adapter lets LibreChat include relevant passages from
scraped pages in the model-visible Web Search result without a hosted reranking
service. Frontend requests larger than the local model's 32-text client limit
are split into bounded batches and their scores are merged. The integrated
route uses CPU-friendly 16-text batches and caps passage reranking at 32
page-diverse passages. If the local model is unavailable or exceeds its bounded
deadline, the adapter returns lexical fallback passages instead of an empty
result.

Crawl4AI runs without direct public DNS or Internet access. Its redundant
in-container destination precheck is disabled because it cannot resolve public
targets in that topology. URL syntax is checked by `web-runner`, and every
actual browser connection still passes through the pinning proxy and
`safe-egress`, which resolves the hostname and rejects private, loopback,
link-local, metadata, and otherwise non-public destinations.

Queries containing an explicit HTTP or HTTPS URL bypass SearXNG discovery. The
supplied URL is returned as the deterministic direct discovery result and
then passes through the same authenticated Firecrawl-compatible scraper. This
prevents direct-page requests from depending on whether a search engine happens
to index the supplied URL.

The `/v1/research` and `/integrated/search` responses include an
`evidence_summary` with independent-domain coverage, likely primary-source
coverage, date and extraction coverage, version context, and explicit warnings
when evidence is thin. Each extracted result also includes an `evidence` list
with stable passage IDs, exact source character spans, section labels when
available, and passage scores. Source classification is based on transparent
domain, path, and query-affinity heuristics. It never represents itself as
claim-level verification or proof that two domains are organizationally
independent.

Use these internal Docker URLs:

| Frontend | Search setting | Scraper setting | Reranker setting |
| --- | --- | --- | --- |
| AnythingLLM | `http://search-gateway:8080/search` or `/integrated/search` | Use the integrated route when its separate scraper cannot be changed | Included in integrated search |
| LibreChat | SearXNG base `http://search-gateway:8080` | Firecrawl base `http://search-gateway:8080` | Jina URL `http://search-gateway:8080/v1/rerank` |
| Open WebUI | SearXNG query URL `http://search-gateway:8080/search?q=<query>&format=json` | `FIRECRAWL_API_BASE_URL=http://search-gateway:8080` | Configure separately in the frontend |
| LobeChat | Configure its preferred search provider separately | `FIRECRAWL_URL=http://search-gateway:8080/v2` | Configure separately in the frontend |

For Open WebUI select the Firecrawl web loader and set `FIRECRAWL_API_KEY`.
For LobeChat include Firecrawl in `CRAWLER_IMPLS` and set the same key. For
LibreChat select SearXNG as the search provider and Firecrawl as the scraper;
permit the private `search-gateway` address in its web-search allowlist.

Examples:

```http
POST /v2/scrape
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{"url":"https://example.com","formats":["markdown"]}
```

```http
POST /v2/search
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{"query":"how to install Docker Compose","limit":5}
```

```http
POST /v1/rerank
Authorization: Bearer <FIRECRAWL_API_KEY>
Content-Type: application/json

{
  "model": "jina-reranker-v2-base-multilingual",
  "query": "Docker Engine Ubuntu installation commands",
  "documents": ["first passage", "second passage"],
  "top_n": 5,
  "return_documents": true
}
```

For direct integrations, the full bounded research endpoint is also available:

```http
POST /v1/research
Content-Type: application/json

{
  "query": "What are the recommended settings for an AW3426DW?",
  "mode": "balanced",
  "max_results": 5,
  "language": "auto",
  "categories": []
}
```

## Updating

From the repository directory on the VPS:

```console
git pull --ff-only
docker compose config --quiet
docker compose up -d --build --remove-orphans --wait
docker compose ps
```

You do not need to run `docker compose down` for a normal update. Existing Redis
cache and reranker downloads remain in named volumes.

If the existing `.env` file already defines reranker settings, set
`GATEWAY_RERANKER_TIMEOUT_SECONDS=8`,
`GATEWAY_RERANKER_MAX_BATCH_SIZE=16`, and
`GATEWAY_RERANKER_MAX_DOCUMENTS=32`; values in `.env` override the Compose
defaults.

## Operations

Useful commands:

```console
docker compose ps
docker compose logs --tail=200 search-gateway searxng reranker
docker compose logs --tail=200 crawl4ai web-runner safe-egress pdf-runner
docker compose restart search-gateway
docker compose down
docker compose up -d --wait
```

### Repeatable quality baseline

The opt-in live evaluation suite covers installation guides, errors, hardware
settings, recommendations, version-sensitive questions, direct URLs,
multi-source comparisons, gaming, academic material, and current news. It is
not run automatically and adds no latency to normal gateway requests.

From inside the running gateway container:

```console
docker compose exec -T search-gateway python evaluate_search_quality.py \
  --base-url http://127.0.0.1:8080
```

Run one case while tuning with `--case docker-compose-install`, or retain the
JSON report with `--output /tmp/search-quality-report.json`. See
`evals/README.md` for the measured fields and limitations.

`docker compose down` preserves named volumes. `docker compose down -v` deletes
the cache and downloaded reranker model and should be used only for a deliberate
full reset.

## Limitations

This can approach hosted search tools for documentation, troubleshooting,
product settings, games, current information, and general research, but it
cannot guarantee the same coverage as commercial providers. Keyless engines may
rate-limit datacenter IPs, some sites block all automated browsers, and no
single VPS has the proprietary search indexes used by Google, Brave, or paid
answer engines. The gateway compensates with adaptive engine cooldowns,
source-neutral query fallbacks, consensus fusion, keyless supplemental indexes,
learned extraction strategies, local reranking, bounded browser fallbacks,
caching, and honest partial results rather than inventing an answer.