Open Scout

An independent, self-hosted web search engine for people and AI agents.

MIT-licensed. No paid search API. Crawl public websites into your own SQLite index, search with BM25 or a local neural model, and read clean page text through a browser, HTTP API, CLI, or MCP tools.

This is a small-corpus search engine, not Exa's global web index or proprietary models. Its API implements a documented Exa-inspired subset, not full SDK compatibility.

Quick start

Install uv, then:

git clone https://github.com/exux23/open-scout.git
cd open-scout
uv sync --locked
uv run scout serve

Open http://127.0.0.1:8000. Add seed URLs in Crawl, then search the indexed pages. The index starts empty. You can also crawl from the command line:

uv run scout crawl https://www.python.org/ --max-pages 10 --depth 1
uv run scout search "programming language"

Related MCP server: qsearch

Local neural search

uv sync --locked --extra neural
uv run scout reindex

Copy .env.example to .env and set SCOUT_NEURAL=true, then restart the server. First use downloads the open BAAI/bge-small-en-v1.5 ONNX embedding model into data/models. Inference runs locally on CPU. No model API key is needed. New crawls embed automatically when neural mode is enabled. Run reindex with the server stopped after enabling/changing models. The default model is English-focused and independently licensed by its publisher.

API

Interactive OpenAPI documentation: http://127.0.0.1:8000/docs.

curl http://127.0.0.1:8000/search -H "Content-Type: application/json" \
  -d '{"query":"programming language","type":"auto","numResults":5}'
curl http://127.0.0.1:8000/contents -H "Content-Type: application/json" \
  -d '{"urls":["https://www.python.org/"]}'

Responses include source URLs, crawl timestamps, and plain-text highlights. Scores are reciprocal-rank-fusion values, not calibrated relevance probabilities. auto uses keyword retrieval unless neural mode is enabled. Explicit neural/hybrid requests return 503 if disabled or incomplete; they never silently pretend to be neural. Unknown request fields are rejected. Publication-date filtering, generated summaries, live global search, reranking, and Exa's agent/webset APIs are not implemented.

MCP for agents

Start the API, then configure an MCP client with:

{
  "mcpServers": {
    "open-scout": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/open-scout", "run", "scout-mcp"],
      "env": {"SCOUT_API_URL": "http://127.0.0.1:8000"}
    }
  }
}

Tools: search, get_contents, find_similar. On Windows use an absolute Windows path with JSON-escaped backslashes. Set SCOUT_API_KEY in the client environment if enabled. Retrieved text is untrusted web content, not instructions for an agent.

Deployment and data

The CLI binds to loopback by default. For network access, set a random SCOUT_API_KEY of at least 24 characters and supply --host 0.0.0.0. Send the key in x-api-key or enter it in the browser's connection settings. The browser keeps it only in memory. Use HTTPS at a reverse proxy for remote access. This release is a single-user, single-process service: run one API worker. Stop the API before CLI crawls/reindexing. Do not expose an unkeyed server through direct uvicorn calls or port forwarding.

Docker (keyword mode):

docker compose up --build

Copy .env.example to .env and provide a strong SCOUT_API_KEY first. Compose publishes only on host loopback; data persists in the scout-data volume. For neural Docker builds set build arg SCOUT_EXTRAS=neural and SCOUT_NEURAL=true.

SQLite, crawled content, and models live under SCOUT_DATA_DIR (default data/) and are gitignored. Back up the data directory with the service stopped. Crawled pages retain their original authors' rights and are not part of this project's MIT license.

Crawl policy and limits

• Respects robots.txt, crawl delays, noindex and nofollow. Unknown robots failures fail closed.

• Same seed hostnames only, including redirects. Add both apex and www URLs when needed.

• Public HTTP(S), ports 80/443 only. DNS answers are validated and connections pinned to a public IP, including redirects; no localhost, private networks, credentials, or proxies.

• One crawl at a time; at most 200 attempted pages, depth 3, 2 MiB per response, 100,000 extracted characters/page, 2,000 pending links, and a 20-minute loop budget.

• HTML/plain text only. No JavaScript execution, login, paywall bypass, PDF parsing, sitemap ingestion, or automatic recrawl. Re-crawling a URL replaces its stored content.

• Jobs persist; a restart marks unfinished jobs interrupted, rather than claiming completion.

Neural retrieval scans stored chunk vectors with NumPy and uses the best chunk per page. Keyword retrieval uses SQLite FTS5 BM25; hybrid retrieval fuses the two rankings. This design is intended for thousands of pages. It has no ANN index, multi-tenant quotas, distributed crawler, or production throughput benchmark.

Development

uv sync --locked --group dev
uv run pytest
uv run ruff check .
uv run ruff format --check .

See the REA investigation, architecture, and verification for evidence, implementation choices, and test scope.