Skip to main content
Glama
README.md
# Scout

A private search engine your coding agent calls over MCP.

Localhost only · five tools · no API key · the read path never touches the network.

**Status:** v1.3, works end to end. Python 3.12, SQLite + faiss. One loopback service, one thin stdio proxy per agent session.

![Terminal: pi calls the scout/search MCP tool with the query "HTTP semantics", the JSON result appears inline, and pi answers with two RFC links](docs/assets/pi-scout-search.png)

*A 52-document RFC corpus. That is the tool call, the JSON it returns, and the answer — nothing left the machine.*

## What it does

- **`search`** — hybrid BM25 + vector search, 8 results inside 1,200 tokens by default
- **`expand`** — the whole stored chunk behind one result
- **`fetch`** — one page's text by URL
- **`corpus_stats`**, **`job_status`** — what you have, what is running
- **Honest coverage** — every answer carries `thin`, a `gap` (`entity:duckdb`) and a `recommend`, so the agent is told when Scout does not know
- **Grows on its own** — thin queries become crawl intents; a Brave key is optional and only ever finds new domains

## Install

Needs macOS or Linux, Python 3.12 and [uv](https://docs.astral.sh/uv/). Ollama is optional — without it, search is lexical only.

```bash
git clone https://github.com/paradise-runner/scout.git
cd scout
./scripts/bootstrap.sh          # .venv from requirements.lock, ~25 s
```

## First run

```bash
# 1. A corpus. 50 RFCs, one to two minutes of polite fetching.
PYTHONPATH=.:src .venv/bin/python -m bootstrap rfc --db scout.db --from 9110 --limit 50

# 2. The service.
./src/api/scout serve --db scout.db &

# 3. The same JSON your agent gets.
./src/api/scout search "HTTP semantics"

# 4. Register: paste the printed block into ~/.pi/agent/mcp.json, then /reload in pi.
./src/api/scout register
```

Ask pi: **Use scout to find what RFC 9110 says about identifiers.**

## Use

| you want | run |
|---|---|
| service up / down | `./src/api/scout serve --db scout.db` · `Ctrl-C` |
| is it alive | `./src/api/scout serve --health` |
| add one page | `./src/api/scout add https://example.com/docs` |
| crawl a site | `PYTHONPATH=.:src .venv/bin/python -m crawl.cli --db scout.db --domain example.com --page-cap 200` |
| a real corpus | `src/bootstrap/README.md` (Wikipedia dump, RFC set) |

## Limits (v1)

- **Corpus** is only what you bootstrap or crawl. It is not an index of the public web.
- **`coverage.thin: true`** means "few results, small corpus". That is the contract working, not a bug.
- **HTML only.** No PDFs, video, or code repositories.
- **No auth.** Loopback only — do not bind `0.0.0.0`.
- **Vector search** needs a local Ollama and a built index; without them you get lexical answers.
- **Speed.** ~2–130 ms per query on the 50-RFC corpus above (`meta.took_ms`, warm to cold). A 3,400-document corpus can take seconds on a cold disk; numbers are in `bench/REPORT.md`.
- **No UI.** The v1.1 browser page is only in git history: `git checkout 183abb5 -- src/ui`.

## How it runs

```
scout serve                  one process: corpus, index, HTTP on 127.0.0.1:8771
python -m src.api.mcp_proxy  one per agent session: five MCP tools, no corpus open
```

Agent sessions hold nothing but a socket, so any number of them share one warm corpus. Raw HTML is kept forever; SQLite, markdown and vectors rebuild from it offline.

## Development

```bash
./scripts/test-all.sh          # 12 packages, one pytest run each
./scripts/test-all.sh api      # just one
```

Every `src/<package>/README.md` documents that package: what it does, its tests, its repro command.

## Docs

[`PROJECT_VISION.md`](PROJECT_VISION.md) · [`SPEC.md`](SPEC.md) · [`ROADMAP.md`](ROADMAP.md) · [`deploy/`](deploy/README.md) · [`roadmap/status/`](roadmap/status/)

## License

Apache-2.0. See [`LICENSE`](LICENSE).