# Scout — a local web research engine for LLM agents

Scout does web search, scraping, and summarization **outside** the LLM's context
window and hands the agent only compact, cited summaries — so research stops
burning tokens. You get a rich web UI to browse full results; the agent (Claude
Code, or any MCP client) gets cheap summaries over MCP.

**Why:** letting an agent `WebFetch` pages drags tens of thousands of tokens of
raw HTML-turned-text through the context window per lookup. Scout inverts that:
a small local LLM reads the page, and only a few hundred tokens of summary ever
reach the agent. Full text is cached locally and viewable by a human in the UI.

## Architecture

**One core, two faces.** `core/` (search/scrape providers + SQLite cache +
local-LLM summarizer) is consumed by both an **MCP server** (cheap, for the
agent) and a **web UI** (rich, for you). `scrape` over MCP returns a *summary*;
the full page lives in the cache for the web UI.

```
core/            shared library
  providers/     search (DuckDuckGo / Tavily / Brave) + scrape (httpx → Playwright → Firecrawl)
  cache.py       SQLite cache — repeat lookups cost zero tokens and zero requests
  summarize.py   page + brief summarization (local Ollama, cloud Haiku fallback,
                 marked extractive stub as last resort — never raw text)
  projects.py    project profiles: standing context (location, verticals, notes)
                 that biases research queries
  config.py      env-driven config (.env)
mcp_server.py    MCP tools over streamable-HTTP, token-gated  — for the agent
web/             FastAPI + static UI, token-gated              — for you
run.sh / run.ps1 start both services
```

### Request flow
1. Agent calls `research(query)` over MCP.
2. Scout searches the web, picks the top N results, and scrapes each one
   (plain `httpx` first, auto-escalating to headless Chromium for JS-heavy
   pages, optional Firecrawl for anti-bot sites).
3. A local Ollama model summarizes each page; a larger model synthesizes a
   single cited brief across sources.
4. The agent receives one compact markdown brief with numbered sources.
   Everything fetched is cached in SQLite with TTLs, so repeats are free.

### Tools exposed over MCP
- `web_search(query, n)` — ranked title/url/snippet results
- `scrape(url, fresh)` — **summary only** (full text cached for the UI)
- `research(query, project, n)` — search + read top N + synthesized, cited brief
- `save_to_collection(...)` / `list_collections(...)` — project-scoped saved items
- `list_projects()` — available research profiles

### Project profiles
A profile (`data/projects.json`) carries standing context — location, radius,
industry verticals, notes — so a query like "sports facilities within 40 mi"
knows where *here* is and what matters, without the agent restating it each call.

## Tech
Python 3.11+ · FastAPI + Uvicorn · MCP (streamable-HTTP) · SQLite ·
httpx + Playwright (+ optional Firecrawl) · Ollama (`llama3.1:8b` pages,
`llama3.1:70b` briefs) with Claude Haiku cloud fallback · DuckDuckGo search
by default (zero keys), Tavily/Brave optional.

## Quick start
```bash
cp .env.example .env      # set SCOUT_TOKEN; SCOUT_SUMMARIZER=haiku for no-GPU boxes
pip install -r requirements.txt
playwright install --with-deps chromium
./run.sh
# open http://localhost:8770/login?token=<SCOUT_TOKEN>
```
Register with Claude Code:
```bash
claude mcp add --transport http scout http://localhost:8771/mcp \
  --header "X-Scout-Token: <SCOUT_TOKEN>"
```
See **DEPLOY.md** for deploying to an always-on LAN host with a GPU.

## Security
The web UI and MCP endpoint are gated by a shared token (`SCOUT_TOKEN`,
`X-Scout-Token` header / login cookie). All API keys are read from environment
variables (`.env`, gitignored) — nothing is hardcoded. Intended for trusted
LANs; put it behind a reverse proxy with TLS if you expose it further.

## Status
- **v1 (done):** core loop — search → scrape → summarize → cache → MCP + web UI, project profiles.
- **v2 (planned):** local business + contacts (Google Places + contact scraping, lead collections).
- **v3 (planned):** media / stock assets (Pexels/Pixabay/Unsplash/Poly Haven/Sketchfab) + gallery.

## License
MIT — see [LICENSE](LICENSE).