Scout — a local web research engine for LLM agents

Scout does web search, scraping, and summarization outside the LLM's context window and hands the agent only compact, cited summaries — so research stops burning tokens. You get a rich web UI to browse full results; the agent (Claude Code, or any MCP client) gets cheap summaries over MCP.

Why: letting an agent WebFetch pages drags tens of thousands of tokens of raw HTML-turned-text through the context window per lookup. Scout inverts that: a small local LLM reads the page, and only a few hundred tokens of summary ever reach the agent. Full text is cached locally and viewable by a human in the UI.

Architecture

One core, two faces. core/ (search/scrape providers + SQLite cache + local-LLM summarizer) is consumed by both an MCP server (cheap, for the agent) and a web UI (rich, for you). scrape over MCP returns a summary; the full page lives in the cache for the web UI.

core/            shared library
  providers/     search (DuckDuckGo / Tavily / Brave) + scrape (httpx → Playwright → Firecrawl)
  cache.py       SQLite cache — repeat lookups cost zero tokens and zero requests
  summarize.py   page + brief summarization (local Ollama, cloud Haiku fallback,
                 marked extractive stub as last resort — never raw text)
  projects.py    project profiles: standing context (location, verticals, notes)
                 that biases research queries
  config.py      env-driven config (.env)
mcp_server.py    MCP tools over streamable-HTTP, token-gated  — for the agent
web/             FastAPI + static UI, token-gated              — for you
run.sh / run.ps1 start both services

Request flow

1. Agent calls research(query) over MCP.

2. Scout searches the web, picks the top N results, and scrapes each one (plain httpx first, auto-escalating to headless Chromium for JS-heavy pages, optional Firecrawl for anti-bot sites).

3. A local Ollama model summarizes each page; a larger model synthesizes a single cited brief across sources.

4. The agent receives one compact markdown brief with numbered sources. Everything fetched is cached in SQLite with TTLs, so repeats are free.

Tools exposed over MCP

• web_search(query, n) — ranked title/url/snippet results

• scrape(url, fresh) — summary only (full text cached for the UI)

• research(query, project, n) — search + read top N + synthesized, cited brief

• save_to_collection(...) / list_collections(...) — project-scoped saved items

• list_projects() — available research profiles

Project profiles

A profile (data/projects.json) carries standing context — location, radius, industry verticals, notes — so a query like "sports facilities within 40 mi" knows where here is and what matters, without the agent restating it each call.

Related MCP server: minimal-mcp-web-search

Tech

Python 3.11+ · FastAPI + Uvicorn · MCP (streamable-HTTP) · SQLite · httpx + Playwright (+ optional Firecrawl) · Ollama (llama3.1:8b pages, llama3.1:70b briefs) with Claude Haiku cloud fallback · DuckDuckGo search by default (zero keys), Tavily/Brave optional.

Quick start

cp .env.example .env      # set SCOUT_TOKEN; SCOUT_SUMMARIZER=haiku for no-GPU boxes
pip install -r requirements.txt
playwright install --with-deps chromium
./run.sh
# open http://localhost:8770/login?token=<SCOUT_TOKEN>

Register with Claude Code:

claude mcp add --transport http scout http://localhost:8771/mcp \
  --header "X-Scout-Token: <SCOUT_TOKEN>"

See DEPLOY.md for deploying to an always-on LAN host with a GPU.

Security

The web UI and MCP endpoint are gated by a shared token (SCOUT_TOKEN, X-Scout-Token header / login cookie). All API keys are read from environment variables (.env, gitignored) — nothing is hardcoded. Intended for trusted LANs; put it behind a reverse proxy with TLS if you expose it further.

Status

• v1 (done): core loop — search → scrape → summarize → cache → MCP + web UI, project profiles.

• v2 (planned): local business + contacts (Google Places + contact scraping, lead collections).

• v3 (planned): media / stock assets (Pexels/Pixabay/Unsplash/Poly Haven/Sketchfab) + gallery.

License

MIT — see LICENSE.