Skip to main content
Glama
exux23

open-scout

by exux23
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Hybrid

Open Scout

An independent, self-hosted web search engine for people and AI agents.

MIT-licensed. No paid search API. Crawl public websites into your own SQLite index, search with BM25 or a local neural model, and read clean page text through a browser, HTTP API, CLI, or MCP tools.

This is a small-corpus search engine, not Exa's global web index or proprietary models. Its API implements a documented Exa-inspired subset, not full SDK compatibility.

Quick start

Install uv, then:

git clone https://github.com/exux23/open-scout.git
cd open-scout
uv sync --locked
uv run scout serve

Open http://127.0.0.1:8000. Add seed URLs in Crawl, then search the indexed pages. The index starts empty. You can also crawl from the command line:

uv run scout crawl https://www.python.org/ --max-pages 10 --depth 1
uv run scout search "programming language"

Related MCP server: qsearch

Local neural search

uv sync --locked --extra neural
uv run scout reindex

Copy .env.example to .env and set SCOUT_NEURAL=true, then restart the server. First use downloads the open BAAI/bge-small-en-v1.5 ONNX embedding model into data/models. Inference runs locally on CPU. No model API key is needed. New crawls embed automatically when neural mode is enabled. Run reindex with the server stopped after enabling/changing models. The default model is English-focused and independently licensed by its publisher.

API

Interactive OpenAPI documentation: http://127.0.0.1:8000/docs.

curl http://127.0.0.1:8000/search -H "Content-Type: application/json" \
  -d '{"query":"programming language","type":"auto","numResults":5}'
curl http://127.0.0.1:8000/contents -H "Content-Type: application/json" \
  -d '{"urls":["https://www.python.org/"]}'

Endpoint

Behavior

POST /search

query, type (auto/keyword/neural/hybrid), numResults, domain filters, contents options

POST /contents

Read already-indexed urls; per-URL status for missing pages

POST /findSimilar

Related pages for an indexed url, excluding the source

POST /crawl

Start a bounded crawl (urls, maxPages, maxDepth); returns a job ID

GET /crawl, GET /crawl/{id}

Persistent crawl progress and per-page errors

GET /stats

Page counts, domains, embedding coverage

DELETE /documents/{id}

Remove a page and its search/vector entries

GET /health

Public liveness endpoint

Responses include source URLs, crawl timestamps, and plain-text highlights. Scores are reciprocal-rank-fusion values, not calibrated relevance probabilities. auto uses keyword retrieval unless neural mode is enabled. Explicit neural/hybrid requests return 503 if disabled or incomplete; they never silently pretend to be neural. Unknown request fields are rejected. Publication-date filtering, generated summaries, live global search, reranking, and Exa's agent/webset APIs are not implemented.

MCP for agents

Start the API, then configure an MCP client with:

{
  "mcpServers": {
    "open-scout": {
      "command": "uv",
      "args": ["--directory", "/absolute/path/to/open-scout", "run", "scout-mcp"],
      "env": {"SCOUT_API_URL": "http://127.0.0.1:8000"}
    }
  }
}

Tools: search, get_contents, find_similar. On Windows use an absolute Windows path with JSON-escaped backslashes. Set SCOUT_API_KEY in the client environment if enabled. Retrieved text is untrusted web content, not instructions for an agent.

Deployment and data

The CLI binds to loopback by default. For network access, set a random SCOUT_API_KEY of at least 24 characters and supply --host 0.0.0.0. Send the key in x-api-key or enter it in the browser's connection settings. The browser keeps it only in memory. Use HTTPS at a reverse proxy for remote access. This release is a single-user, single-process service: run one API worker. Stop the API before CLI crawls/reindexing. Do not expose an unkeyed server through direct uvicorn calls or port forwarding.

Docker (keyword mode):

docker compose up --build

Copy .env.example to .env and provide a strong SCOUT_API_KEY first. Compose publishes only on host loopback; data persists in the scout-data volume. For neural Docker builds set build arg SCOUT_EXTRAS=neural and SCOUT_NEURAL=true.

SQLite, crawled content, and models live under SCOUT_DATA_DIR (default data/) and are gitignored. Back up the data directory with the service stopped. Crawled pages retain their original authors' rights and are not part of this project's MIT license.

Crawl policy and limits

  • Respects robots.txt, crawl delays, noindex and nofollow. Unknown robots failures fail closed.

  • Same seed hostnames only, including redirects. Add both apex and www URLs when needed.

  • Public HTTP(S), ports 80/443 only. DNS answers are validated and connections pinned to a public IP, including redirects; no localhost, private networks, credentials, or proxies.

  • One crawl at a time; at most 200 attempted pages, depth 3, 2 MiB per response, 100,000 extracted characters/page, 2,000 pending links, and a 20-minute loop budget.

  • HTML/plain text only. No JavaScript execution, login, paywall bypass, PDF parsing, sitemap ingestion, or automatic recrawl. Re-crawling a URL replaces its stored content.

  • Jobs persist; a restart marks unfinished jobs interrupted, rather than claiming completion.

Neural retrieval scans stored chunk vectors with NumPy and uses the best chunk per page. Keyword retrieval uses SQLite FTS5 BM25; hybrid retrieval fuses the two rankings. This design is intended for thousands of pages. It has no ANN index, multi-tenant quotas, distributed crawler, or production throughput benchmark.

Development

uv sync --locked --group dev
uv run pytest
uv run ruff check .
uv run ruff format --check .

See the REA investigation, architecture, and verification for evidence, implementation choices, and test scope.

Available Tools

3 tools
find_similarC

Find pages related to an indexed URL, excluding that source URL.

NameRequiredDescriptionDefault
urlYes
num_resultsNo

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full disclosure burden. It does contribute one genuine behavioral fact (the source URL is excluded from results), but omits what happens when the URL is not indexed, how similarity is determined, and whether results are ranked or paginated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single tight sentence with no filler, and the core purpose plus the key exclusion constraint are front-loaded. It is concise to the point of under-specification, but nothing written is wasteful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-parameter tool with no annotations and no output schema, the description leaves major gaps: return shape, ranking, result count semantics, and failure conditions are all unstated, so an agent cannot reliably predict behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It only hints that 'url' is the indexed source page; 'num_results' (default 10) is never mentioned in the description nor documented in the schema, leaving its meaning and default unexplained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and result type ('find pages related to') plus a scoping detail ('excluding that source URL'). It is distinguishable from siblings like search and get_contents, though it never explicitly contrasts itself with them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'related to an indexed URL' implies the input must already be in the index and that the caller wants neighbors rather than a query-based search, but no explicit when-to-use or when-not guidance is given relative to the sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_contentsA

Read indexed page text with source URLs. Missing URLs must be crawled separately.

NameRequiredDescriptionDefault
urlsYes

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must carry the full behavioral burden. It discloses the key limitation that only indexed URLs are supported and that missing URLs require separate crawling, but it does not describe return format, error behavior, or whether multiple URLs are fetched independently.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences with no filler. The core purpose is front-loaded, followed immediately by the important indexed-URL constraint.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter read tool, the description covers the essential scope constraint. It remains incomplete because there is no output schema and the description does not explain what is returned beyond 'page text with source URLs', nor does it clarify behavior for multiple or invalid URLs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for the single 'urls' parameter. It adds meaningful semantics by indicating the URLs must be indexed and that missing ones are out of scope, but it does not explain the array behavior or whether multiple URLs are accepted in one call.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: 'Read indexed page text'. This is clear enough for an agent to know it fetches page content, but it does not explicitly differentiate its role from the sibling tools search or find_similar.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The sentence 'Missing URLs must be crawled separately' implies the tool only works for already-indexed URLs and that unindexed content requires another mechanism. However, it gives no explicit guidance on when to use this tool instead of search or find_similar.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 3 tool updatesv0.1.0
    • First observedfind_similar
    • First observedget_contents
    • First observedsearch
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3.4/5.0

Scored across 3 tools

Disambiguation4/5

search is query-based, get_contents is URL-based, and find_similar discovers related pages—distinct purposes. There is minor overlap since both search and get_contents return page text, but the descriptions clarify the query vs URL distinction.

Naming Consistency4/5

All names are verb-led and readable, but only get_contents and find_similar follow a verb_noun/adjective pattern; search is a bare verb. This is a minor inconsistency rather than a confusing mix.

Tool Count5/5

Three tools cleanly cover the core read-only operations of a search index: querying, fetching content, and finding related pages. The set is well-scoped with no bloat or missing essentials.

Completeness4/5

The surface supports search, retrieval, and similarity discovery, but lacks tools for browsing or managing the index (e.g., listing indexed pages, stats). Crawling is explicitly out of scope, so the core workflows are covered with only minor gaps.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • wigolo

    KnockOutEZ
    A
    license
    A
    quality
    A
    maintenance
    Provides local-first web intelligence over MCP with tools for search, fetch, crawl, extract, cache, find-similar, research, and autonomous agent loops, requiring no API keys.
    10
    1,093 npm
    5,503
    AGPL 3.0

  • qsearch

    theYahia
    A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to perform web searches with full content retrieval and multi-engine provenance, including trust scoring and local corpus persistence, via MCP integration.
    1 npm
    3
    Apache 2.0

  • CrawlEyes

    waiky-github
    A
    license
    B
    quality
    C
    maintenance
    Enables AI agents to perform web searches and extract full-text content from web pages via standard MCP tools, with fallback search and semantic reranking.
    2
    2
    MIT

  • wigolo

    CraftArt
    A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI agents to perform local-first web search, fetch, crawl, extract, cache, research, and autonomous information gathering through MCP, with no API keys or cloud dependencies.
    1,093 npm
    AGPL 3.0