seeker
Optional integration with Ollama for using local LLMs, such as llama3.2:3b, to perform HyDE-style query rewriting and improve search query formulation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@seeker search for a beginner-friendly explanation of how attention works in transformers"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
seeker
Open-source semantic web search for AI agents, with an Exa-compatible API and an MCP server. Runs on a laptop, costs $0.
Most open-source "search for agents" projects wrap SearXNG (i.e. scrape Google/Bing). seeker has its own index: pages from FineWeb-Edu (quality-filtered Common Crawl), whose precomputed bge-small-en-v1.5 embeddings we reuse. So there's no crawling and no embedding cost.
How it works
query ─┬─ BM25 (LanceDB FTS) ───────┐
└─ bge-small vector search ──┴─ Reciprocal Rank Fusion ─ cross-encoder rerank ─ results
(+ highlights, live fetch)Search | What it does |
| BM25 only |
| dense vectors only |
| hybrid (RRF), no reranking |
| hybrid; plus cross-encoder rerank if |
Related MCP server: qsearch
Quickstart
git clone https://github.com/bottercode/seeker && cd seeker
brew install uv # or: curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
uv run hf auth login # free HF account, avoids anonymous rate limits
uv run seeker ingest --n 100000 # ~10 min; ~0.5 GB per 100k pages
uv run seeker search "how do vaccines train the immune system"
uv run seeker serve # http://127.0.0.1:8000/docsRun ingest again (with a different --seed) to grow the index; URLs are deduplicated. Stop seeker serve while ingesting, because indexing cleans up old table versions.
HTTP API (Exa-shaped)
curl -s localhost:8000/search -H 'content-type: application/json' -d '{
"query": "beginner-friendly explanation of how attention works in transformers",
"numResults": 5,
"contents": {"text": {"maxCharacters": 500}, "highlights": true}
}'Endpoints: POST /search, POST /findSimilar, POST /contents, GET /health.
Filters: includeDomains, excludeDomains, includeText, excludeText, startPublishedDate, endPublishedDate.
The official Exa SDK works by changing the base URL:
from exa_py import Exa
exa = Exa(api_key="unused", base_url="http://127.0.0.1:8000")
exa.search("how do solar panels work", num_results=5)MCP (Claude Code, Claude Desktop, …)
claude mcp add seeker -- uv --directory /path/to/seeker run seeker mcpTools: web_search, find_similar, get_contents.
Configuration
Environment variables (SEEKER_*) or .env. See src/seeker/config.py:
Variable | Default | |
|
| drop spam hosts (essay mills, pharma spam, throwaway subdomains) |
|
| boost well-known sites (Tranco top-1M, .gov/.edu); didn't help in eval |
|
|
|
|
|
|
|
| candidates reranked |
| (off) | e.g. |
|
|
|
Evaluation
eval/run_eval.py runs a known-item retrieval test: a local LLM writes a natural search query for each of N random indexed pages (without copying phrases), then we measure where that page ranks.
ollama pull llama3.2:3b
uv run python eval/run_eval.py --n 150eval/relevance.py runs a graded relevance test: 40 everyday queries (eval/queries.txt), with each top-5 result graded 0/1/2 by an LLM judge (Claude Code CLI by default, or --judge ollama). It compares search configurations.
uv run python eval/relevance.py --configs baseline,spam_onlyResults: see Benchmarks.
Benchmarks
Known-item eval, 150 LLM-written queries, 99,973 pages, MacBook Air M2 (8 GB), warm models:
| Recall@1 | Recall@10 | MRR@10 | p50 latency |
| 0.653 | 0.867 | 0.732 | 9 ms |
| 0.760 | 0.933 | 0.824 | 39 ms |
| 0.767 | 0.927 | 0.824 | 45 ms |
Reranker A/B on the same queries (auto with SEEKER_RERANK):
Reranker | MRR@10 | p50 |
none (hybrid) | 0.820 | ~45 ms |
ms-marco-MiniLM-L6 / 256 tok, blend | 0.804 | 230 ms |
ms-marco-MiniLM-L6 / 512 tok, blend | 0.820 | 500 ms |
bge-reranker-base / 512 tok, pure | 0.806 | 3300 ms |
Rerankers didn't pay off here. Note that known-item eval only credits the one source page, so it under-rewards rerankers that surface other relevant pages.
Relevance (LLM-judged, 40 queries × top 5)
Judge: Claude. Same 40 queries throughout.
Index | Config | Relevant@5 | Highly relevant@5 | Top-1 relevant |
100k | no spam filter | 0.53 | 0.11 | 0.65 |
100k | spam filter | 0.63 | 0.205 | 0.85 |
100k | spam filter + popularity prior | 0.625 | 0.14 | 0.85 |
397k | spam filter | 0.765 | 0.26 | 0.875 |
990k | spam filter (default) | 0.82 | 0.30 | 0.875 |
Index size is the biggest lever. Known-item scores drop as the index grows because there are more near-duplicate competitors for each target. That's expected, and why the graded eval is the better measure:
Index |
|
|
|
|
|
100k | 0.732 | 0.824 | 0.824 | 0.927 | 45 ms |
397k | 0.614 | 0.661 | 0.684 | – | 32 ms |
990k | 0.544 | 0.571 | 0.609 | 0.800 | 47 ms |
From 250k pages up, the vector search uses an IVF_HNSW_SQ index (at 397k: 97.5% recall@10 vs exact search, 18 ms vs 142 ms). The 990k index takes 4.7 GB on disk; ingesting ran at about 6 min per 100k pages on an M2 MacBook Air.
Limitations
The index is a snapshot of educational/informational pages, so it's weak on news and very recent events.
FineWeb has no
<title>or publish date. Titles are derived heuristically, andpublishedDateis the Common Crawl crawl date./answerand Exa'scategoryfilter are not implemented yet.Spam filtering is rule-based (spam keywords and random-looking subdomains on domains outside Tranco/.gov/.edu). It catches essay mills and pharma spam, but not every low-quality site.
Data & license
Code: MIT. Host popularity list: Tranco (Le Pochat et al., NDSS 2019), downloaded on first ingest. Index data: FineWeb-Edu, ODC-BY 1.0, also subject to the Common Crawl terms of use.
Available Tools
3 toolsfind_similarC
Find pages similar in meaning to the given URL.
Inputs: url (required), num_results, exclude_source_domain. Returns: result
get_contentsC
Get the full text of a page: from the index if present, otherwise fetched live.
Inputs: url (required), max_characters.
web_searchC
Search the web index. Returns title, url, crawl date, relevant highlights and the start of each page.
Inputs: query (required), num_results, max_characters, exclude_domains, +2 more. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 3 tools
Each tool has a distinct input shape: find_similar takes a URL, web_search takes a query, and get_contents retrieves full text. There is mild conceptual overlap between find_similar and web_search (both surface related pages), but the URL-vs-query distinction is clear enough to avoid misselection.
All three names follow a clean verb_noun snake_case pattern: find_similar, get_contents, web_search. No mixed conventions or vague verbs.
Three tools is lean but each covers a distinct retrieval mode (search, fetch, similarity), so nothing feels redundant. It sits at the thin edge of the range, with no explicit way to submit or index a page.
The surface covers the core read path — search, retrieve full text, and find similar pages — but there is no tool to add or refresh pages in the index, no result pagination control, and no metadata-only lookup. Agents can work around these gaps but cannot manage the index.
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
Web search API and MCP server for developers building AI agents.
Turn any public website into an MCP server for agents to search, read and navigate.
Free web search for AI agents. No API key required. Hosted MCP in active development.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceMCP server for local semantic search over web content, enabling AI agents to ingest, index, and query pages with hybrid retrieval and token budget control.4GPL 2.0
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to perform web searches with full content retrieval and multi-engine provenance, including trust scoring and local corpus persistence, via MCP integration.2 npm4Apache 2.0
- AlicenseBqualityDmaintenanceEnables AI agents to search and recall locally captured browser pages with full source provenance (URL, title, timestamp) over MCP.11Apache 2.0
- AlicenseBqualityBmaintenanceEnables AI agents to perform web searches and extract full-text content from web pages via standard MCP tools, with fallback search and semantic reranking.22MIT