🔎 Agent Search

A self-hosted, MCP-native web-search backend for AI agents — meta-search, clean extraction, RAG with citations, GitHub project selection, and a Tavily-compatible API. All free, all local.

English · 简体中文

Why?

Built-in WebSearch / WebFetch give you links and snippets. Your agent still has to search → fetch → read → reconcile by hand, and the results are easily polluted by SEO blogs and inflated stars.

Agent Search turns "search primitives" into "search outcomes": aggregate many engines, rank with official-source priority, extract clean text, and answer with chunk-level citations — exposed as one MCP server any agent (Claude Code, Codex, Cursor, …) can call by default. It also does the things the built-ins can't: typed GitHub search, first-party project comparison for tech selection, site mapping, and a Tavily-compatible endpoint.

Related MCP server: searxng-mcp

✨ Features

• Meta-search over 9 engines via SearXNG (Google/Bing/DDG/Brave/Wikipedia/GitHub/StackOverflow/Reddit/News) with URL dedup.

• Smart local reranking — boosts official docs / API / pricing / changelog pages, down-weights SEO content farms, multi-query expansion for doc & pricing intent.

• Robust extraction — trafilatura → Jina Reader → requests fallback chain, ratio-based noise cleaning (keeps tables/code/prices/dates), optional Crawl4AI for JS-heavy pages.

• RAG with citations — search → parallel multi-source fetch → LLM summary with [1][2] references and per-source excerpts (chunk-level evidence); bad body falls back to snippet.

• GitHub, done right — typed repos/code/issues/prs search via the gh CLI, returning license / last-commit / archived / forks for real evaluation, not just stars.

• 🆕 Tech-selection compare — github_compare pulls first-party facts (gh api) + OpenSSF Scorecard health (via the free deps.dev API) and flags archived / stale / no-release / copyleft. Evidence, not verdicts.

• 🆕 Universal solution compare — compare_solutions builds a comparison matrix for any candidates (OSS libs / SaaS / frameworks), not just GitHub repos: GitHub candidates reuse first-party repo_facts; non-GitHub ones get official-page rule extraction (price/version/license). Every cell carries source_url + excerpt + confidence (official/secondary/llm) — traceable, not a black box.

• 🆕 Deep research reports — web_research runs a plan → fan-out → evidence → per-section synthesis pipeline: the LLM drafts an outline (sections + sub-queries), all sub-queries fire concurrently, top sources get fetched & quality-gated into a globally numbered source pool, then each section is written against its own sources with [n] citations, plus a conclusion and a code-assembled reference list. Pick a report type (report_type=): standard (default), detailed (5–6 deeper sections, more sources), comparison (sections organized as comparison dimensions, tables + charts, verdict-style selection advice), or outline (planning-only, returns in seconds — confirm the structure, then run the full report). A gap-reflection round then reviews the draft, re-searches under-evidenced sections from new angles, and rewrites them (new sources keep global numbering; RESEARCH_MAX_ROUNDS). Sections emit Markdown tables, Vega-Lite charts rendered to inline SVG (vector, via the optional vl-convert-python — no Node/browser; falls back to a vega-lite spec block for the consumer to render), and mermaid diagrams for flow/architecture. One call → a multi-section, citation-backed Markdown report (planning degrades gracefully to static fan-out if the LLM output can't be parsed).

• 🆕 Recursive deep crawl — web_crawl follows links 2–6 levels deep (BFS / best-first), returning clean per-page Markdown. Uses Crawl4AI's deep-crawl strategy when installed, else a dependency-free pure-Python BFS. Budget guards (depth/page/time/byte caps) + per-URL SSRF check on every enqueued link. web_map scouts (one level, links only); web_crawl goes deep (many levels, full text).

• Typo-tolerant search — layered fuzzy fallback: consume SearXNG corrections → rapidfuzz edit-distance correction → fuzzy rank bonus → LLM spelling rewrite (all silently degrade if deps absent). A query like skil still finds skill.

• Site mapping — sitemap.xml first, page-link fallback, same-domain dedup.

• Tavily-compatible API — drop-in /tavily/search with stable include_raw_content.

• Caching — SQLite TTL cache; works offline against the cache.

🎬 Demo

Tech-selection comparison — first-party facts + OpenSSF Scorecard health, never just stars:

repo                 stars   license       last commit   scorecard   flags
fastapi/fastapi      99669   MIT           2026-06-25     7.8        -
django/django        87997   BSD-3-Clause  2026-06-25     6.8        [no release]
encode/starlette     12432   BSD-3-Clause  2026-06-19     7.5        -

Search that prefers official docs (content farms down-ranked automatically):

$ agent-search "python asyncio tutorial"
[1] A Conceptual Overview of asyncio — Python 3 docs   https://docs.python.org/3/howto/...
[3] asyncio — Asynchronous I/O — Python 3 docs         https://docs.python.org/3/library/asyncio.html
...

🆚 How it compares

No single OSS project covers this niche — most are end-user apps, single-capability tools, or higher-level orchestrators.

🏗️ Architecture

flowchart TD
    A["Agent / MCP client"] -->|"web_search · web_ask · web_extract · web_map<br/>web_crawl · compare_solutions · github_search · github_compare"| B["Agent Search<br/>FastAPI · MCP · CLI"]
    B --> C["SearXNG · 9 engines<br/>meta-search + rerank"]
    B --> D["trafilatura / Jina / requests<br/>(+ Crawl4AI) · clean extraction"]
    B --> E["LLM (OpenAI-compatible)<br/>RAG with citations"]
    B --> F["gh CLI<br/>typed GitHub search"]
    B --> G["deps.dev + OpenSSF Scorecard<br/>project selection"]

🚀 Quickstart

1. Start SearXNG (and optional FlareSolverr):

cp .env.example .env          # then edit: SEARXNG_SECRET_KEY, (optional) LLM key
docker compose up -d searxng  # add `flaresolverr` only if you need anti-bot handling

2. Install the Python side:

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt              # core
pip install -r requirements-optional.txt     # optional: better extraction (trafilatura)

Or install the CLIs globally with pipx / uv (from a clone):

pipx install .        # → `agent-search`, `agent-search-mcp`, `agent-search-server`

3. Use it — three ways:

# CLI
python search.py "python asyncio tutorial"
python search.py "Anthropic Claude API pricing" --answer

# HTTP API (binds 127.0.0.1 by default)
python server.py          # → http://127.0.0.1:8077/docs

# MCP (Claude Code / Cursor / Codex …)
cp .mcp.json.example .mcp.json   # set the absolute path to this repo

🧰 MCP tools

💡 Coverage depends on your SearXNG instance & region. The bundled config ships some China-friendly engines (e.g. Doubao), so an instance hosted in or tuned for mainland China tends to rank Chinese sources higher and some international/English sources lower (and vice-versa elsewhere). For the widest reach, have your agent run its native WebSearch/WebFetch in parallel and merge — Agent Search for aggregation/RAG/GitHub, native search for extra reach. You can also add/remove engines in searxng/settings.yml.

⚠️ Notes & limitations

• web_ask (RAG) and web_research (report) need an OpenAI-compatible LLM key; everything else (search/extract/map/github) needs no API key. web_research is the heaviest tool (sections+2 LLM calls, ~1–3 min); install the optional vl-convert-python to get inline SVG charts instead of raw Vega-Lite specs.

• Extraction does not render JS by default — install the optional crawl4ai and use deep=True for JS-heavy pages.

• Built for local / trusted use: the HTTP server binds 127.0.0.1 by default and extraction has an SSRF guard (blocks localhost / private / cloud-metadata IPs). Add auth + a reverse proxy before exposing it.

• This is a personal project, maintained best-effort. Issues/PRs welcome but no SLA.

🙏 Acknowledgements

Stands on the shoulders of: SearXNG · trafilatura · Jina Reader · Crawl4AI · FlareSolverr · OpenSSF Scorecard + deps.dev · GitHub CLI · FastAPI · the Model Context Protocol. RAG summaries via any OpenAI-compatible endpoint (e.g. DeepSeek).

📄 License

MIT — do whatever, no warranty. Agent Search orchestrates SearXNG as a separate service (it does not bundle or modify SearXNG's source), so its AGPL does not extend to this project.