A local MCP server that gives an agent eleven tools for web, academic, code, and model search plus paginated page/PDF fetching as markdown.
web_search — general SearXNG metasearch (Google/Bing/…) with automatic DuckDuckGo fallback, engine pinning, and day|week|month|year time filtering.
scholar_search — Google Scholar results via SearXNG (no offline fallback).
github_search — GitHub repos, issues, and code (code search needs GITHUB_TOKEN).
huggingface_search — HF models/datasets by name substring, optionally filtered by pipeline task tag.
libgen_search — best-effort book lookup with authors, year, pages, md5, and download links.
arxiv_search — official arXiv API entries (id, title, authors, abstract, URL) with retry and shared 429 cooldown.
alphaxiv_lookup — AI-generated structured overview of an arXiv paper: problem, solution, key insights, results, bibtex, cited papers.
alphaxiv_fulltext — full arXiv paper text as paginated markdown for equations/tables/sections.
fetch_page — JS-rendered pages to markdown via headless Chromium, paginated; arXiv URLs route through alphaxiv first.
fetch_stealth — stealth-patched browser fetch for blocked/challenge pages, with optional Cloudflare Turnstile solving.
fetch_pdf — PDF → markdown via PyMuPDF, paginated, with alphaxiv routing for arXiv PDFs.
Runs locally (127.0.0.1 only, no auth) as a per-session stdio server or a shared HTTP server on :9000.
Provides arXiv search and paper retrieval: queries the arXiv API for entries, routes arXiv paper URLs through alphaxiv for AI overviews or full text, and converts PDFs to markdown, with built-in rate-limit handling.
Provides an automatic DuckDuckGo fallback for general web search when the local SearXNG backend is unavailable.
Provides GitHub search across repositories, issues, and code via the GitHub REST API; code search requires a GITHUB_TOKEN.
Enables general web searches using Google through the local SearXNG metasearch backend, with optional engine pinning.
Provides scholarly search against Google Scholar through SearXNG, returning academic search results without an offline fallback.
Uses a local SearXNG instance as the metasearch backend for general and scholarly web search, with JSON API access, engine selection, and time-range filtering.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@local-web search the web for recent papers on RAG evaluation"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A local web-search and fetch MCP server for Claude Code. It wraps four backends into eleven tools: SearXNG for metasearch, Crawl4AI for JavaScript-rendered pages, PyMuPDF for PDFs, and the arXiv / alphaxiv / GitHub / HuggingFace APIs for citation and code lookups.
Component | Role | Where it runs |
SearXNG | Metasearch (Google, Bing, DuckDuckGo, Scholar, …) over a JSON API |
|
Crawl4AI | JavaScript-rendered page → LLM-ready markdown (Playwright Chromium) | inside the MCP server process |
PyMuPDF 2.x | PDF → markdown | inside the MCP server process |
arXiv / alphaxiv / GitHub / HuggingFace | Dedicated citation and code lookups | remote APIs, no local component |
The MCP project (this folder) is a uv project (pyproject.toml +
uv.lock). SearXNG is a separate git checkout with its own venv, built from
searxng/requirements.txt.
Related MCP server: mcp-web
web_search — web_search(query, num_results=8, engine="auto", searx_engines="", time_range=""). General web search via SearXNG (Google, Bing, …) with an automatic DuckDuckGo (ddgs) fallback. Pin engines with searx_engines="google"; time_range is day|week|month|year.
scholar_search — scholar_search(query, num_results=8). Google Scholar via SearXNG (engines=google scholar); no offline fallback, so SearXNG must be running.
github_search — github_search(query, kind="repo"|"issue"|"code", num_results=8). GitHub REST API. Code search needs the GITHUB_TOKEN env var; unauthenticated the limit is 10 search requests/minute.
huggingface_search — huggingface_search(query, kind="model"|"dataset", num_results=8, pipeline=""). HuggingFace API. query matches name/ID substrings (e.g. "wan"), not full text; pipeline filters by task tag (e.g. "video-classification").
libgen_search — libgen_search(query, num_results=8). Scrapes the index.php?req= results table on libgen.li (best-effort; the site changes and may block). Returns title, authors, publisher, year, language, pages, size, extension, and md5, plus item_url (edition page) and file_url (file page with download mirrors).
arxiv_search — arxiv_search(query, max_results=5). arXiv API entries (id, title, authors, date, abstract, abs URL; the PDF is the same URL with .pdf). Retries up to 3× with backoff on transient timeouts. arXiv allows ≤1 request/3s on one connection; the server spaces arxiv.org requests 5 s apart and, on a 429, pauses the whole domain ~5 min for all agents instead of retrying into the burst counter.
alphaxiv_lookup — alphaxiv_lookup(arxiv_id, lang="en"). AI-generated overview of an arXiv paper (blurb, problem, solution, key insights, results, the machine-readable report, abstract, bibtex, cited papers). Accepts a bare ID (2402.02085) or URL. fetch_page/fetch_pdf of arXiv URLs already route through alphaxiv first, so call this directly when you want the structured overview rather than raw text.
alphaxiv_fulltext — alphaxiv_fulltext(arxiv_id, offset=0, max_chars=20000). Full extracted paper text as markdown, paginated like fetch_page; use it for equations, tables, and section detail the lookup card lacks.
fetch_page — fetch_page(url, offset=0, max_chars=20000). Rendered markdown, paginated: call again with offset += max_chars while offset < total_chars. Browser concurrency is capped by a semaphore (BROWSER_SLOTS, default 4). arXiv paper URLs (abs/pdf) route through alphaxiv first; the result's source field says which answered (alphaxiv | browser | arxiv-direct).
fetch_stealth — fetch_stealth(url, offset=0, max_chars=20000, solve_cloudflare=False). Scrapling StealthyFetcher (stealth-patched headless Chromium: real browser UA, canvas noise, WebRTC blocking, spoofed locale/timezone). Use when fetch_page is blocked or returns a challenge page. solve_cloudflare=True solves Cloudflare Turnstile/interstitials (slow, 1–3 min when it has to). Paginated the same way.
fetch_pdf — fetch_pdf(url, offset=0, max_chars=20000). PDF → markdown via PyMuPDF 2.x, paginated the same way. For arXiv paper PDFs the default route is alphaxiv's extracted text; the direct download happens only when alphaxiv lacks the paper.
Two registration modes expose the same eleven tools:
Name | Transport | Lifecycle |
| stdio (default) | Claude Code spawns |
| HTTP | One long-lived server shared by all sessions (shared rate-limit gate + browser pool). |
Only the shared server and SearXNG need to stay running, and both come up automatically:
At logon — the scheduled task local-web-mcp-stack runs autostart.vbs
→ check-and-start.bat.
At every session start — a SessionStart hook re-runs check-and-start.bat.
Idempotent and headless — check-and-start.bat starts only whatever port
is not already listening, and launches each server as a hidden background
process (no console window) via run-hidden.vbs.
Logs. Each server writes to a file with 5 MB size-based rotation (one prior
generation kept as .1):
logs/searxng.log — SearXNG
logs/mcp.log — the shared MCP server
For debugging you can also launch with visible windows: start-all.bat /
./start-all.sh, or the individual start-searxng.* / start-server.*
(the server launchers set MCP_TRANSPORT=http).
Register both at user scope:
claude mcp add -s user local-web -- <path to uv> run --directory <this folder> python server.py
claude mcp add -s user local-web-shared --transport http http://127.0.0.1:9000/mcp
claude mcp list # both should show ConnectedWith both registered, a session sees the eleven tools twice (once per server
prefix). Use local-web (stdio) for everyday work; the shared server is for
running several sessions or agents concurrently against one browser pool.
# MCP project (this folder)
uv add fastmcp crawl4ai httpx pymupdf beautifulsoup4 ddgs "scrapling[fetchers]" markdownify
uv run python -m playwright install chromium # Crawl4AI + Scrapling browsers
uv run scrapling install # marks Scrapling's browser deps as installed
# SearXNG (searxng/ — a depth-1 clone of github.com/searxng/searxng)
uv venv searxng/.venv --python 3.11
uv pip install --python searxng/.venv -r searxng/requirements.txtSearXNG settings live in searxng/searx/settings/user-settings.yml:
search.formats must include json (the default is html-only, which returns
403 for JSON), and server.limiter is false (the dev runner has no
valkey/redis, so a limiter would fail).
Cloning SearXNG on Windows — the repo contains a file whose name includes a
colon (…/sites-available/searxng.conf:socket), which Windows cannot check
out. Clone with --no-checkout and restore only what is needed:
git checkout HEAD -- searx requirements.txt.
uv run pytest # unit tests: pure helpers + domain gate (fake clock, no network, ~1 s)
uv run python smoke_test.py # live end-to-end, all 11 tools
SKIP_ARXIV=1 uv run python smoke_test.py # skip arXiv while it is rate-limitingUnit tests run in CI (.github/workflows/ci.yml). The smoke test hits the live
HTTP server, so it is local-only — start the stack with check-and-start.bat
first.
The MCP server binds to 127.0.0.1 only, with no authentication — do not
expose it (no 0.0.0.0 / Tailscale binding).
Fetched web content is untrusted data (a prompt-injection vector). Keep side-effecting tools (shell, file writes) out of any agent that browses.
Search and fetch hit the public internet normally; the browser semaphore and Crawl4AI's cache mitigate re-hammering.
Symptom | Cause / fix |
| SearXNG is not running — start it. The result's |
SearXNG JSON returns 403 |
|
|
|
Crawl4AI setup check |
|
Ports in use | SearXNG |
| The arXiv burst counter is active (can last ~15 min). Do not retry in a loop — that refreshes the window. Fetch specific papers via |
Full extracted text of an arXiv paper as markdown from alphaxiv.org. Not subject to arXiv rate limits; use instead of fetch_page/fetch_pdf for arXiv papers when you need section-level detail. Paginated like fetch_page. Accepts a bare ID or an arxiv/alphaxiv URL.
Inputs: arxiv_id (required), offset, max_chars. Returns: object
Look up an arXiv paper on alphaxiv.org: AI-generated structured overview (blurb, problem, solution, key insights, results), the machine-readable report, abstract, bibtex, and cited papers. Accepts a bare ID (2402.02085, version suffix ok) or an arxiv/alphaxiv URL. Use alphaxiv_fulltext when you need equations, tables, or section-level detail.
Inputs: arxiv_id (required), lang. Returns: object
Search arXiv (all fields) via the official API. Returns entries with arxiv_id, title, authors, published (YYYY-MM-DD), abstract, and url (abs page; PDF = same URL with '.pdf'). Retries up to 3 times (15s apart) on transient timeouts; an HTTP 429 rate limit pauses the whole domain for all agents (see DOMAIN_COOLDOWNS) and returns immediately.
Inputs: query (required), max_results. Returns: object
Fetch a web page (JS-rendered via headless Chromium) as markdown.
Paginated: returns the slice [offset, offset+max_chars) plus total_chars;
call again with a larger offset to continue. For arXiv paper URLs the
default route is alphaxiv.org's extracted text (not subject to arXiv rate
limits); the browser is used only when alphaxiv doesn't have the paper.
source says which one answered ("alphaxiv" | "browser" | "arxiv-direct").
On failure returns {"success": false, "error": ...}.
Inputs: url (required), offset, max_chars. Returns: object
Download a PDF and convert it to markdown (PyMuPDF 2.x).
Paginated like fetch_page. For arXiv paper PDFs the default route is
alphaxiv.org's extracted text (not subject to arXiv rate limits); the
direct download is used only when alphaxiv doesn't have the paper.
source says which one answered ("alphaxiv" | "direct" | "arxiv-direct").
On failure returns {"success": false, "error": ...}.
Inputs: url (required), offset, max_chars. Returns: object
Fetch a page with Scrapling's StealthyFetcher (stealth-patched headless Chromium: real browser UA, canvas noise, WebRTC blocking, spoofed locale/timezone) as markdown. Use when fetch_page gets blocked or returns a challenge/error page. solve_cloudflare=True also solves Cloudflare Turnstile/interstitial challenges (slow: 1-3 min when it has to). Paginated like fetch_page. On failure returns {"success": false, "error": ...}.
Inputs: url (required), offset, max_chars, solve_cloudflare. Returns: object
Search GitHub. kind: "repo" or "issue" (code search requires a GITHUB_TOKEN env var, which the server picks up automatically). Unauthenticated rate limit is 10 search requests/minute.
Inputs: query (required), kind, num_results. Returns: object
Search HuggingFace. kind: "model" or "dataset". query matches name/ID
substrings (e.g. "wan", "detection"), NOT full text. pipeline filters by
task tag (models: "text-generation", "video-classification", ...;
datasets: "text-classification", ...).
Inputs: query (required), kind, pipeline, num_results. Returns: object
Search libgen.li (scrapes the index.php?req= results table; best-effort, the site changes and may block datacenter-style requests). Per result: title, authors, publisher, year, language, pages, size, extension, md5, item_url (edition page) and file_url (the file page with download mirrors).
Inputs: query (required), num_results. Returns: object
Google Scholar search (via SearXNG, engines='google scholar'). No offline fallback. Returns {"engine": "searxng-scholar", "results": [...]}. For papers, arxiv_search is usually richer (abstracts + PDF links).
Inputs: query (required), num_results. Returns: object
General web search (Google/Bing/… via SearXNG). engine: "auto" (SearXNG, then DuckDuckGo fallback), "searxng", or "ddg". searx_engines pins SearXNG engines, e.g. "google" or "google, bing". time_range: day|week|month|year. Returns {"engine": str, "results": [{"title", "url", "snippet"}], "error"?: str}.
Inputs: query (required), engine, time_range, num_results, +1 more. Returns: object
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 11 tools
The search tools are cleanly separated by domain (web, scholar, arXiv, GitHub, HF, libgen), and fetch_page/fetch_stealth/fetch_pdf have described fallback roles. There is still minor overlap between fetch_page, alphaxiv_fulltext, and fetch_pdf for retrieving arXiv/PDF content, but the descriptions mostly resolve when to use each.
All names are snake_case and readable, with fetch_* for retrieval and *_search for domain searches. The mix of verb-first (fetch_page), domain-first (arxiv_search), and prefixed tools (alphaxiv_lookup) is a minor inconsistency rather than a confusing one.
Eleven tools are well within a reasonable scope for a web research server. Each search domain and fetch variant covers a distinct source or retrieval mode, so the set does not feel bloated.
The surface covers general web search, academic/arXiv search, code/model/book search, plus page, PDF, stealth, and arXiv full-text fetching. Minor gaps remain around crawling/site mapping or generic file extraction, but core research workflows are covered.
Web search, fetch, extract, and research for AI agents. Markdown output + AI-synthesized answers.
A real browser for your agent: screenshots, page checks, Markdown and PDFs.
Hosted browser for agents: open pages, search the web, extract data, read PDFs and take screenshots.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.