CrawlEyes
Summary: This server enables AI agents to search the web and extract clean webpage content as Markdown.
Web search with
search(query, limit?): uses SearXNG first, automatically falls back to Tavily keyless, and returns JSON results with title/url/description (default 5 results).Webpage extraction with
extract(url, max_words?): fetches the main page content as Markdown, auto-denoises navigation/ads, retries on failure, and returns JSON with title/markdown/length (default 8000 words).
Provides web search through a self-hosted SearXNG meta-search instance, with automatic fallback to Tavily keyless API, shared caching, and optional semantic reranking of results.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@CrawlEyes Find the latest MCP Python SDK docs and extract the top result"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
CrawlEyes โ Web Scraping & Search Toolkit for AI Agents
English | ไธญๆ
CrawlEyes gives AI agents reliable full-text extraction (web_extract) and robust search (web_search) backends โ the "eyes" that let agents see and read the web. Built and tested against Hermes Agent.
Also ships as a standard MCP server, so any MCP client (Claude Desktop, Cursor, other agents) can reuse the same search + extraction capabilities.

Why CrawlEyes?
Most agent toolkits cover one slice of the pipeline. CrawlEyes is the rare all-in-one that you can actually run behind the Great Firewall without external accounts.
Typical agent toolkit | CrawlEyes | |
๐ Search | API key required, often blocked in CN | โ SearXNG (self-hosted) + Tavily keyless fallback โ zero config, zero key |
๐ Extraction | Separate scraper, or Firecrawl SaaS | โ Built-in Crawl4AI full-text extract, ~89% noise removal |
๐ง Semantic rerank | Rarely included | โ Local fastembed rerank โ no torch, ~50MB model |
๐ MCP server | Often missing | โ
Standard MCP tools ( |
๐ China-friendly | Mostly English/GFW-blocked | โ Tested on a real mainland China server (baidu + yandex) |
Zero API keys. Zero external accounts. One command. CrawlEyes is the only toolkit in this space that combines search + extraction + semantic reranking + MCP in a single, China-friendly, self-hosted package.
Related MCP server: qsearch
Features
Capability | Where | Why it matters |
Full-text extraction |
| Headless-browser scraping โ clean Markdown; handles ~80% of JS/dynamic/UA-blocked pages |
Content denoising (P1) |
| Prunes nav/ads/comments via Crawl4AI's |
Retry with backoff (P3) |
| Exponential backoff (1s/2s/4s) on transient failures |
Browser session reuse (P4) |
| Reuses the browser context across scrapes in one process โ no cold-start per URL |
Keyword-focused extraction |
| Keeps only paragraphs relevant to a keyword (experimental โ BM25 is English-centric; works best on English docs) |
Search (primary) | SearXNG (self-hosted meta-search) | Privacy-friendly search aggregator |
Search (fallback) | Tavily keyless API | Zero-config, no-key fallback when SearXNG is down/empty |
Search orchestration |
| Hermes plugin provider: SearXNG first โ auto-fallback to Tavily keyless; three-state circuit breaker (3 fails โ 60s cooldown โ half-open) + shared SQLite cache (TTL 3600s) |
Semantic reranking (P2) |
| Local embedding rerank of search results with |
MCP server (P5) |
| Exposes |
Sitemap discovery (P1) |
|
|
Multi-format extract (P0) |
|
|
RAG-ready interfaces |
| One-liners |
Deep research |
|
|
Verification |
| Clean subprocess scripts to verify each backend end-to-end per Hermes profile |
Project layout
plugins/searxng-tavily/ Hermes web-search provider plugin (SearXNG โ Tavily keyless fallback)
+ three-state circuit breaker + shared SQLite cache
scripts/
crawl4ai_cli.py Universal scraping CLI (URL โ Markdown), with denoise/retry/session/BM25
crawl_search_standalone.py Standalone search (SearXNG โ Tavily) + optional semantic rerank.
No Hermes dependency โ usable anywhere, powers the MCP server.
mcp_crawl_server.py Standard MCP server exposing search + extract + deep_research + sitemap (stdio)
single_env_check.py Verify crawl4ai provider registered+available+extracts (one profile)
verify_searxng_tavily.py Verify searxng-tavily provider: normal path + forced fallback
agent_link_check.py Verify full agent tool chain: web_search_tool dispatch + logsQuick start
1. Install Crawl4AI (China-friendly mirrors)
python3 -m venv .venv
# Use Tsinghua PyPI mirror for speed (or any mirror you prefer)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple crawl4ai
# Playwright browser kernel โ use npmmirror binary mirror if cdn.playwright.dev is blocked
PLAYWRIGHT_DOWNLOAD_HOST=https://registry.npmmirror.com/-/binary/playwright \
.venv/bin/python -m playwright install chromium
.venv/bin/crawl4ai-setup2. Scrape a page
.venv/bin/python scripts/crawl4ai_cli.py https://example.com # stdout Markdown
.venv/bin/python scripts/crawl4ai_cli.py https://example.com -o out.md # to file
.venv/bin/python scripts/crawl4ai_cli.py URL --text --max-words 5000 # plain text, truncated
# Multi-format extraction (markdown|fit|raw|markdown_with_citations)
.venv/bin/python scripts/crawl4ai_cli.py URL --format raw # unfiltered source markdown
.venv/bin/python scripts/crawl4ai_cli.py URL --format markdown_with_citations # + source URLs
# Respect robots.txt (opt-in, default off)
.venv/bin/python scripts/crawl4ai_cli.py URL --respect-robots
# Denoise nav/ads + retry 3x + reuse session across scrapes
.venv/bin/python scripts/crawl4ai_cli.py URL --noise-filter --retry 3 --session s13. Use the search + rerank (standalone, no Hermes)
# Optional: local semantic rerank of results (fastembed + bge-small-zh, auto-downloaded)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple fastembed
# SearXNG first, Tavily keyless fallback, then rerank
SEARXNG_URL=https://your-searxng .venv/bin/python -c "
import sys; sys.path.insert(0, 'scripts')
from crawl_search_standalone import CrawlSearch
r = CrawlSearch(rerank=True).search('your query')
print(r['data']['web'])"China-network note: the embedding model downloads from HuggingFace, which is blocked on mainland networks. Set
HF_ENDPOINT=https://hf-mirror.comandHF_HUB_DISABLE_XET=1(hf-mirror doesn't support the xet protocol and returns 401 without this).
4. Run as an MCP server (any client)
# Any MCP client can connect via stdio (default):
.venv/bin/python scripts/mcp_crawl_server.py
# Exposes tools:
# search(query, limit) - SearXNG โ Tavily keyless, rerank, retry+rate-limit
# extract(url, max_words, format) - markdown|fit|raw|markdown_with_citations
# deep_research(topic, num_questions) - multi-round cited report
# sitemap(origin, max_urls) - URL map from sitemap.xml / robots.txt
# Or serve over HTTP (streamable-http) for remote clients:
.venv/bin/python -m crawleyes.mcp_crawl_server --transport http --port 8765 --host 127.0.0.1
# โ clients connect to http://127.0.0.1:8765/mcp
# (host/port configurable; default 127.0.0.1:8765)For Hermes specifically, add to config.yaml:
mcp_servers:
crawl:
command: "/path/to/crawl/.venv/bin/python"
args: ["/path/to/crawl/scripts/mcp_crawl_server.py"]
timeout: 90
connect_timeout: 604b. Firecrawl-compatible /scrape endpoint
Already using Firecrawl's Python SDK? Point it at CrawlEyes and keep your code:
.venv/bin/python -m crawleyes.firecrawl_api --port 8899 --host 127.0.0.1
# POST /v2/scrape โ { success, data: { markdown, metadata } }
# GET /healthz โ health checkfrom firecrawl import Firecrawl
fc = Firecrawl(api_url="http://127.0.0.1:8899", api_key="ignored")
doc = fc.scrape(url="https://example.com") # โ { markdown, metadata }This is a pragmatic subset of the Firecrawl API โ the core /scrape contract
(success + data.markdown + data.metadata), backed by CrawlEyes' own
extraction engine. It does not implement Firecrawl's async /crawl queue,
/search, or /map โ see the design notes for the rationale.
5. Install the search plugin (Hermes)
Copy plugins/searxng-tavily/ into a Hermes plugins dir, then:
hermes plugins enable web/searxng-tavily
hermes config set web.search_backend searxng-tavilySet SEARXNG_URL in your Hermes profile .env to point at your SearXNG instance. If unset or unreachable, the provider automatically falls back to the Tavily keyless API (no API key required).
Note: the plugin only takes effect for newly started agent sessions.
6. Verify
# Requires the Hermes source tree + its venv
venv/bin/python scripts/verify_searxng_tavily.py $HERMES_HOME
venv/bin/python scripts/agent_link_check.py $HERMES_HOMEDesign notes
Layered composition: no single tool covers everything. Crawl4AI handles extraction; SearXNG + Tavily cover search; each layer has a tested fallback.
Tavily keyless works with zero configuration and no account โ a cheap resilience net for the whole search path.
Circuit breaker is SearXNG-only: a Tavily fallback success does not reset the breaker (otherwise it would never trip).
record_success()is only called when SearXNG itself succeeds.Shared SQLite cache lives in the real user home (via
pwd.getpwuid, not$HOMEโ which Hermes profiles override), so all profiles share one cache. WAL + 5s timeout + try/except degrade-to-no-cache under concurrency.Semantic rerank is cheap: fastembed (ONNX) avoids the ~2GB torch dependency; model loads in ~0.6s once cached, embeddings in ~50ms.
MCP server is standalone: it does not import Hermes internals, so it runs on any Python 3.12 env and serves any MCP client.
MCP transport is dual: stdio (default, standard MCP clients) or
streamable-http(--transport http), so a single codebase serves both local process and remote HTTP clients.Unified rate limiting is layered: MCP tools and deep-research's internal search/extract all share one sliding-window limiter (per-tool cost), so concurrent agent fan-out can't hammer SearXNG/Tavily/Crawl4AI even through multi-round deep research.
Credits & inspiration
This project builds on a set of excellent open-source tools. All code here is an independent implementation (no copied code), but the ideas and interfaces are drawn from the following projects โ full credit to their authors:
Feature in this repo | Inspired by | License |
Extraction engine (Crawl4AI wrapper) | Crawl4AI โ direct dependency | Apache-2.0 |
Content denoising (P1) | Readability, GeneralNewsExtractor (idea) | Apache-2.0 / MIT |
Semantic reranking (P2) | Vane, Perplexica (idea) | MIT / MIT |
Retry with backoff (P3) | Crawlee (idea) | Apache-2.0 |
Browser session reuse (P4) | camoufox (idea) | MIT |
MCP server (P5) | playwright-mcp, exa-mcp-server (idea) | Apache-2.0 / MIT |
Search orchestration / fallback | SearXNG โ self-hosted (official Docker image, no source modification), accessed via HTTP API only ยท Tavily keyless | AGPL-3.0 (server software, not linked/embedded) / proprietary API |
Design independence: the implementations here are written from scratch โ we studied the above projects' approaches (denoising thresholds, rerank pipelines, backoff strategies, MCP tool patterns) but did not copy their source code. Dependencies are declared in
requirements.txt. If you believe any attribution is missing or incorrect, please open an issue.
Compliance
CrawlEyes is a general-purpose fetch toolkit for legitimate research and personal use. It deliberately does not include proxy pools, fingerprint rotation, or CAPTCHA-solving (anti-scraping evasion) โ those are out of scope.
Robots.txt is opt-in (default off): pass respect_robots=True to extract / markdown (or --respect-robots on the CLI) to check each target's robots.txt (RFC 9309) and refuse URLs it explicitly disallows. It's default-off so legitimate scraping isn't silently blocked by aggressive or broken robots rules โ compliance is the caller's informed choice per use case. Always review each site's terms of service before scraping at scale.
License
MIT โ see LICENSE.
๐ฎ ๅ ณๆณจๆไปฌ
ๅ ฌไผๅท๏ผAI Aperture โโ ๅไบซ AI ๅทฅ็จๅฎ่ทตใๅค Agent ็ณป็ปไธๅผๆบ้กน็ฎ็ป้ชใ

ๆซ็ ๅ ณๆณจ๏ผ่ทๅ้กน็ฎๆดๆฐไธๆดๅคๆๆฏๅ ๅฎนใ
Available Tools
2 toolsextractB
ๆๅ็ฝ้กตๆญฃๆไธบ Markdownใ่ชๅจๅปๅช๏ผ่ฟๆปคๅฏผ่ช/ๅนฟๅ๏ผ๏ผๅคฑ่ดฅ่ชๅจ้่ฏใ Returns JSON with title/markdown/length.
Inputs: url (required), max_words. Returns: result
searchB
ๆ็ดข็ฝ้กตใSearXNG ไผๅ ๏ผๅคฑ่ดฅ่ชๅจ fallback Tavily keyless๏ผๆ ้ API key๏ผใ Returns JSON with title/url/description for each result.
Inputs: query (required), limit. Returns: result
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogโฆ
TDQS
Scored across 2 tools
Search and extract have clearly distinct purposes: one discovers URLs via web search, the other fetches and cleans page content. There is no overlap or ambiguity between the two tools.
Both tools use simple, consistent single-word imperative verbs: search and extract. The naming style is uniform and predictable.
Two tools is borderline thin for a server named CrawlEyes. The pair is coherent for a basic search-then-extract workflow, but the surface feels minimal for a crawling-focused server.
The core web research pipeline of searching and extracting content is covered, but there is no explicit crawling, pagination, or multi-page navigation capability implied by the server name. Agents can work around this by chaining searches and extracts, but it is a notable gap.
Maintenance
Related MCP Connectors
Scrape, crawl and search the web for AI agents via MCP.
Live AI-native web search with citations. One tool for every MCP client. Flat per-request pricing.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Free web search for AI agents. No API key required. Hosted MCP in active development.
Related MCP Servers
- AlicenseAqualityDmaintenanceEnables AI agents to perform multi-engine web search, fetch web pages, and extract clean Markdown content via MCP, with no API keys required.377 PyPI11MIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to perform web searches with full content retrieval and multi-engine provenance, including trust scoring and local corpus persistence, via MCP integration.4Apache 2.0
- AlicenseAqualityBmaintenanceEnables AI agents to search the web and fetch webpage content as clean text via MCP, with optional headless rendering for dynamic pages.3MIT
- AlicenseNot gradedqualityCmaintenanceEnables AI agents to search the live web and extract readable page content as MCP tools, with ranked results, domain filters, and news support.50 npmMIT