CrawlEyes — Web Scraping & Search Toolkit for AI Agents

CrawlEyes gives AI agents reliable full-text extraction (web_extract) and robust search (web_search) backends — the "eyes" that let agents see and read the web. Built and tested against Hermes Agent.

Also ships as a standard MCP server, so any MCP client (Claude Desktop, Cursor, other agents) can reuse the same search + extraction capabilities.

Features

Related MCP server: Web Search MCP Server

Project layout

plugins/searxng-tavily/   Hermes web-search provider plugin (SearXNG → Tavily keyless fallback)
                          + three-state circuit breaker + shared SQLite cache
scripts/
  crawl4ai_cli.py          Universal scraping CLI (URL → Markdown), with denoise/retry/session/BM25
  crawl_search_standalone.py  Standalone search (SearXNG → Tavily) + optional semantic rerank.
                             No Hermes dependency — usable anywhere, powers the MCP server.
  mcp_crawl_server.py      Standard MCP server exposing search + extract (stdio transport)
  single_env_check.py      Verify crawl4ai provider registered+available+extracts (one profile)
  verify_searxng_tavily.py Verify searxng-tavily provider: normal path + forced fallback
  agent_link_check.py      Verify full agent tool chain: web_search_tool dispatch + logs
research_log/              Honest test notes (P0 Crawl4AI / P3 Firecrawl / P4 MediaCrawler)
docs/original/             Archived official READMEs (crawl4ai / firecrawl / media-crawler)

Quick start

1. Install Crawl4AI (China-friendly mirrors)

python3 -m venv .venv
# Use Tsinghua PyPI mirror for speed (or any mirror you prefer)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple crawl4ai
# Playwright browser kernel — use npmmirror binary mirror if cdn.playwright.dev is blocked
PLAYWRIGHT_DOWNLOAD_HOST=https://registry.npmmirror.com/-/binary/playwright \
  .venv/bin/python -m playwright install chromium
.venv/bin/crawl4ai-setup

2. Scrape a page

.venv/bin/python scripts/crawl4ai_cli.py https://example.com          # stdout Markdown
.venv/bin/python scripts/crawl4ai_cli.py https://example.com -o out.md  # to file
.venv/bin/python scripts/crawl4ai_cli.py URL --text --max-words 5000   # plain text, truncated

# Denoise nav/ads + retry 3x + reuse session across scrapes
.venv/bin/python scripts/crawl4ai_cli.py URL --noise-filter --retry 3 --session s1

3. Use the search + rerank (standalone, no Hermes)

# Optional: local semantic rerank of results (fastembed + bge-small-zh, auto-downloaded)
.venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple fastembed

# SearXNG first, Tavily keyless fallback, then rerank
SEARXNG_URL=https://your-searxng .venv/bin/python -c "
import sys; sys.path.insert(0, 'scripts')
from crawl_search_standalone import CrawlSearch
r = CrawlSearch(rerank=True).search('your query')
print(r['data']['web'])"

China-network note: the embedding model downloads from HuggingFace, which is blocked on mainland networks. Set HF_ENDPOINT=https://hf-mirror.com and HF_HUB_DISABLE_XET=1 (hf-mirror doesn't support the xet protocol and returns 401 without this).

4. Run as an MCP server (any client)

# Any MCP client can connect via stdio:
.venv/bin/python scripts/mcp_crawl_server.py
# Exposes tools: search(query, limit) + extract(url, max_words)

For Hermes specifically, add to config.yaml:

mcp_servers:
  crawl:
    command: "/path/to/crawl/.venv/bin/python"
    args: ["/path/to/crawl/scripts/mcp_crawl_server.py"]
    timeout: 90
    connect_timeout: 60

5. Install the search plugin (Hermes)

Copy plugins/searxng-tavily/ into a Hermes plugins dir, then:

hermes plugins enable web/searxng-tavily
hermes config set web.search_backend searxng-tavily

Set SEARXNG_URL in your Hermes profile .env to point at your SearXNG instance. If unset or unreachable, the provider automatically falls back to the Tavily keyless API (no API key required).

Note: the plugin only takes effect for newly started agent sessions.

6. Verify

# Requires the Hermes source tree + its venv
venv/bin/python scripts/verify_searxng_tavily.py $HERMES_HOME
venv/bin/python scripts/agent_link_check.py $HERMES_HOME

Design notes

• Layered composition: no single tool covers everything. Crawl4AI handles extraction; SearXNG + Tavily cover search; each layer has a tested fallback.

• Tavily keyless works with zero configuration and no account — a cheap resilience net for the whole search path.

• Circuit breaker is SearXNG-only: a Tavily fallback success does not reset the breaker (otherwise it would never trip). record_success() is only called when SearXNG itself succeeds.

• Shared SQLite cache lives in the real user home (via pwd.getpwuid, not $HOME — which Hermes profiles override), so all profiles share one cache. WAL + 5s timeout + try/except degrade-to-no-cache under concurrency.

• Semantic rerank is cheap: fastembed (ONNX) avoids the ~2GB torch dependency; model loads in ~0.6s once cached, embeddings in ~50ms.

• MCP server is standalone: it does not import Hermes internals, so it runs on any Python 3.12 env and serves any MCP client.

• Honest limitations are documented in research_log/ (e.g. CAPTCHA walls on some sites are deliberately recorded as not-solvable rather than being worked around).

Credits & inspiration

This project builds on a set of excellent open-source tools. All code here is an independent implementation (no copied code), but the ideas and interfaces are drawn from the following projects — full credit to their authors:

Design independence: the implementations here are written from scratch — we studied the above projects' approaches (denoising thresholds, rerank pipelines, backoff strategies, MCP tool patterns) but did not copy their source code. Dependencies are declared in requirements.txt. If you believe any attribution is missing or incorrect, please open an issue.

Compliance

Respect each target site's robots.txt and terms of service. This toolkit is for legitimate research and personal use. It deliberately does not include proxy pools, fingerprint rotation, or CAPTCHA-solving (anti-scraping evasion) — those are out of scope.

License

MIT — see LICENSE.