Hive Web

Русская версия

Hive Web is a local MCP server for agents that need web access without flooding the model with raw pages. It gives Hermes, Claude Code, Codex, OpenCode, and other MCP clients a small set of tools for search, page extraction, and controlled browser sessions.

The server is built around two paths:

static-web   cheap search and page extraction through SearXNG and Firecrawl
action-web   live Playwright sessions with compact snapshots and safe actions

Static tools handle normal research. Action tools are for sites that need a browser, forms, or navigation. Both return compact, structured data and store larger payloads as local artifacts.

Naming

The project is called Hive Web. The GitHub repository is hive-web.

The Python package and executable are named hive-web-runtime because they run the local MCP server. MCP clients usually register that server as hive_web.

GitHub repository   hive-web
Python package      hive-web-runtime
CLI command         hive-web-runtime
MCP server name     hive_web
Python module       hive_web_runtime

Related MCP server: Local-MCP-server

Tools

Static tools:

• static_web_search(query, limit=5, max_tokens=2000)

• static_web_extract(url, max_tokens=3000, format="markdown")

• static_web_get_artifact(artifact_id, name?)

Browser tools:

• action_web_session_create(name?, headless?)

• action_web_navigate(session_id, url? | search_query?)

• action_web_snapshot(session_id, max_tokens=1200)

• action_web_click(session_id, ref? | selector?, confirm_sensitive=false)

• action_web_type(session_id, text, ref? | selector?, clear=true)

• action_web_press(session_id, key)

• action_web_close(session_id)

Requirements

• Python 3.11+

• uv

• Local or remote SearXNG endpoint

• Local or remote Firecrawl endpoint

• Playwright browsers for action-web sessions

Default endpoints:

SEARXNG_URL=http://localhost:8888
FIRECRAWL_API_URL=http://localhost:3002
HIVE_WEB_ARTIFACT_DIR=~/.cache/hive-web-runtime/artifacts
HIVE_WEB_SEARCH_ENGINES=bing,wikipedia,wikidata
HIVE_WEB_SEARCH_FALLBACK_ENGINES=bing
HIVE_WEB_REQUEST_TIMEOUT_SECONDS=45

static_web_search pins SearXNG to reliable engines by default instead of letting blocked/CAPTCHA-heavy public engines silently return empty JSON. Override HIVE_WEB_SEARCH_ENGINES if your SearXNG instance has a better engine mix. Search responses include warnings from SearXNG unresponsive_engines, so CAPTCHA/rate-limit/backend failures are visible to the caller instead of looking like "the web has no results".

Action-web browser options:

HIVE_WEB_BROWSER_HEADLESS=false          # default true
HIVE_WEB_BROWSER_CHANNEL=chrome         # optional: use installed Google Chrome instead of bundled Chromium
HIVE_WEB_BROWSER_PROXY_URL_FILE=~/.config/hive-web-runtime/browser-proxy-url
HIVE_WEB_BROWSER_ARGS=--disable-blink-features=AutomationControlled
HIVE_WEB_BROWSER_LOCALE=ru-RU
HIVE_WEB_BROWSER_TIMEZONE=Europe/Moscow
HIVE_WEB_PAGE_TIMEOUT_MS=30000
# or HIVE_WEB_BROWSER_PROXY_URL=http://user:password@host:port

Prefer HIVE_WEB_BROWSER_PROXY_URL_FILE for authenticated proxies so secrets stay out of MCP YAML config. The file should be user-readable only, for example mode 0600.

If you already run SearXNG and Firecrawl somewhere else, set those environment variables before starting the MCP server.

Running the backend services

Hive Web is an MCP runtime, not a bundled search/scrape stack. Start SearXNG and Firecrawl first, then start or register the MCP server.

If you use the companion local Docker stack on macOS, the normal flow is:

cd ~/workspace/hermes/hermes-local-web-stack
./scripts/start.sh          # docker compose up -d and endpoint checks
./scripts/status.sh         # verifies SearXNG search and Firecrawl /v1/scrape
./scripts/ensure-running.sh # idempotent readiness/repair check used by autostart

Expected healthy status:

SearXNG: 10 results
Firecrawl scrape: HTTP 200 ... markdown chars

scripts/status.sh should check a real Firecrawl scrape, not just that port 3002 answers. A root HTTP 200 only proves the API process is listening; static_web_extract requires /v1/scrape to work.

For a generic setup, any SearXNG instance that serves JSON search and any Firecrawl-compatible API work:

export SEARXNG_URL=http://localhost:8888
export FIRECRAWL_API_URL=http://localhost:3002

Install

git clone https://github.com/YegorMy/hive-web.git
cd hive-web
uv run playwright install chromium
bash scripts/install-hermes-mcp.sh

The installer runs uv sync, writes a hive_web entry into ~/.hermes/config.yaml, and tests the MCP connection. The Playwright install step is needed for action_web_* browser tools; static_web_* tools only need SearXNG and Firecrawl.

The installer also persists the search engine and timeout options shown above. If you set browser options such as HIVE_WEB_BROWSER_CHANNEL, HIVE_WEB_BROWSER_PROXY_URL_FILE, locale, timezone, or HIVE_WEB_BROWSER_ARGS before running it, those safe runtime options are copied into the Hermes MCP entry too. Secrets should stay in files or client secret stores, not in public examples.

After changing MCP config, reload MCP in your client or start a new session.

/reload-mcp

Manual server start, mostly useful for debugging:

uv run hive-web-runtime

The server speaks MCP over stdio, so it waits for an MCP client.

Quick local smoke test

With SearXNG and Firecrawl already healthy:

cd /path/to/hive-web
uv run pytest -q
hermes mcp test hive_web
uv run python scripts/test-mcp-client.py
uv run python scripts/smoke-action-web.py

scripts/test-mcp-client.py initializes the MCP server over stdio, lists tools, and calls static_web_extract against https://example.com. It fails if the MCP tool returns an error, so a green run means Firecrawl extraction actually worked.

Development checks

Run the unit tests:

HIVE_WEB_ARTIFACT_DIR=/tmp/hive-web-runtime-artifacts uv run pytest -q

Run a live MCP smoke test. This requires Firecrawl to be reachable at FIRECRAWL_API_URL:

uv run python scripts/test-mcp-client.py

For live browser changes, also run:

uv run python scripts/smoke-action-web.py

Hermes setup

If you skipped the install script above, run it from the cloned repository:

bash scripts/install-hermes-mcp.sh

You can override the server name and endpoints:

SERVER_NAME=hive_web \
SEARXNG_URL=http://localhost:8888 \
FIRECRAWL_API_URL=http://localhost:3002 \
bash scripts/install-hermes-mcp.sh

Manual Hermes config:

mcp_servers:
  hive_web:
    command: /absolute/path/to/uv
    args: ["run", "--project", "/absolute/path/to/hive-web", "hive-web-runtime"]
    env:
      SEARXNG_URL: "http://localhost:8888"
      FIRECRAWL_API_URL: "http://localhost:3002"
      HIVE_WEB_ARTIFACT_DIR: "/absolute/path/to/artifacts"
      HIVE_WEB_SEARCH_ENGINES: "bing,wikipedia,wikidata"
      HIVE_WEB_SEARCH_FALLBACK_ENGINES: "bing"
      HIVE_WEB_REQUEST_TIMEOUT_SECONDS: "45"
      HIVE_WEB_PAGE_TIMEOUT_MS: "30000"
    connect_timeout: 60
    enabled: true

After changing MCP config, reload MCP in the client or start a new session.

Other MCP clients

Claude Code:

claude mcp add -s user hive_web -- uv run --project /absolute/path/to/hive-web hive-web-runtime

Codex CLI:

codex mcp add hive_web -- uv run --project /absolute/path/to/hive-web hive-web-runtime

OpenCode uses the same stdio command in its MCP config:

{
  "mcp": {
    "hive_web": {
      "command": "uv",
      "args": ["run", "--project", "/absolute/path/to/hive-web", "hive-web-runtime"]
    }
  }
}

Safety model

Hive Web tries to keep browser automation boring and predictable:

• Search and extraction return compact markdown or structured JSON, not full raw HTML by default.

• Larger payloads go into the artifact store and can be fetched by artifact_id.

• static_web_get_artifact picks the most useful artifact file automatically when name is omitted (content.md, snapshot.json, results.json, then raw.json).

• Browser snapshots expose refs like @1 and @2 instead of dumping the full DOM.

• action_web_click blocks payment, password, 2FA, and CAPTCHA-looking targets unless the caller explicitly confirms the action.

Reliability behavior

• Static search and extraction use configurable request timeouts (HIVE_WEB_REQUEST_TIMEOUT_SECONDS).

• Action-web navigation uses HIVE_WEB_PAGE_TIMEOUT_MS instead of waiting forever on stuck pages.

• Duplicate named browser sessions are rejected instead of silently overwriting and leaking an old browser context.

• Firecrawl error payloads are surfaced as explicit errors and their raw response is saved as an artifact for debugging.

• Browser snapshots are trimmed to the requested token budget, including interactives, while the raw snapshot remains available in the artifact store.

This is not a CAPTCHA bypass tool, a shopping bot, or a payment automation layer. Treat it as read/search/extract plus careful browser control.

Network routing

Hive Web does not mutate system routes or install routing helpers. For browser sessions that need a special exit path, configure an action-web browser proxy with HIVE_WEB_BROWSER_PROXY_URL_FILE or HIVE_WEB_BROWSER_PROXY_URL; static-web search/extract remains routed through its SearXNG/Firecrawl backends.

Development

uv sync
HIVE_WEB_ARTIFACT_DIR=/tmp/hive-web-runtime-artifacts uv run pytest -q
uv run python scripts/test-mcp-client.py
uv run python scripts/smoke-action-web.py

Use a temporary artifact directory for tests if your normal cache directory is not writable in the current environment.

License

MIT