worbrow

Agent search CLI: drives local headless browsers (Chrome/Edge via hand-written CDP, Firefox via hand-written Marionette) to search general web search engines, emitting a stable JSON contract for AI agents to invoke as a subprocess.

Architecture & decisions: docs/design.md; feature roadmap: docs/roadmap.md. 简体中文版见 README.zh-CN.md.

Quick Start

Prerequisites: Chrome/Edge (>= 109) or Firefox (>= 55) installed on the system.

cargo run -- list                    # list available engines
cargo run -- doctor                  # environment self-check (browser binaries/engines/backend status)
cargo run -- "rust async runtime" --json   # default engine duckduckgo→baidu→bing fallback, default timeout 60s
cargo run -- "rust" --engine duckduckgo --timeout 30 --max-results 5
cargo run -- "rust" --pages 2 --max-results 15 --lang zh-hans --region zh-CN   # multi-page aggregation + language/region
cargo run -- "rust" --freshness week --safesearch strict                       # freshness filter + safe search
cargo run -- "rust" --site doc.rust-lang.org --filetype pdf                    # site/file-type filter
cargo run -- "rust" --engine bing,duckduckgo   # engine fallback chain (auto-tries the next on captcha/low-quality/low-yield)
cargo run -- "rust" --retry 2                  # backoff retry on transient network errors (exponential backoff capped at 8s)
cargo run -- "rust" --proxy http://127.0.0.1:7890  # HTTP/HTTPS proxy (ADR-013)
# body fetch + structured extraction (ADR-009): pass an explicit URL, get cleaned text and optional fields
cargo run -- fetch https://example.com/rust --json
cargo run -- fetch https://example.com --extract price,rating --json    # field extraction (allowlist)
cargo run -- fetch https://example.com --no-text --extract price        # fields only, saves tokens
cargo run -- fetch https://example.com --format markdown --json         # Markdown output (ADR-014)

Backend status: firefox (Marionette, hand-written protocol) and chrome (CDP, hand-written protocol) are both implemented; fake is for tests/smoke. Protocol implementation: ADR-002.

HTTP/HTTPS proxy (ADR-013): --proxy http://host:port (or https://host:port) applies to the browser launch (CDP --proxy-server, Marionette network.proxy.*) and the static SERP HTTP GET (ureq). Invalid proxy URLs → exit 2, checked before the browser starts. Without --proxy, browsers use system proxy settings and the HTTP client reads HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY. Design: ADR-013.

Installation

Prebuilt binaries are attached to every GitHub Release (v* tags): worbrow-linux-x86_64.tar.gz, worbrow-linux-x86_64-musl.tar.gz (statically linked, runs on any Linux), worbrow-macos-arm64.tar.gz, worbrow-windows-x86_64.zip, each with a SHA256SUMS checksum. Debian/Ubuntu users can also install the release .deb (includes MCP support, worbrow mcp):

sudo apt install ./worbrow_*.deb

Or build from source (cargo install worbrow, or make deb for a local .deb). Runtime soft dependency on Firefox (Recommends: firefox | firefox-esr).

MCP (Model Context Protocol)

cargo build --release

Run worbrow mcp as an MCP stdio server, exposing tools to MCP clients:

• web_search (query/engine/browser/max_results/timeout/lang/region/pages/freshness/safesearch/site/filetype/retry/no_cache/compact)

• fetch_page (url/browser/timeout/max_chars/extract/text/wait_selector/retry: fetches an explicitly passed URL, returns cleaned body text and optional structured fields)

• list_engines (list available engines), doctor (environment self-check: browser binaries/versions/engine registry)

Tool results reuse the output contract (schema v1). With compact=true, web_search results contain only rank/title/url (saves agent context tokens, meta complete). Design: ADR-005 and ADR-009. (If MCP is not needed: cargo build --no-default-features)

worbrow mcp --idle-timeout <secs>: exits automatically after this long without any request (prevents orphan processes after an agent crashes; 0 = disabled, the default). worbrow mcp --proxy http://host:port applies a proxy to all sessions in the process (server-level; tools have no per-request proxy).

Session pooling (MCP long-running): browser processes are reused inside the MCP process, removing the 2-5s spawn overhead per search. --max-sessions <n> concurrency cap (default 1 = serial reuse, excess queued), --session-ttl <sec> idle-session reclamation threshold (default 60s); idle sessions past TTL are recycled, crashed sessions auto-rebuilt, transparent to agents (schema v1 unchanged). Design: ADR-007.

Network retry & result cache (ADR-008): the retry parameter (default 0) applies exponential backoff to transient network errors (capped at 8s, counted in the timeout budget); meta.retries records the actual retry count. Inside the long-running MCP process, repeated calls with identical parameters within a 60s TTL hit the cache directly (meta.cached=true, elapsed_ms=0); no_cache bypasses it. The CLI --retry <n> supports retry as well (no cache).

Body fetch & structured extraction (ADR-009)

fetch_page / worbrow fetch <url> fetches an explicitly passed URL and returns cleaned body text, optionally extracting structured fields via the extract allowlist (title/author/published_at/price/currency/rating/rating_max/reviews_count; extraction priority JSON-LD → meta → DOM; missing fields are omitted, never fabricated):

worbrow fetch https://example.com --json --extract price,rating

• Closing the loop: pass results[i].url from web_search explicitly to fetch_page — "search links → read content → compare fields" in one step; never auto-follows search results (fetches explicit URLs only)

• Parameters: --max-chars <n> (body truncation, default 20000, flagged by meta.truncated), --no-text (extracted fields only, saves tokens), --extract a,b (allowlist, invalid values exit 2), --wait-selector <css> (SPA: wait for this selector before extracting text, best-effort), --format text|markdown (body output format, ADR-014: markdown preserves headings/links/lists/code blocks, the standard agent-consumption format)

• Known behavior: navigation success always yields a success payload (body may be empty); meta.http_status reports the page HTTP status (best-effort via PerformanceNavigationTiming.responseStatus; null on Firefox < 105 / data: URLs), so 4xx/5xx/404 no longer go unnoticed; meta.final_url records the redirect landing page; SPA/lazy-loaded content may be missing — pass --wait-selector <css> to wait for the content to render

• Safety boundary: http/https only (missing scheme defaults to https://); navigates with a real browser, page JS runs inside the browser (equivalent to clicking the link yourself); can reach localhost/intranet (equivalent to your local browser — do not feed untrusted input if you want to avoid being induced to fetch intranet content); no bulk fetching; rate discipline still applies

• Compliance: fetch is an explicit full-page fetch by the user, a separate path from search engines' snippet-only crawling policy

Related MCP server: free-search-mcp

Agent Integration

worbrow offers two agent integration paths: MCP (recommended; long-running process + tool semantics) and CLI subprocess (zero-dependency, one-shot). Both share the same core and output contract (schema v1).

Claude Code

Register in claude_desktop_config.json (or .claude.json under mcpServers):

{
  "mcpServers": {
    "worbrow": {
      "command": "worbrow",
      "args": ["mcp", "--idle-timeout", "300"]
    }
  }
}

Tip: --idle-timeout 300 makes the long-running process exit 5 minutes after the agent session goes idle, avoiding orphans. Ensure worbrow is on PATH (automatically satisfied after make deb).

Cursor / generic MCP clients

Project-level .mcp.json (Cursor) or an equivalent global client config:

{
  "mcpServers": {
    "worbrow": {
      "command": "worbrow",
      "args": ["mcp", "--idle-timeout", "300"]
    }
  }
}

Exposed tools: web_search (parameters include engine/browser/max_results/timeout/lang/region/pages/freshness/safesearch/site/filetype).

CLI subprocess (without an MCP client)

worbrow "rust async" --engine bing --max-results 8 --timeout 60 --json

• Read stdout JSON (validate schema_version), logs on stderr

• On non-zero exit, stdout still carries the error JSON payload (code/message/detail)

• Each result carries domain/https, no need to parse URLs yourself to judge the source

Output contract (agent side)

• stdout only emits JSON (--json), all logs go to stderr

• Semantic exit codes: 0 success / 2 argument error / 3 environment error / 4 search failure / 124 timeout / 1 internal error

• Versioned schema: top-level schema_version field; fields only grow, never change

• No interaction; hard timeout defaults to 60s

Example success payload:

{
  "schema_version": 1,
  "query": "rust",
  "results": [{ "rank": 1, "title": "…", "url": "https://…", "snippet": "…",
                "domain": "example.com", "https": true,
                "published_at": "2025年5月25日", "is_ad": false,
                "url_resolved": true, "result_kind": "web" }],
  "meta": { "engine": "bing", "started_at": "…", "elapsed_ms": 1200,
            "result_count": 3, "pages": 1, "low_yield": false,
            "captcha": false, "engine_error": null,
            "engine_tried": ["bing"], "cached": false, "retries": 0 }
}

Use as a library

worbrow's public library surface is a type-level top-level API (ADR-006): consumers assemble everything with one use worbrow::..., no need to know the internal module tree.

use worbrow::{BrowserKind, Config, search};

fn main() -> Result<(), worbrow::Error> {
    let outcome = search(Config::new("rust async", "bing", BrowserKind::Firefox)
        .with_max_results(5))?;
    for r in &outcome.results {
        println!("{} - {}", r.rank, r.title);
    }
    Ok(())
}

Body fetch (ADR-009) is a first-class library API too:

use worbrow::{BrowserKind, ExtractField, FetchConfig, fetch};

fn main() -> Result<(), worbrow::Error> {
    let page = fetch(FetchConfig::new("https://example.com/rust", BrowserKind::Firefox)
        .with_extract(vec![ExtractField::Title, ExtractField::PublishedAt]))?;
    println!("{}: {} chars", page.url, page.chars);
    println!("title: {:?}", page.extracted.get("title"));
    Ok(())
}

Pick the entry point by whether you are already inside a tokio runtime, to avoid nested-runtime panics:

• search (sync): use when there is no runtime context (main/CLI/scripts/spawn_blocking closures); builds its own runtime internally, do not call in async contexts

• run (async): use when a runtime already exists (MCP handlers / #[tokio::main] / #[tokio::test]), awaiting reuses the external runtime; to block-synchronously wait inside async, use tokio::task::block_in_place(|| handle.block_on(run(cfg))) (needs a multi-thread runtime)

• Dependency surface: library consumers can use default-features = false to drop the MCP dependency (rmcp) — the mcp feature is enabled by default only to serve the CLI binary. Note this also drops the default tls-ring provider, so re-select one (tls-aws-lc/tls-none, see below)

• TLS provider (ADR-016): ureq is built with rustls-no-provider, so pick the rustls provider explicitly — tls-ring (default, musl static-link friendly), tls-aws-lc (align with a reqwest/aws-lc-rs dependency), or tls-none (use the process-wide CryptoProvider::install_default, falling back to the browser when absent). Choosing tls-aws-lc under default-features = false avoids compiling two TLS backends (ring + aws-lc-sys) and its cmake build dependency when you also depend on reqwest (issue #62)

• Extension: implement a custom engine via SearchProvider and inject it with Config::with_provider; implement a custom browser backend via BrowserDriver and inject it with Config::with_driver

• Contract serialization: SuccessPayload/ErrorPayload (including schema_version) can be serde_json::to_string'd directly; runnable examples live in examples/ (cargo run --example basic_search)

Quality commands

No just; the unified entry point is make:

make check      # fmt + clippy(-D warnings, cognitive complexity <= 10) + lizard (CCN <= 15) + test
make test       # cargo test (mcp enabled by default, CI needs no browser)
make complexity # lizard cyclomatic complexity (CCN <= 15)
make deny       # cargo-deny license/vulnerability check
make machete    # unused-dependency check
make doctor     # run worbrow doctor
make bench-search  # dump live search JSON for external evaluation (network; docs/benchmark.md)

Search tips

• Avoid stacking nouns separated by spaces in Chinese queries: Bing may anchor the whole query to the first strong entity (e.g. a fund-data query returns China wiki/baike pages) — results are plentiful and well-formed but completely off-topic. The relevance gate falls back to DuckDuckGo/Baidu automatically, but it is more reliable to cut noisy terms; use double quotes for exact phrases (e.g. "天天基金网 净值查询")

• Engine choice: --engine duckduckgo (html endpoint; HTTP GET first, ADR-011), --engine baidu (Chinese long-tail; reachable from CN networks with no CAPTCHA wall), --engine bing (English queries); the default chain duckduckgo,baidu,bing tries the next engine when the quality gates fail

• Query quality: avoid piling up high-noise words such as best/learn (Bing tends to misclassify them as dictionary intent; see docs/roadmap-result-quality.md); rephrase with synonyms, quotes, or site: constraints when results are poor

Layout

src/
  main.rs    # thin entry + CLI parsing (clap, bin-private)
  lib.rs     # public library surface: top-level re-exports (Config/BrowserKind/..., ADR-006)
  app.rs domain.rs error.rs ports.rs output.rs extract.rs http_serp.rs mcp.rs
  drivers/   # resolve · jsonrpc · cdp · marionette · fake · pool · discovery
  engines/   # resolve/AVAILABLE · duckduckgo · bing · baidu
tests/       # integration tests + fixtures (offline HTML golden)

Dependency direction: cli → app → domain/ports ← adapters(drivers/engines), reverse is forbidden.

License

MIT OR Apache-2.0 (see LICENSE-MIT; Apache-2.0 text at https://www.apache.org/licenses/LICENSE-2.0).