keyless-web-mcp

Web search and page reading for MCP clients — no API key, no account, no configuration.

keyless-web-mcp is a small Model Context Protocol server that gives an agent two tools:

Both run over stdio, start in well under a second, and work the moment the process starts. There is nothing to sign up for and no key to paste.

The google_web engine renders google.com/search in an installed Chrome or Edge, so that one engine needs a browser on the machine. Every other engine, and web_fetch, is pure HTTP: with no browser present google_web simply reports itself unavailable and the rest keep working.

Why this exists

Most search servers for MCP require either a paid search API key (Brave, Tavily, Serper, Bing…), a vendor account, or a hosted endpoint your machine may not be able to reach. That is a poor fit when you just want an agent to look something up:

• Onboarding is the whole feature. A key means a signup, a credit card and a config edit before the first useful call — for a workflow that is often throwaway.

• Some networks block the popular search backends while plain HTML endpoints still work. This server tries a direct connection first and can fall back to an explicit proxy without changing any tool call.

• The answers are ordinary web pages. No proprietary index, no ranking you cannot inspect: the scraper is a normal HTTP client, and every result keeps its source URL so it can be cited.

Related MCP server: codex-search-mcp

Install

# run without installing (recommended)
uvx keyless-web-mcp

# or install it as a tool
pipx install keyless-web-mcp

# or into a project environment
pip install keyless-web-mcp

Requires Python 3.10+. The console script keyless-web-mcp and python -m keyless_web are equivalent.

Configure your client

DeepSeek Harness (DSH)

DSH loads MCP servers from a Cordis profile patch. Add an entry to ~/.dsh/profiles/<profile>/cordis.patch.yml (or pass it with dsh web --patch <file>):

# ~/.dsh/profiles/<profile>/cordis.patch.yml
- insert:
    - id: keyless-web
      name: '@deepseek-ai/dsh-mcp-client'
      config:
        serverName: web
        transport: stdio
        command: uvx
        args: ['keyless-web-mcp']
        env:
          MCP_WEB_PROXY: ''
        toolCallTimeoutMs: 120000

On Windows, if the client does not resolve uvx through PATH, point command at the absolute executable path (...\uv.exe / ...\uvx.exe).

Claude Desktop

Edit claude_desktop_config.json (%APPDATA%\Claude\ on Windows, ~/Library/Application Support/Claude/ on macOS) and restart the app:

{
  "mcpServers": {
    "web": {
      "command": "uvx",
      "args": ["keyless-web-mcp"],
      "env": {
        "MCP_WEB_PROXY": ""
      }
    }
  }
}

Any other stdio-capable MCP client works the same way: run uvx keyless-web-mcp, pipx run keyless-web-mcp or python -m keyless_web as the server command.

Tools

web_search

Returns a JSON array (a string with [] when nothing matched):

[
  {
    "title": "NCCL: Scalable Multi-GPU Communication",
    "url": "https://developer.nvidia.com/blog/...",
    "snippet": "Collective communication primitives for multi-GPU...",
    "engine": "duckduckgo"
  },
  {
    "title": "NVIDIA NCCL 2.27 release notes",
    "url": "https://news.google.com/rss/articles/...",
    "snippet": "...",
    "engine": "google",
    "source": "NVIDIA Developer",
    "date": "Tue, 04 Nov 2026 09:12:00 GMT"
  }
]

source and date are present only for news hits (engine: "google"). Engines are queried in the order you give them; hits are merged and de-duplicated by URL.

web_fetch

Returns plain text that starts with Title: / URL: followed by the extracted main content. JSON endpoints are pretty-printed instead of parsed as HTML.

Both tools report failures as text rather than raising, so the model can react:

error[KEYLESS_BAD_URL]: only http:// and https:// URLs are supported (got 'file:///etc/passwd').
Hint: Pass an absolute http:// or https:// URL, for example https://docs.python.org/3/.

Engines

Default order: google_web, firecrawl, duckduckgo, google. Every engine runs concurrently and the hits are interleaved round-robin, so one answer can cite several independent sources without any backend disappearing behind a longer list. A single engine failing never fails the call; if every engine fails or finds nothing, web_search returns [].

A backend that answers a different query

A consumer search endpoint may answer a scripted client with something other than its own results: Bing hands back a decoy SERP built from other queries, DuckDuckGo an "anomaly" page. Both look legitimate — same blocks, same result count, same echoed query in the title — so the only in-process evidence is that the hits have nothing to do with the query.

Each engine's hits are therefore checked against the query's terms (substring, case-insensitive, Latin words of two characters or more, CJK runs as-is). A group where no hit shares a term is discarded whole, before the merge, and the reason is recorded. Only whole groups are judged: a relevant hit can share no term with the query at all ("NCCL timeout" → a fault-tolerance article that never names it), so individual hits are never dropped. If every non-empty group would be discarded, the gate switches itself off for that call rather than turning "some data" into "no data".

web_search still returns the same JSON array; the per-engine record is available to code that asks for it (keyless_web.engines.search_with_report, and the engines member of the JSON-lines worker's search result):

{"name": "bing", "returned": 10, "matched": 0, "dropped": "no hit shared a term with the query"}

The gate is a safety net, not a ranking: it cannot rescue a backend that answers with plausible but wrong hits that happen to mention a query term, and it cannot tell NCCL the library from NCCL the catechetical-leaders conference.

Configuration

Everything is read from the environment, once, at start-up.

Example:

MCP_WEB_PROXY=http://127.0.0.1:7890 KEYLESS_WEB_MIN_INTERVAL=2 keyless-web-mcp

KEYLESS_WEB_USER_AGENT applies to the HTML scrapers and to web_fetch. The Wikipedia engine always identifies itself as keyless-web-mcp/<version>, because Wikimedia's robot policy requires a descriptive agent and answers HTTP 403 to browser-impersonating clients.

Proxy fallback and TLS

Each request is attempted in order:

1. Direct — a normal client that honours HTTP_PROXY/HTTPS_PROXY from the environment.

2. Explicit proxy — only when MCP_WEB_PROXY is set (for example http://127.0.0.1:7890, or 10.20.30.40:8080 which is normalized to http://10.20.30.40:8080). Plain host:port is accepted, and an IP works even when local DNS is broken because the proxy resolves hostnames.

Only transport failures (timeout, connect/DNS/TLS error) move to the next backend. Once a response arrives, its HTTP status is final — 4xx/5xx is reported, not retried through another path. When every backend fails, all per-backend errors are collected into one KEYLESS_NETWORK message, so you can see which route failed and why.

TLS verification stays on. On start-up the server loads the operating system certificate store via truststore, so TLS-interception proxies and desktop antivirus products that inject their own root CA work without --insecure flags or REQUESTS_CA_BUNDLE settings.

Politeness: rate limiting and caching

• Per-host minimum interval (KEYLESS_WEB_MIN_INTERVAL, default 1 request/second). The limiter keys on host, so hammering one site is throttled while other hosts proceed independently.

• In-process TTL cache: search results 300 s, fetched pages 900 s, keyed on engine, query and every output-affecting argument. Re-reading the page an agent just found costs nothing.

Both are deliberately conservative: this is a single-user tool, not a crawler. Raise the interval if you script many queries, and see the compliance note below.

Errors

Failures are reported to the model as error[CODE]: message. Hint: ....

Compliance

This project fetches public web pages with an ordinary HTTP client and no API agreement with the sites it reads. By using it you agree to:

• respect each site's Terms of Service, robots.txt and rate limits — keep KEYLESS_WEB_MIN_INTERVAL at a polite value;

• not use it for bulk crawling, scraping behind logins, or republishing copyrighted content;

• treat results as pointers, not as licensed data: cite and link the original page.

Scraping HTML endpoints is inherently unstable: an upstream markup change can break an engine without notice. No availability or correctness guarantee is offered. Engines may stop working, be blocked from your network, or return degraded results; that is a property of the endpoints, not a bug report target. Two of those degradations look exactly like success, so they are worth naming: DuckDuckGo's HTML endpoints answer HTTP 202 with an "anomaly" page (no results, no error), and Bing answers a scripted client with a decoy SERP whose hits belong to other queries while the page title still echoes the requested one. Prefer a backend that reports its own results (google_web, or the firecrawl JSON API), and treat a result set you cannot match to your query as untrustworthy. MIT licensed, provided "as is".

Development

git clone https://github.com/ai-v5/keyless-web-mcp
cd keyless-web-mcp
python -m venv .venv
.venv/Scripts/activate        # Windows; use .venv/bin/activate elsewhere
pip install -e ".[dev]"

ruff check .
ruff format --check .
mypy src
pytest -q

The architecture keeps policy out of the parsers:

src/keyless_web/
├── config.py            # environment -> frozen Settings
├── errors.py            # KEYLESS_* codes, actionable hints
├── cache.py             # TTL cache + cache_key()
├── browser.py           # headless Chrome/Edge -> rendered DOM (--dump-dom)
├── net/transport.py     # direct -> proxy fallback, rate limiting, truststore
├── engines/             # one module per engine, each just (client, query, n)
│   ├── base.py          # SearchResult (pydantic) + Engine protocol
│   ├── duckduckgo.py  bing.py  google_news.py  wikipedia.py  firecrawl.py
│   ├── google_web.py    # parses the rendered Google page, recovers /goto URLs
│   └── __init__.py      # ENGINES registry + parallel run + round-robin merge
├── fetch/extract.py     # trafilatura/BeautifulSoup extraction, JSON passthrough
└── server.py            # MCP tool shell (declarations + argument validation only)

Engines never build their own client: they receive an httpx.Client, which is why every engine can be unit-tested with httpx.MockTransport and zero network access. The one exception is google_web, which takes its DOM from browser.dump_dom() and only uses the client to resolve redirect links -- so its parser is still exercised offline against a fixture.

Roadmap

Optional next steps, not commitments:

• result scoring (dedupe near-duplicates, rank by title/query overlap) to shrink the JSON the model has to read;

• additional keyless engines behind the same registry;

• optional robots.txt pre-check for web_fetch;

• reuse one browser process across searches instead of one launch per query (the launch dominates the ~4–8 s the google_web engine costs).

License

MIT — see LICENSE. Security policy: SECURITY.md. Contributing: CONTRIBUTING.md.

中文文档见 README.zh-CN.md。