Web-MCP

Give any AI agent the parts of the web it can't reach.

Reddit and X block LLM traffic. Ordinary pages sit behind Cloudflare challenges. Search returns the marketing layer while the real answers sit in comment threads. A video's content is locked in its audio. And a local model with no search tool can't look up anything at all.

One endpoint fixes all of it. Deploy once to your own Cloudflare account, register one URL and one token, and every agent session gets the same reach.

flowchart LR
    A["<b>Your agents</b><br/>Claude Code · OpenClaw<br/>local LLMs · any MCP client"]
    W["<b>Web-MCP</b><br/>your Cloudflare Worker"]
    A -->|one URL + one token| W
    W --> R["<b>Reddit</b><br/>posts · scored comments<br/>community discovery"]
    W --> X["<b>X</b><br/>posts · replies"]
    W --> Y["<b>YouTube</b><br/>videos · comments<br/><b>transcripts</b>"]
    W --> S["<b>Open web</b><br/>keyword + semantic search"]
    W --> P["<b>Walled pages</b><br/>bot-blocked · JS shells · PDFs"]

What it actually gets you

Ask "what's the best local video model right now?" — a question no search engine answers honestly:

That's information that doesn't exist in any search index, and none of it needed a browser.

Related MCP server: xengager-mcp

Start free — three keys, no card, most of the capability

These three cost nothing, ever:

Stop there and you already have: community discovery · Reddit + YouTube search with engagement signals · full scored comment threads · keyword web search · and page fetching for every site that isn't actively blocking you.

That's four of the five tools working, for £0.

Then add what you actually need

MCP_AUTH_TOKEN is the only required secret. Every source is independent, and the server only advertises tools whose providers are configured — your agent is never offered something it can't use, and never discovers that by failing.

The tools

Reach the discussion — where practitioners actually talk

Reach the page — anything that blocks you

Reach the open web — for everything else

Five tools. No LLM inside the worker, no ranking, no summarising — raw signals go straight to the calling model, which does the thinking. Sprawling tool surfaces bloat context windows and confuse routing.

How it works

fetch_page escalates — free first, paid only when necessary

flowchart TD
    U["fetch_page(url)"] --> V{"Video URL?"}
    V -->|yes| T["Transcript<br/><i>1 credit</i>"]
    V -->|no| D["Direct fetch + HTMLRewriter<br/><b>FREE</b>"]
    D --> C{"Blocked, JS shell,<br/>or PDF?"}
    C -->|no| OK["Clean text ✓"]
    C -->|yes| F["FireCrawl stealth proxy<br/><i>1–5 credits</i>"]
    F --> OK2["Clean text ✓"]
    F --> ERR["Explicit error —<br/><b>never</b> the challenge page's HTML"]

That last box matters more than it looks. The common failure mode elsewhere is an agent confidently summarising a Cloudflare interstitial it mistook for content. Web-MCP tells you it failed.

Keyword search can merge two independent engines

flowchart LR
    Q["keyword query"] --> TA["Tavily<br/><i>page text</i>"]
    Q --> BR["Brave<br/><i>publication dates</i>"]
    TA --> M["dedupe + merge"]
    BR --> M
    M --> RES["One result carrying<br/><b>date AND full text</b><br/><i>neither engine returns alone</i>"]

Their indexes are largely disjoint, so recall roughly doubles. Every result lists which engines found it — cross-index agreement is handed to your model as a signal, not baked into a ranking here.

Everything is cached in KV and every paid provider has a daily spend ceiling that fails with a readable message rather than a surprise bill.

Extending it, or want the reasoning behind the design? ARCHITECTURE.md covers the principles, how to add a provider, and the per-provider gotchas that aren't in any vendor's docs.

Setup

1. Generate your token — the only credential your clients need.

echo "webmcp_$(openssl rand 32 | base64 | tr '+/' '-_' | tr -d '=\n')"

2. Grab the provider keys you want (all optional):

At reddit.com/prefs/apps: create another app… → type script → redirect URI http://localhost (unused). Copy the client ID (under the app name) and secret.

Set REDDIT_USER_AGENT in wrangler.toml to identify your app — e.g. cf-worker:web-mcp:v1 (by /u/yourusername). Reddit requires a unique, descriptive UA.

console.cloud.google.com → create/pick a project → APIs & Services → Library → "YouTube Data API v3" → Enable → Credentials → Create Credentials → API key.

Empty API-restrictions dropdown? The API isn't enabled on that project yet — the dropdown only lists enabled APIs.

Quota: search costs ~100× more than metadata or comments, so only search is scarce and only search is budgeted here.

Configure this if your MCP client has no web search of its own (a local LLM, for instance).

• Tavily — 1,000 credits/month free, native domain filters, full page text, no publication dates

• Brave — independent index, publication dates, no free tier since Feb 2026

KEYWORD_SEARCH_PROVIDER = auto | tavily | brave | both. Defaults to both (parallel + merged). For free-tier-only running, set it to auto.

dashboard.exa.ai. Matches meaning, not keywords — and does find-similar-by-URL, which nothing else here can do.

Complementary to keyword search, not a replacement: semantic is measurably weaker at plain factual lookups.

twitterapi.io — instant key, no X developer approval, ~$0.15/1k tweets. A small free allowance lets you test, throttled to 1 request/5s until you add credits.

firecrawl.dev. fetch_page works without it on open pages — it only escalates at a genuine wall. 1 credit, up to 5 with the enhanced proxy.

supadata.ai, 100/month free. Required because the official YouTube API cannot return transcripts: captions.download needs the video owner's OAuth, and auto-generated captions aren't exposed at all.

3. Run locally (optional):

npm install
cp .env.example .dev.vars   # fill in whichever keys you have
npm run dev                 # http://localhost:8787/mcp

4. Deploy:

export CLOUDFLARE_ACCOUNT_ID=<your-account-id>   # npx wrangler whoami

npx wrangler kv namespace create KV
# paste the returned id into wrangler.toml

echo "<value>" | npx wrangler secret put MCP_AUTH_TOKEN   # required
# ...then one line per provider key you're using

npm run deploy

The free Workers plan covers personal use comfortably.

5. Connect your client:

The server is stateless Streamable HTTP — any MCP client works with just a URL and a header.

Claude Code:

claude mcp add --transport http --scope user web-mcp \
  https://web-mcp.<your-subdomain>.workers.dev/mcp \
  --header "Authorization: Bearer <MCP_AUTH_TOKEN>"

Restart afterwards — Claude Code loads MCP servers at startup.

Jan, or any client with an "add MCP server" dialog:

Leave Command / Args / Env empty — those are for stdio servers.

The server speaks both halves of Streamable HTTP: plain JSON responses, or SSE when your client asks for text/event-stream (including the GET stream clients open for server-initiated messages). Long-running calls emit keepalive heartbeats, so transcripts and bot-protection escalations won't trip your client's idle timeout.

Clients that only speak stdio can bridge with mcp-remote: command npx, args -y mcp-remote <url> --header Authorization:${AUTH_HEADER}, and AUTH_HEADER=Bearer <token> in env. The colon has no space after it deliberately — several clients mangle spaces inside args when invoking npx.

Good to know

• Search technique matters. Reddit, X and YouTube match keywords, not meaning — a natural-language question silently degrades into "merely popular posts". Short keyword queries win, and scoping to a subreddit is the biggest lever. The tool descriptions teach your model this, so you don't have to.

• Transcripts cost 1 credit using existing captions. generate: true transcribes audio instead, billed per minute — a 60-minute talk ≈ 120 credits. Opt-in only.

• TwitterAPI.io is third-party — ~30× cheaper than the official X API, but grey-market. src/providers/ is abstracted, so swapping is contained.

• Privacy. Single-user by design: one shared token, no accounts. Don't publish your worker URL and token together.

Roadmap

Discord — searching servers you're a member of. There's no read API for it and datacenter IPs are blocked, so it likely needs a small local companion process rather than a worker provider. Same MCP pattern, different host.

Issues and ideas welcome.

License

MIT