Web-MCP

Give any AI agent the parts of the web it can't reach.

Reddit and X block LLM traffic. Ordinary pages sit behind Cloudflare challenges. Search returns the marketing layer while the real answers sit in comment threads. A video's content is locked in its audio. And a local model with no search tool can't look up anything at all.

One endpoint fixes all of it. Deploy once to your own Cloudflare account, register one URL and one token, and every agent session gets the same reach.

flowchart LR
    A["<b>Your agents</b><br/>Claude Code · OpenClaw<br/>local LLMs · any MCP client"]
    W["<b>Web-MCP</b><br/>your Cloudflare Worker"]
    A -->|one URL + one token| W
    W --> R["<b>Reddit</b><br/>posts · scored comments<br/>community discovery"]
    W --> X["<b>X</b><br/>posts · replies"]
    W --> Y["<b>YouTube</b><br/>videos · comments<br/><b>transcripts</b>"]
    W --> S["<b>Open web</b><br/>keyword + semantic search"]
    W --> P["<b>Walled pages</b><br/>bot-blocked · JS shells · PDFs"]

What it actually gets you

Ask "what's the best local video model right now?" — a question no search engine answers honestly:

That's information that doesn't exist in any search index, and none of it needed a browser.

Related MCP server: twscrape-twitter-mcp

Start free — three keys, no card, most of the capability

These three cost nothing, ever:

Stop there and you already have: community discovery · Reddit + YouTube search with engagement signals · full scored comment threads · keyword web search · and page fetching for every site that isn't actively blocking you.

That's four of the five tools working, for £0.

Then add what you actually need

MCP_AUTH_TOKEN is the only required secret. Every source is independent, and the server only advertises tools whose providers are configured — your agent is never offered something it can't use, and never discovers that by failing.

The tools

Reach the discussion — where practitioners actually talk

Reach the page — anything that blocks you

Reach the open web — for everything else

Five tools. No LLM inside the worker, no ranking, no summarising — raw signals go straight to the calling model, which does the thinking. Sprawling tool surfaces bloat context windows and confuse routing.

How it works

fetch_page escalates — free first, paid only when necessary

flowchart TD
    U["fetch_page(url)"] --> V{"Video URL?"}
    V -->|yes| T["Transcript<br/><i>1 credit</i>"]
    V -->|no| D["Direct fetch + HTMLRewriter<br/><b>FREE</b>"]
    D --> C{"Blocked, JS shell,<br/>or PDF?"}
    C -->|no| OK["Clean text ✓"]
    C -->|yes| F["FireCrawl stealth proxy<br/><i>1–5 credits</i>"]
    F --> OK2["Clean text ✓"]
    F --> ERR["Explicit error —<br/><b>never</b> the challenge page's HTML"]

That last box matters more than it looks. The common failure mode elsewhere is an agent confidently summarising a Cloudflare interstitial it mistook for content. Web-MCP tells you it failed.

Keyword search can merge two independent engines

flowchart LR
    Q["keyword query"] --> TA["Tavily<br/><i>page text</i>"]
    Q --> BR["Brave<br/><i>publication dates</i>"]
    TA --> M["dedupe + merge"]
    BR --> M
    M --> RES["One result carrying<br/><b>date AND full text</b><br/><i>neither engine returns alone</i>"]

Their indexes are largely disjoint, so recall roughly doubles. Every result lists which engines found it — cross-index agreement is handed to your model as a signal, not baked into a ranking here.

Everything is cached in KV and every paid provider has a daily spend ceiling that fails with a readable message rather than a surprise bill.

Extending it, or want the reasoning behind the design? ARCHITECTURE.md covers the principles, how to add a provider, and the per-provider gotchas that aren't in any vendor's docs.

Setup

The fast path — clone the repo, then one command:

./setup.sh

It generates your token, creates your KV namespace, deploys, and — if you want Discord — offers to install the companion on that machine right away, passing the token along so you never paste it twice. Re-run any time; existing tokens and namespaces are kept unless you say otherwise.

1. Generate your token — the only credential your clients need:

echo "webmcp_$(openssl rand 32 | base64 | tr '+/' '-_' | tr -d '=\n')"

2. Grab the provider keys you want (all optional):

At reddit.com/prefs/apps: create another app… → type script → redirect URI http://localhost (unused). Copy the client ID (under the app name) and secret.

Set REDDIT_USER_AGENT in wrangler.toml to identify your app — e.g. cf-worker:web-mcp:v1 (by /u/yourusername). Reddit requires a unique, descriptive UA.

console.cloud.google.com → create/pick a project → APIs & Services → Library → "YouTube Data API v3" → Enable → Credentials → Create Credentials → API key.

Empty API-restrictions dropdown? The API isn't enabled on that project yet — the dropdown only lists enabled APIs.

Quota: search costs ~100× more than metadata or comments, so only search is scarce and only search is budgeted here.

Configure this if your MCP client has no web search of its own (a local LLM, for instance).

• Tavily — 1,000 credits/month free, native domain filters, full page text, no publication dates

• Brave — independent index, publication dates, no free tier since Feb 2026

KEYWORD_SEARCH_PROVIDER = auto | tavily | brave | both. Defaults to both (parallel + merged). For free-tier-only running, set it to auto.

dashboard.exa.ai. Matches meaning, not keywords — and does find-similar-by-URL, which nothing else here can do.

Complementary to keyword search, not a replacement: semantic is measurably weaker at plain factual lookups.

twitterapi.io — instant key, no X developer approval, ~$0.15/1k tweets. A small free allowance lets you test, throttled to 1 request/5s until you add credits.

firecrawl.dev. fetch_page works without it on open pages — it only escalates at a genuine wall. 1 credit, up to 5 with the enhanced proxy.

supadata.ai, 100/month free. Required because the official YouTube API cannot return transcripts: captions.download needs the video owner's OAuth, and auto-generated captions aren't exposed at all.

3. Run locally (optional):

npm install
cp .env.example .dev.vars   # fill in whichever keys you have
npm run dev                 # http://localhost:8787/mcp

4. Deploy:

export CLOUDFLARE_ACCOUNT_ID=<your-account-id>   # npx wrangler whoami

npx wrangler kv namespace create KV
# paste the returned id into wrangler.toml

echo "<value>" | npx wrangler secret put MCP_AUTH_TOKEN   # required
# ...then one line per provider key you're using

npm run deploy

The free Workers plan covers personal use comfortably.

5. Connect your client:

The server is stateless Streamable HTTP — any MCP client works with just a URL and a header.

Claude Code:

claude mcp add --transport http --scope user web-mcp \
  https://web-mcp.<your-subdomain>.workers.dev/mcp \
  --header "Authorization: Bearer <MCP_AUTH_TOKEN>"

Restart afterwards — Claude Code loads MCP servers at startup.

Jan, or any client with an "add MCP server" dialog:

Leave Command / Args / Env empty — those are for stdio servers.

The server speaks both halves of Streamable HTTP: plain JSON responses, or SSE when your client asks for text/event-stream (including the GET stream clients open for server-initiated messages). Long-running calls emit keepalive heartbeats, so transcripts and bot-protection escalations won't trip your client's idle timeout.

Clients that only speak stdio can bridge with mcp-remote: command npx, args -y mcp-remote <url> --header Authorization:${AUTH_HEADER}, and AUTH_HEADER=Bearer <token> in env. The colon has no space after it deliberately — several clients mangle spaces inside args when invoking npx.

Good to know

• Search technique matters. Reddit, X and YouTube match keywords, not meaning — a natural-language question silently degrades into "merely popular posts". Short keyword queries win, and scoping to a subreddit is the biggest lever. The tool descriptions teach your model this, so you don't have to.

• Geo-targeted fetch. Pass country (ISO 3166-1 alpha-2, e.g. US, DE, JP) to fetch_page to read a page as seen from that country — for geo-restricted or geo-varying content. Country-level only, and it forces the paid FireCrawl tier (the free fetch uses the worker's own IP and can't be geo-located).

• Transcripts cost 1 credit using existing captions. generate: true transcribes audio instead, billed per minute — a 60-minute talk ≈ 120 credits. Opt-in only.

• TwitterAPI.io is third-party — ~30× cheaper than the official X API, but grey-market. src/providers/ is abstracted, so swapping is contained.

• Privacy. Single-user by design: one shared token, no accounts. Don't publish your worker URL and token together.

Discord companion

Discord is shipped as platform: "discord" on social_search — but it needs one extra piece, because Discord's Cloudflare edge blocks every request whose TLS handshake originates from a Cloudflare Worker (403 regardless of exit IP or headers). The fix: a tiny companion process on any always-on machine you own (Mac, Linux box, NAS). It holds an outbound WebSocket to your worker; Discord searches are relayed through it and executed with that machine's real residential/network IP and TLS stack.

MCP clients ──▶ CF Worker ──▶ Durable Object ──WebSocket──▶ companion (your box) ──▶ Discord

Multiple companions can connect simultaneously — requests rotate across them and fail over if one goes down. Any companion can serve any request, so install it on every machine you have.

Setup — the companion needs the worker to already have DISCORD_USER_TOKEN set (see below for how to get it). Then, on each machine you want as a companion (Linux or macOS):

curl -fsSL https://raw.githubusercontent.com/mike-nott/web-mcp/main/companion/install/install.sh | bash
#   — asks for your MCP_AUTH_TOKEN (the same token your MCP clients use;
#     there is no third secret to create), then installs as systemd
#     (Linux) or launchd (macOS)

Running ./setup.sh in a fresh clone does all of this — worker deploy, Discord token, and the companion on that machine — in one pass, passing the token along so it is never typed twice. The curl | bash form above is for adding more machines later.

Verify from any MCP client: social_search { query: "test", platform: "discord", time: "month" }

Getting DISCORD_USER_TOKEN: log into a dedicated Discord account (never your primary) in a browser, open DevTools → Network → click any discord.com/api request → Request Headers → copy the Authorization value. It rotates if that account changes its password or hits "log out all devices" — the tool's 401 message names the fix. A search is only that account's own servers, kept low-volume by a 100/day cap; there are no reactions in search results, so results carry no score.

What the companion is and isn't: it refuses anything that is not a GET under discord.com/api/v10, needs only Node ≥18 (zero npm dependencies), and never writes the Discord token to disk — the worker sends it per-request over the authenticated WebSocket. If no companion is connected, Discord search returns a readable error telling you to start one; every other platform is unaffected.

Issues and ideas welcome.

License

MIT