webmesh-mcp

A token-economical web browsing, scraping, and crawling suite delivered as a standard MCP server. Any MCP-capable agent (Claude, Cursor, Cline, etc.) can use it as ordinary tools with no bespoke integration.

Why this is cheaper than a full browser loop

Related MCP server: krwl3r

Tools

Installation

npm install -g webmesh-mcp
# or use directly with npx (no global install needed):
npx webmesh-mcp

Chromium (optional — only needed for JS-rendered pages)

webmesh-mcp uses playwright-core and does not bundle a browser. You have three options:

A) Install Playwright's managed Chromium (simplest):

npx playwright install chromium

B) Use your system Chrome / Edge — set executablePath in your MCP config (see below).

C) Connect to a running browser — Playwright supports CDP attach; pass --cdp-endpoint flags in args if you want to hook into an already-running instance.

Static pages (most blogs, docs, GitHub, npm, etc.) never trigger the browser path at all.

Connecting to your agent

Add to your MCP config (e.g. ~/.claude/claude_desktop_config.json, .cursor/mcp.json, etc.):

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["webmesh-mcp"]
    }
  }
}

Or, if you prefer to run from source:

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["tsx", "/absolute/path/to/webmesh-mcp/src/index.ts"]
    }
  }
}

Agent Skill Integration

This repository ships with a pre-configured Agent Skill in skills/web-agent/SKILL.md.

It provides AI coding assistants (Claude Code, Antigravity, Cursor, Windsurf, Gemini CLI, etc.) with a complete decision tree, trigger keywords, and parameter guidance for using webmesh-mcp tools token-efficiently.

How to use the skill in your project

Copy or symlink the skills/ folder into your AI assistant's skills directory:

• Claude / Antigravity / Gemini CLI: Place in .agents/skills/web-agent/SKILL.md or ~/.gemini/config/skills/web-agent/SKILL.md

• Cursor: Copy contents to .cursor/rules/web-agent.mdc

• Windsurf: Copy contents to .windsurfrules

Tool reference

web_scrape

web_check

web_diff

Returns changed: null on the first call (nothing to compare against), true/false on subsequent calls.

web_interact

Action shape:

{ type: "click" | "fill" | "select" | "press" | "waitFor", selector?: string, value?: string, timeoutMs?: number }

web_session_close

web_crawl

Full markdown for every visited page is cached in SQLite for web_crawl_get_page.

web_crawl_get_page

Environment variables

Both the SQLite watch/crawl cache (cache/watch.sqlite) and session state files (sessions/) live under MCP_WEB_AGENT_DATA_DIR. Override it to control where runtime data is stored.

Security

SSRF protection

All outbound fetches — static HTTP, browser navigation, and crawl link-following — are validated against a blocklist before any network connection is made. Requests to the following are rejected:

• Loopback addresses (127.x.x.x, ::1, localhost)

• Private RFC-1918 ranges (10.x, 172.16–31.x, 192.168.x)

• Link-local and cloud metadata addresses (169.254.x.x, including the AWS/GCP/Azure instance metadata endpoint)

• Multicast and reserved ranges (224.x and above)

• Non-HTTP/HTTPS schemes

This matters because web_crawl follows links automatically, and scraped page content can contain prompt-injection attempts that try to redirect the next fetch to an internal address. The guard is applied at every fetch entry point so ignoreRobots: true does not bypass it.

DNS rebinding caveat: The guard resolves hostnames and checks the returned IPs, but the actual TCP connection is made moments later by fetch/Playwright using their own DNS resolution. A determined attacker with control of a DNS record could exploit this window. This is the known residual risk; full mitigation requires IP pinning at the HTTP-client level and is a planned follow-up.

Architecture

index.ts (MCP server, stdio transport)
├── tools/scrape.ts        — web_scrape
├── tools/check.ts         — web_check
├── tools/diff.ts          — web_diff
├── tools/interact.ts      — web_interact
├── tools/crawl.ts         — web_crawl
├── tools/crawlGetPage.ts  — web_crawl_get_page
├── tieredFetch.ts         — static HTTP → browser escalation
├── browserPool.ts         — singleton Chromium process (playwright-core)
├── ssrfGuard.ts           — SSRF protection (blocks private/reserved addresses)
├── extract.ts             — HTML → clean markdown / plain text / schema JSON
├── hostGate.ts            — robots.txt parser + per-host rate-limiting queue
├── sessions.ts            — disk-backed storageState persistence
├── cache.ts               — SQLite: watch hashes + crawled page markdown
└── constants.ts           — shared USER_AGENT, DATA_DIR

Development

git clone https://github.com/creatorpiyush/webmesh-mcp
cd webmesh-mcp
npm install
npx playwright install chromium    # optional, for browser-tier testing

npm run dev        # run MCP server in dev mode (tsx, no compile step)
npm run demo -- https://example.com
npm test           # run the full integration test suite
npm run typecheck  # tsc --noEmit, no output files
npm run format     # format code with prettier
npm run build      # compile to dist/

License

MIT — see LICENSE.