webmesh-mcp

A token-economical web browsing, scraping, and crawling suite delivered as a standard MCP server. Any MCP-capable agent (Claude, Cursor, Cline, etc.) can use it as ordinary tools with no bespoke integration.

Why this is cheaper than a full browser loop

Tools

Installation

npm install -g webmesh-mcp
# or use directly with npx (no global install needed):
npx webmesh-mcp

Chromium (optional — only needed for JS-rendered pages)

webmesh-mcp uses playwright-core and does not bundle a browser. You have three options:

A) Install Playwright's managed Chromium (simplest):

npx playwright install chromium

B) Use your system Chrome / Edge — set executablePath in your MCP config (see below).

C) Connect to a running browser — Playwright supports CDP attach; pass --cdp-endpoint flags in args if you want to hook into an already-running instance.

Static pages (most blogs, docs, GitHub, npm, etc.) never trigger the browser path at all.

Connecting to your agent

Add to your MCP config (e.g. ~/.claude/claude_desktop_config.json, .cursor/mcp.json, etc.):

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["webmesh-mcp"]
    }
  }
}

Or, if you prefer to run from source:

{
  "mcpServers": {
    "web-agent": {
      "command": "npx",
      "args": ["tsx", "/absolute/path/to/webmesh-mcp/src/index.ts"]
    }
  }
}

Agent Skill Integration

This repository ships with a pre-configured Agent Skill in skills/web-agent/SKILL.md.

It provides AI coding assistants (Claude Code, Antigravity, Cursor, Windsurf, Gemini CLI, etc.) with a complete decision tree, trigger keywords, and parameter guidance for using webmesh-mcp tools token-efficiently.

How to use the skill in your project

Copy or symlink the skills/ folder into your AI assistant's skills directory:

• Claude / Antigravity / Gemini CLI: Place in .agents/skills/web-agent/SKILL.md or ~/.gemini/config/skills/web-agent/SKILL.md

• Cursor: Copy contents to .cursor/rules/web-agent.mdc

• Windsurf: Copy contents to .windsurfrules

Tool reference

web_scrape

web_check

web_diff

Returns changed: null on the first call (nothing to compare against), true/false on subsequent calls.

web_interact

Action shape:

{ type: "click" | "fill" | "select" | "press" | "waitFor", selector?: string, value?: string, timeoutMs?: number }

web_session_close

web_crawl

Full markdown for every visited page is cached in SQLite for web_crawl_get_page.

web_crawl_get_page

Environment variables

Both the SQLite watch/crawl cache (cache/watch.sqlite) and session state files (sessions/) live under MCP_WEB_AGENT_DATA_DIR. Override it to control where runtime data is stored.

Security

SSRF Protection

All outbound fetches — static HTTP, browser navigation, intermediate HTTP redirects, and crawl link-following — are strictly validated against an SSRF blocklist before network connections are established.

• Protocol Restriction: Rejects all non-HTTP/HTTPS schemes.

• Loopback Addresses: 127.x.x.x, IPv6 ::1, localhost, IPv6 unspecified :: / 0:0:0:0:0:0:0:0.

• Private RFC-1918 & Unique Local IPv6: 10.x, 172.16–31.x, 192.168.x, fc00::/7, fd00::/7.

• Link-Local & Cloud Metadata: 169.254.x.x (AWS/GCP/Azure IMDS), fe80::/10.

• Special & Reserved Ranges: Multicast (224.x), Site-Local (fec0::/10), Documentation (2001:db8::/32), and IPv4-mapped IPv6 formats (::ffff:x.x.x.x).

• HTTP Redirect Hardening: Static fetch enforces manual redirect validation loops up to 5 hops, checking ssrfGuard.assertPublicUrl() on every intermediate Location header before following.

• Browser Route Interception: Chromium contexts attach route interceptors (page.route("**/*")) to block subresource requests or redirects targeting private IP space.

Architecture

For a detailed technical architecture and end-to-end data flow specification, see Architecture.md.

index.ts (MCP server, stdio transport)
├── tools/scrape.ts        — web_scrape
├── tools/check.ts         — web_check
├── tools/diff.ts          — web_diff
├── tools/interact.ts      — web_interact
├── tools/crawl.ts         — web_crawl
├── tools/crawlGetPage.ts  — web_crawl_get_page
├── tieredFetch.ts         — static HTTP → browser escalation (manual redirect validation)
├── browserPool.ts         — singleton Chromium process + route interceptor (playwright-core)
├── ssrfGuard.ts           — SSRF protection (blocks private/reserved IPv4 & IPv6 addresses)
├── extract.ts             — HTML → clean markdown / plain text / schema JSON
├── hostGate.ts            — robots.txt parser (regex escaped) + per-host rate-limiting queue
├── sessions.ts            — disk-backed storageState persistence
├── cache.ts               — SQLite: watch hashes + crawled page markdown
└── constants.ts           — shared USER_AGENT, DATA_DIR

Development

git clone https://github.com/creatorpiyush/webmesh-mcp
cd webmesh-mcp
npm install
npx playwright install chromium    # optional, for browser-tier testing

npm run dev        # run MCP server in dev mode (tsx, no compile step)
npm run demo -- https://example.com
npm test           # run the full integration test suite
npm run typecheck  # tsc --noEmit, no output files
npm run format     # format code with prettier
npm run build      # compile to dist/

License

MIT — see LICENSE.