crawler-mcp

A web-crawler MCP server, built in the same style as weather-mcp (TypeScript + @modelcontextprotocol/sdk + Express + Zod).

It fetches pages with a fast axios + cheerio path and automatically falls back to a Playwright headless browser when a page needs JavaScript to render. Designed to be used as a Claude connector (local stdio or remote HTTP).

Tools

Each tool accepts a render option: auto (default — static first, browser if needed), static (never use a browser), or browser (always render with Playwright).

Related MCP server: turbowebfetch

Setup

cd crawler-mcp
npm install

# Optional — only needed for JavaScript-heavy / SPA sites:
npx playwright install chromium

cp .env.example .env   # then edit if you want

Run

# Local (stdio) — for Claude Desktop:
npm start

# Remote (HTTP) — for a custom/remote connector:
npm run start:http     # serves http://localhost:3001/mcp

Use it as a Claude connector

Claude Desktop (local, stdio)

Add to your claude_desktop_config.json (Settings → Developer → Edit Config):

{
  "mcpServers": {
    "crawler": {
      "command": "npx",
      "args": ["tsx", "D:\\MCP\\crawler-mcp\\server.ts"]
    }
  }
}

Restart Claude Desktop. The four crawler tools appear under the connectors menu.

Remote HTTP connector (claude.ai / Desktop "Add custom connector")

1. Start the server in HTTP mode and expose it over HTTPS (e.g. behind a reverse proxy or a tunnel such as cloudflared / ngrok):

   npm run start:http

2. In Claude → Settings → Connectors → Add custom connector, enter the URL:

   https://your-host/mcp

Remote connectors on claude.ai generally require a public HTTPS URL. http://localhost:3001/mcp works for local testing tools but not for the hosted claude.ai web app.

Configuration (.env)

Notes

• Fetched pages are cached in memory for 5 minutes to avoid refetching during a crawl.

• crawl_site is bounded (max_depth ≤ 3, max_pages ≤ 50) and adds a politeness delay between requests.

• Playwright is an optional dependency — if it isn't installed, render: auto still works for static sites and render: browser returns a clear error.

• This server does not currently parse robots.txt; crawl responsibibly and only sites you are authorized to crawl.