website-content-mcp

An MCP server that exposes a website's content in agent-readable structured form. Instead of scraping raw HTML, agents (Claude, Codex, or any MCP client) pull clean markdown and structured metadata through a small set of tools.

The server fetches pages from a configurable target site, strips them to clean markdown, caches results to disk, respects robots.txt, and rate-limits requests to be a polite citizen of the web.

Features

• Clean extraction — HTML → markdown via Mozilla Readability + Turndown (real DOM parsing, never regex).

• Discovery — page listing from sitemap.xml, sitemaps advertised in robots.txt, or a configured page list.

• Disk cache — fetched pages cached with a configurable TTL; reads prefer cache, then network, then stale-on-error.

• Polite by default — respects robots.txt disallow rules and rate-limits to ~1 request/second.

• Two transports — Streamable HTTP and stdio.

Related MCP server: Crawl4AI MCP Server

Tools

Requirements

• Node.js 22+

Install

git clone https://github.com/SarutobiSasuke8/website-content-mcp.git
cd website-content-mcp
npm install
npm run build

Configuration

Configuration is via environment variables (see .env.example):

Run

Streamable HTTP (default transport):

SITE_BASE_URL=https://example.com npm start
# → website-content-mcp listening on http://127.0.0.1:3215/mcp (site: https://example.com/)

The MCP endpoint is POST /mcp; a plain health probe is available at GET /healthz.

stdio (for local MCP clients):

SITE_BASE_URL=https://example.com npm run start:stdio

Example MCP client entry (stdio):

{
  "mcpServers": {
    "website-content": {
      "command": "node",
      "args": ["/path/to/website-content-mcp/dist/src/stdio.js"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

Development

npm run dev        # HTTP transport with --watch
npm run dev:stdio  # stdio transport
npm run check      # typecheck + lint + build + test

Testing

• Unit tests cover HTML→markdown conversion, the disk cache/TTL logic, robots parsing, and sitemap parsing (all offline, using a local fixture).

• A live integration test runs against https://example.com, exercising health, list_pages, get_page, and search.

npm test

Security & etiquette

• Binds to 127.0.0.1 by default.

• No authentication and no API keys — intended for public content only.

• Respects robots.txt; disallowed paths are refused.

• Rate-limited to ~1 request/second against the target site.

• Never logs full page bodies (only URLs, status codes, and sizes).

License

MIT