website-content-mcp

Give an AI agent a reliable, site-scoped view of a website: clean content now, and deterministic evidence when it changes. It is a free, self-hosted MCP server—not a general-purpose scraper—so the agent only reads the website you configure, respects robots.txt, and can show the hash and HTTP validators behind a result.

What you can do

• Monitor a site with evidence. Read selected competitor or market pages on a schedule in a separate workflow, then compare content hashes, ETags, prices and availability facts before alerting a human or agent.

• Keep an agent current on your own site. Let a support, sales or content agent list pages, fetch the current Markdown, and answer from what is live rather than from a stale upload.

• Build a research foundation. Turn a site's sitemap and page content into a bounded, attributable input for briefs, audits, catalog analysis or change review—without granting the agent arbitrary web-fetch access.

Unlike Firecrawl and generic scraping APIs, this project is site-scoped, robots-compliant, deterministic about change evidence, self-hosted and free. It is the content-access layer; scheduling, snapshots, diffs and alerts belong in the workflow you build around it.

Related MCP server: Crawl4AI MCP Server

Quick start: one minute to useful search

Use stdio for a local desktop agent. Add this to its MCP configuration:

{
  "mcpServers": {
    "website-content": {
      "command": "npx",
      "args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

Then ask the agent to call content_refresh once. When it finishes, ask it to call content_search for a topic. The cache begins empty, so search has no pages to search until you refresh it or fetch pages individually.

Use HTTP when several approved remote agents need a shared endpoint. The public HTTP transport is deliberately read-only: populate search with STARTUP_REFRESH_LIMIT or a restricted operator endpoint rather than exposing content_refresh anonymously.

A real agent flow

Prompt: “Summarize what is new on astraeus.ie.”

1. The agent calls content_list_pages to discover the site structure.

2. It calls content_get_page for the relevant current pages.

3. It summarizes the returned Markdown and cites the source URLs, retaining contentHash, ETag and fetchedAt for the next comparison.

For a recurring change workflow, persist those deterministic fields outside the MCP server, re-read the same pages later, and only ask AI to classify or summarize a verified difference.

Features

• Clean extraction — HTML → markdown via Mozilla Readability + Turndown (real DOM parsing, never regex). Extraction runs once per page and is cached.

• Deterministic change evidence — each page includes a SHA-256 of the complete normalized markdown plus upstream ETag / Last-Modified validators when available.

• Commerce-aware metadata — bounded schema.org Product / Offer JSON-LD is returned as structured product, SKU, GTIN, brand, price, currency and availability facts.

• Discovery — page listing from sitemap.xml, sitemaps advertised in robots.txt, or a configured page list.

• Disk cache — fetched pages cached with a configurable TTL and a size bound; reads prefer cache, then a conditional revalidation, then stale-on-error.

• Polite by default — respects robots.txt disallow rules, rate-limits to ~1 request/second, honours Retry-After, and sends If-None-Match / If-Modified-Since so unchanged pages cost a 304.

• Scoped to one site — fetches are refused for any host outside the configured site.

• Two transports — Streamable HTTP and stdio.

What a page result looks like

content_get_page returns clean Markdown plus compact metadata that another workflow can retain for comparison. Fields are omitted when the source does not provide them.

{
  "url": "https://shop.example/products/blue-widget",
  "title": "Blue Widget",
  "canonicalUrl": "https://shop.example/products/blue-widget",
  "markdown": "# Blue Widget\n\nA useful blue widget.",
  "contentHash": "8f3c...64-character-sha256...a91d",
  "contentLength": 38,
  "truncated": false,
  "fetchedAt": "2026-08-16T16:00:00.000Z",
  "fromCache": false,
  "etag": "W/\"widget-v4\"",
  "lastModified": "Sat, 16 Aug 2026 12:00:00 GMT",
  "products": [{
    "name": "Blue Widget",
    "sku": "BW-1",
    "brand": "Widget Co",
    "offers": [{ "price": "19.99", "priceCurrency": "EUR", "availability": "https://schema.org/InStock" }]
  }]
}

Tools

The cache starts empty, so content_search finds nothing until pages have been fetched. Run content_refresh once after starting the server (it is rate-limited to ~1 request/second, so a 50-page pass takes about a minute), or fetch pages individually with content_get_page.

Requirements

• Node.js 22+

Install

MCP clients can run the stdio transport without cloning the repository:

{
  "mcpServers": {
    "website-content": {
      "command": "npx",
      "args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

For source development:

git clone https://github.com/SarutobiSasuke8/website-content-mcp.git
cd website-content-mcp
npm install
npm run build

Configuration

Configuration is via environment variables (see .env.example):

Run

Streamable HTTP (default transport):

SITE_BASE_URL=https://example.com npm start
# → website-content-mcp listening on http://127.0.0.1:3215/mcp (site: https://example.com/)

The MCP endpoint is POST /mcp; a plain health probe is available at GET /healthz.

Streamable HTTP is deliberately public read-only by default. It omits content_refresh, refuses content_get_page(refresh: true), and redacts the local cache path from MCP health output. Normal uncached reads can still reach the configured public site and should be rate-limited at the reverse proxy. Set STARTUP_REFRESH_LIMIT to populate search without exposing a public cache-warming tool.

For a loopback Node process behind nginx/systemd, see docs/production-deployment.md.

stdio (for local MCP clients):

SITE_BASE_URL=https://example.com npm run start:stdio

Example MCP client entry from a local source checkout (stdio):

{
  "mcpServers": {
    "website-content": {
      "command": "node",
      "args": ["/path/to/website-content-mcp/dist/src/stdio.js"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

Development

npm run dev        # HTTP transport with --watch
npm run dev:stdio  # stdio transport
npm run check      # typecheck + lint + build + test

Testing

• Unit tests cover HTML→markdown conversion, the disk cache (TTL and eviction), robots parsing, sitemap parsing, the fetcher (size cap, conditional headers, Retry-After retries, rate limiting) and the content service (host scoping, robots enforcement, truncation, cache warming, 304 revalidation, stale fallback). All offline, against a local fixture and a stubbed fetch.

• A live integration test runs against https://example.com, exercising health, list_pages, get_page, and search.

npm test

Live checks are opt-in. To validate real deployment targets, set RUN_LIVE_TESTS=1 and provide a comma-separated LIVE_SITE_URLS list of sites you own or operate before running npm test, for example:

RUN_LIVE_TESTS=1 LIVE_SITE_URLS=https://astraeus.ie npm test

Without LIVE_SITE_URLS, the live check falls back to https://example.com.

Security & etiquette

• Binds to 127.0.0.1 by default.

• No authentication and no API keys — intended for public content only.

• Scoped to the configured site: a URL on any other host is refused, so the server cannot be used as a general-purpose fetcher. Widen deliberately with SITE_ALLOWED_HOSTS.

• Redirects are followed manually and every destination is checked against the same host allowlist before a network request is made.

• Streamable HTTP defaults to a read-only tool surface. Keep HTTP_ALLOW_REFRESH=false for anonymous deployments.

• Respects robots.txt, fetched and enforced per origin; disallowed paths are refused.

• Rate-limited to ~1 request/second against the target site.

• Response bodies are capped at FETCH_MAX_BYTES and the cache at CACHE_MAX_ENTRIES.

• Never logs full page bodies (only URLs, status codes, and sizes).

What this server is not

This project is the content-access foundation for monitoring workflows; it is not itself a scheduler or alerting service. Competitor monitoring additionally needs durable snapshots, deterministic diffs, a scheduler, notifications and an evidence-retention policy. AI can classify and summarize verified changes, but should not replace the underlying hashes, fields and source records.

License

MIT