Skip to main content
Glama
website-content-mcp

website-content-mcp

website-content-mcp

Give an AI agent a reliable, site-scoped view of a website: clean content now, and deterministic evidence when it changes. It is a free, self-hosted MCP server (not a general-purpose scraper), so the agent only reads the website you configure, respects robots.txt, and can show the hash and HTTP validators behind a result.

What you can do

  • Monitor a site with evidence. Read selected competitor or market pages on a schedule in a separate workflow, then compare content hashes, ETags, prices and availability facts before alerting a human or agent.

  • Keep an agent current on your own site. Let a support, sales or content agent list pages, fetch the current Markdown, and answer from what is live rather than from a stale upload.

  • Build a research foundation. Turn a site's sitemap and page content into a bounded, attributable input for briefs, audits, catalog analysis or change review without granting the agent arbitrary web-fetch access.

Unlike Firecrawl and generic scraping APIs, this project is site-scoped, robots-compliant, deterministic about change evidence, self-hosted and free. It returns deterministic evidence only (hashes, validators, line diffs and structured facts). No language model runs inside the server. Scheduling and alerting stay outside it, and a companion watcher command is included: see docs/watcher.md.

Related MCP server: mdb-mcp

Use stdio for a local desktop agent. Add this to its MCP configuration:

{
  "mcpServers": {
    "website-content": {
      "command": "npx",
      "args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

Then ask the agent to call content_refresh once. When it finishes, ask it to call content_search for a topic. The cache begins empty, so search has no pages to search until you refresh it or fetch pages individually.

Use HTTP when several approved remote agents need a shared endpoint. The public HTTP transport is deliberately read-only: populate search with STARTUP_REFRESH_LIMIT or a restricted operator endpoint rather than exposing content_refresh anonymously.

A real agent flow

Prompt: “Summarise what is new on astraeus.ie.”

  1. The agent calls content_list_pages to discover the site structure.

  2. It calls content_get_page for the relevant current pages.

  3. It summarises the returned Markdown and cites the source URLs, retaining contentHash, ETag and fetchedAt for the next comparison.

For a recurring change workflow, content_diff keeps the previous version of a page and returns a bounded line diff with old and new hashes, and the watcher runs a whole watchlist and writes a Markdown report of only the pages that changed. Ask AI to classify or summarise a verified difference, not to find it.

More examples

Prompt: “Has the Starter price on the competitor's pricing page changed since last week?”

The agent calls content_diff for the page. On the first call it stores a baseline and says so. On later calls it reports changed, the old and new contentHash and fetchedAt, the added and removed lines, and a one-line summary such as “Amounts no longer shown: €49; now shown: €59.”

Prompt: “What does this page offer and what does it ask visitors to do?”

The agent calls content_extract_facts and reads the outline, calls to action, pricing signals (amount, plan, billing period) and schema.org data. Anything the page does not state is left out rather than guessed.

Prompt: “Give me a site-health pass for my own site.”

The agent calls content_site_health and gets findings grouped as critical, high, medium and low: non-200 sitemap pages, broken internal links, missing or duplicate titles, canonicals that point elsewhere, missing H1s and more. It is bounded and polite, so keep it to sites you own.

Offline regression tests

Run npm ci followed by npm run check for typecheck, lint and tests. The saved synthetic HTML under tests/fixtures/ covers article formatting, relative links, code blocks, Unicode spacing, client-rendered shells and malformed structured data. A fake fetcher also checks response truncation, content hashes and cache reuse. No live website or API key is needed. The existing live-site integration test remains opt-in and is skipped by default.

Features

  • Clean extraction: HTML -> markdown via Mozilla Readability + Turndown (real DOM parsing, never regex). Extraction runs once per page and is cached.

  • Deterministic change evidence: each page includes a SHA-256 of the complete normalised markdown plus upstream ETag / Last-Modified validators when available.

  • Extraction-quality evidence: results state whether content came from Readability, the cleaned body, plain text or metadata-only fallback, with actionable warnings for client-rendered shells.

  • Commerce-aware metadata: bounded schema.org Product / Offer JSON-LD is returned as structured product, SKU, GTIN, brand, price, currency and availability facts.

  • Change evidence: content_diff stores the previous normalized Markdown per URL in a bounded store (retention and entry limits) and returns changed or not, old and new hashes and fetch times, a compact line diff and a deterministic summary. First sight, unchanged pages, 304 responses and stale-on-error fallbacks are each reported explicitly.

  • Structured facts: content_extract_facts returns title, description, canonical, outline, calls to action, pricing signals, schema.org types and dated index items. Unknown values are omitted.

  • Plan-level prices: pages with plan cards are read into a table of plans with price or range, currency, billing period, per-unit basis, free and trial flags and listed features. Renames are reported as renames, and prices carry absolute and percentage change. Currencies are never converted.

  • Price history and competitor intelligence: the watcher keeps a bounded price series per page, exports it as CSV, writes a living competitor comparison note and a weekly digest, and emits typed events for new posts, releases, roles, integrations, certifications and positioning changes. All deterministic. See docs/watcher.md.

  • Site health: content_site_health checks titles, descriptions, canonicals, JSON-LD, headings, non-200 sitemap pages, broken internal links and pages missing from the sitemap, ranked by severity.

  • Multiple sites, still allowlisted: tools accept an optional site from SITE_ALLOWLIST. Each site has its own robots.txt rules, cache scope, search scope and request pacing, and redirects to a different site are refused.

  • Discovery: page listing from sitemap.xml, sitemaps advertised in robots.txt, or a configured page list.

  • Disk cache: fetched pages cached with a configurable TTL and a size bound; reads prefer cache, then a conditional revalidation, then stale-on-error.

  • Polite by default: respects robots.txt disallow rules, rate-limits to about 1 request/second, honours Retry-After, and sends If-None-Match / If-Modified-Since so unchanged pages cost a 304.

  • Scoped to the configured sites: fetches are refused for any host outside the base site and the operator's allowlist.

  • Public-network safe by default: allowed hostnames are resolved before every request and redirect hop; private, loopback, link-local and special-use destinations require an explicit operator opt-in.

  • Two transports: Streamable HTTP and stdio.

What a page result looks like

content_get_page returns clean Markdown plus compact metadata that another workflow can retain for comparison. Fields are omitted when the source does not provide them.

{
  "url": "https://shop.example/products/blue-widget",
  "title": "Blue Widget",
  "canonicalUrl": "https://shop.example/products/blue-widget",
  "markdown": "# Blue Widget\n\nA useful blue widget.",
  "contentHash": "8f3c...64-character-sha256...a91d",
  "contentLength": 38,
  "truncated": false,
  "fetchedAt": "2026-08-16T16:00:00.000Z",
  "fromCache": false,
  "extractionMethod": "readability",
  "extractionQuality": "full",
  "sourceTrust": "untrusted-web-content",
  "etag": "W/\"widget-v4\"",
  "lastModified": "Sat, 16 Aug 2026 12:00:00 GMT",
  "products": [{
    "name": "Blue Widget",
    "sku": "BW-1",
    "brand": "Widget Co",
    "offers": [{ "price": "19.99", "priceCurrency": "EUR", "availability": "https://schema.org/InStock" }]
  }]
}

Tools

Tool

Purpose

content_list_pages

List discoverable pages (sitemap → robots.txt sitemaps → configured list). Returns URL, title, last-modified when known.

content_refresh

Walk the discoverable pages and warm the cache so content_search has something to search. Skips robots-disallowed pages. Available on stdio; opt-in on HTTP.

content_get_page

Fetch a page URL, strip to clean markdown, return content + metadata, content hash and any schema.org Product/Offer facts. Supports max_length; forced refresh is disabled on public HTTP by default.

content_extract_facts

Bounded structured facts from a page: title, meta description, canonical, H1 to H3 outline, calls to action, pricing signals, schema.org types and Product/Offer data, and dated blog or news items when the page is an index. Unknown values are omitted.

content_diff

Compare a page with the version stored by the previous call: changed or not, old and new contentHash and fetchedAt, a compact added and removed line diff and a one-line summary. Stores the new version as the baseline. Available where refresh is allowed (stdio, or HTTP with HTTP_ALLOW_REFRESH=true).

content_price_history

Read the stored time series of plan prices for a page (date, plan, price, currency, billing period, unit, content hash) and the latest price of each plan. Never calls the network; history is written by content_diff and by the watcher. Available where refresh is allowed.

content_site_health

Bounded health check of a site's sitemap pages and internal links, grouped by severity. Available where refresh is allowed, because it fetches many pages.

content_search

Keyword search over cached pages. Returns URL, score, snippet and the last completed refresh time.

content_get_sitemap

Return the raw sitemap structure (URLs + last-modified/priority/change-frequency when present).

content_health

Server status: configured site, allowed hosts, cache directory, cache size, last fetch time and last completed refresh time.

On HTTP, up to 50 discoverable pages are warmed after startup and refreshed every 5 minutes by default. Each pass advances through the page list, so larger sites are covered over successive passes. content_search reports last_refresh_at; content_health reports lastRefreshAt. Set STARTUP_REFRESH_LIMIT=0 to disable automatic HTTP refreshes. Stdio users can run content_refresh (about 1 request per second) or fetch pages individually with content_get_page.

Requirements

  • Node.js 22+

Install

MCP clients can run the stdio transport without cloning the repository:

{
  "mcpServers": {
    "website-content": {
      "command": "npx",
      "args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

For source development:

git clone https://github.com/SarutobiSasuke8/website-content-mcp.git
cd website-content-mcp
npm install
npm run build

Configuration

Configuration is via environment variables (see .env.example):

Variable

Required

Default

Description

SITE_BASE_URL

✅

-

The site whose content is exposed.

SITE_SITEMAP_URL

<base>/sitemap.xml

Sitemap location.

SITE_PAGES

-

Comma-separated fallback page list (absolute or base-relative).

SITE_ALLOWLIST

-

Comma-separated hosts (or URLs) that tools may select with the optional site argument, in addition to the base site, which stays the default. Each site gets its own robots.txt rules, cache entries, search scope and request pacing. Redirects to a different site are refused.

SITE_ALLOWED_HOSTS

-

Extra hosts that may be fetched. The base URL's host is always allowed.

CACHE_DIR

.cache

Disk cache directory.

CACHE_TTL_SECONDS

3600

Cache TTL (0 disables caching).

CACHE_MAX_ENTRIES

500

Cache size bound, evicting oldest-first (0 = unbounded).

SNAPSHOT_DIR

<CACHE_DIR>/snapshots

Where content_diff keeps the current and previous normalized Markdown per URL.

SNAPSHOT_RETENTION_DAYS

90

Previous versions and entries not compared for this long are removed.

SNAPSHOT_MAX_ENTRIES

1000

Maximum stored URLs, removing the least recently compared first. 0 disables snapshots.

SNAPSHOT_MAX_MARKDOWN_CHARS

200000

Maximum Markdown characters stored per version. The hash always covers the full text.

FETCH_TIMEOUT_MS

10000

Per-request timeout.

FETCH_MIN_INTERVAL_MS

1000

Minimum spacing between fetches (about 1 request per second).

FETCH_MAX_BYTES

5000000

Hard cap on a single response body.

FETCH_MAX_RETRIES

1

Retries on 429/503, honouring Retry-After.

FETCH_ALLOW_PRIVATE_NETWORK

false

Permit configured hosts to resolve to private/special-use IPs. Use only for deliberate internal-site deployments.

USER_AGENT

website-content-mcp/0.4.0 …

Outbound User-Agent.

HOST

127.0.0.1

HTTP bind host.

PORT

3215

HTTP bind port.

HTTP_ALLOW_REFRESH

false

Expose content_refresh and permit forced origin revalidation over HTTP. Enable only behind an authenticated or restricted reverse proxy. Stdio always permits refresh.

HTTP_RATE_LIMIT_WINDOW_MS

60000

Per-IP request window for /mcp.

HTTP_RATE_LIMIT_MAX_REQUESTS

60

Maximum /mcp requests per IP in each window; excess requests receive 429 and Retry-After.

HTTP_TRUST_PROXY

empty

Comma-separated trusted proxy IPs, CIDRs or aliases such as loopback. X-Forwarded-For is ignored unless a matching proxy is configured.

STARTUP_REFRESH_LIMIT

50

Pages refreshed on startup and each scheduled HTTP pass. Set to 0 to disable automatic refresh.

HTTP_REFRESH_INTERVAL_SECONDS

300

Seconds between scheduled HTTP passes. Set to 0 for startup-only warming.

Run

Streamable HTTP (default transport):

# Using a .env file (recommended):
cp .env.example .env
npm start

# Or passing environment variables inline:
# POSIX / Bash:
SITE_BASE_URL=https://example.com npm start
# PowerShell:
$env:SITE_BASE_URL="https://example.com"; npm start
# -> website-content-mcp listening on http://127.0.0.1:3215/mcp (site: https://example.com/)

The MCP endpoint is POST /mcp; a plain health probe is available at GET /healthz.

Streamable HTTP is deliberately public read-only by default. It omits content_refresh, refuses content_get_page(refresh: true), and redacts the local cache path from MCP health output. Expired pages are revalidated on content_get_page reads. The bounded background refresh keeps search current without exposing a public cache-warming tool. content_search describes the actual HTTP refresh mode and returns the last completed refresh time.

Inbound /mcp requests are limited per client IP. The server uses the socket IP unless HTTP_TRUST_PROXY explicitly lists the reverse proxy, so untrusted X-Forwarded-For values cannot bypass the quota. Behind local nginx, set HTTP_TRUST_PROXY=loopback; use the proxy's specific address or CIDR for other topologies.

For a loopback Node process behind nginx/systemd, see docs/production-deployment.md.

stdio (for local MCP clients):

# POSIX / Bash:
SITE_BASE_URL=https://example.com npm run start:stdio
# PowerShell:
$env:SITE_BASE_URL="https://example.com"; npm run start:stdio

Example MCP client entry from a local source checkout (stdio):

{
  "mcpServers": {
    "website-content": {
      "command": "node",
      "args": ["/path/to/website-content-mcp/dist/src/stdio.js"],
      "env": { "SITE_BASE_URL": "https://example.com" }
    }
  }
}

Troubleshooting

Startup refresh fails or logs DNS errors (ENOTFOUND)

When starting the HTTP transport with STARTUP_REFRESH_LIMIT enabled (default 50), the server performs an initial cache-warming pass from the site sitemap. If SITE_BASE_URL cannot be resolved (for example when offline or targeting an unresolvable hostname), a warning is logged:

Startup cache refresh failed: getaddrinfo ENOTFOUND ...

The server continues running and responds to HTTP and MCP requests. To disable automatic startup warming, set STARTUP_REFRESH_LIMIT=0 in .env.

Missing SITE_BASE_URL error on startup

SITE_BASE_URL is the only mandatory configuration setting. If omitted, startup fails with a validation error:

[ { "code": "invalid_type", "expected": "string", "received": "undefined", "path": [ "SITE_BASE_URL" ], "message": "Required" } ]

Create a .env file from .env.example and configure SITE_BASE_URL before launching.

Keyword search returns empty results

content_search queries only pages present in the local disk cache. On a fresh server instance, the cache begins empty:

  • On stdio: run content_refresh to warm the cache, or fetch individual target pages using content_get_page.

  • On HTTP: allow startup cache warming to complete (STARTUP_REFRESH_LIMIT), or fetch pages individually with content_get_page.

Private IP address or loopback destination refused

Outbound network fetches to private, loopback (127.0.0.1, localhost), link-local, or cloud metadata IP ranges are blocked by default to prevent Server-Side Request Forgery (SSRF). When intentionally pointing the server at an internal service or local test environment, set FETCH_ALLOW_PRIVATE_NETWORK=true in .env.

Pages refused by robots.txt

If content_get_page or content_refresh reports RobotsDisallowedError, the requested URL is restricted by the remote site's robots.txt. The server strictly enforces origin robots policies and does not bypass crawl restrictions.

Development

npm run dev        # HTTP transport with --watch
npm run dev:stdio  # stdio transport
npm run check      # typecheck + lint + build + test

Testing

  • Contract suite: npm run eval:contract checks the tool list, input schemas and golden outputs against a local fixture site.

  • Unit tests cover HTML→markdown conversion, the disk cache (TTL and eviction), robots parsing, sitemap parsing, the fetcher (size cap, conditional headers, Retry-After retries, rate limiting) and the content service (host scoping, robots enforcement, truncation, cache warming, 304 revalidation, stale fallback). All offline, against a local fixture and a stubbed fetch.

  • Round 2 tests cover the unavailable-versus-removed distinction (503, 429, timeouts, 404 once and twice), hostile text in tasks and notes, https-only extra sites, same-host sitemaps, plan extraction and matching, noise rules, confirmation runs, price thresholds, price history and CSV, typed events, optional rendering with a fake renderer, the comparison note and the digest.

  • Feature tests cover the multi-site allowlist (refusals, per-site robots, cache and pacing isolation, redirect refusal), content_diff (first sight, unchanged, 304, stale-on-error, bounds), fact extraction against saved HTML fixtures, site-health checks, the watcher (baseline, no-change idempotence, forced change, dry run, exit codes) and the vault feed against a temporary directory.

  • A live integration test runs against https://example.com, exercising health, list_pages, get_page, and search.

npm test

Live checks are opt-in. To validate real deployment targets, set RUN_LIVE_TESTS=1 and provide a comma-separated LIVE_SITE_URLS list of sites you own or operate before running npm test, for example:

RUN_LIVE_TESTS=1 LIVE_SITE_URLS=https://astraeus.ie npm test

Without LIVE_SITE_URLS, the live check falls back to https://example.com.

Dogfood monitoring

The source repository includes a bounded snapshot-and-diff proof harness (the older, hash-only precursor of the watcher). It stores hashes, validators, extraction quality and warnings without retaining page bodies:

npm run dogfood
npm run dogfood

The first run establishes baselines and the second reports unchanged or changed pages. See docs/dogfood-monitoring.md. Scheduling, production retention and alert routing remain outside the core.

The published CLI can run the same bounded proof without a source checkout:

npx -y -p @sarutobi-sasuke/website-content-mcp website-content-dogfood

Security & etiquette

  • Binds to 127.0.0.1 by default.

  • No authentication and no API keys; intended for public content only.

  • Scoped to the configured site: a URL on any other host is refused, so the server cannot be used as a general-purpose fetcher. Widen deliberately with SITE_ALLOWED_HOSTS.

  • Redirects are followed manually and every destination is checked against the same host allowlist before a network request is made.

  • DNS results are checked before each request and redirect hop; private, loopback, link-local and special-use destinations are refused unless FETCH_ALLOW_PRIVATE_NETWORK=true is explicitly set by the operator.

  • Returned page text is labelled untrusted-web-content; clients must treat it as source data, never as instructions that override their own policy.

  • Streamable HTTP defaults to a read-only tool surface. Keep HTTP_ALLOW_REFRESH=false for anonymous deployments.

  • Respects robots.txt, fetched and enforced per origin; disallowed paths are refused.

  • Rate-limited to about 1 request per second against the target site.

  • Response bodies are capped at FETCH_MAX_BYTES and the cache at CACHE_MAX_ENTRIES.

  • Never logs full page bodies (only URLs, status codes, and sizes).

What this server is not

The server is the content-access and evidence layer. It is not a scheduler or an alerting service, and nothing in it calls a language model. content_diff gives per-page change evidence and the companion watcher adds a watchlist, snapshots and reports, but running it on a schedule, routing alerts and deciding how long to keep evidence remain the operator's job. AI can classify and summarise verified changes, but should not replace the underlying hashes, fields and source records.

License

MIT License

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      Not graded
      maintenance
      An MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.
      48 npm
      -
    • A
      license
      A
      quality
      C
      maintenance
      MCP server for mdbrowse that enables agents to fetch, search, and browse web pages as clean, token-efficient markdown with provenance, caching, and structured navigation.
      16
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch web pages and crawl sites locally, converting content into clean Markdown with preserved links, caching, robots.txt compliance, and minimal token usage.
      MIT