website-content-mcp
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
website-content-mcp
Give an AI agent a reliable, site-scoped view of a website: clean content now,
and deterministic evidence when it changes. It is a free, self-hosted
MCP server (not a general-purpose scraper), so
the agent only reads the website you configure, respects robots.txt, and can
show the hash and HTTP validators behind a result.
What you can do
Monitor a site with evidence. Read selected competitor or market pages on a schedule in a separate workflow, then compare content hashes,
ETags, prices and availability facts before alerting a human or agent.Keep an agent current on your own site. Let a support, sales or content agent list pages, fetch the current Markdown, and answer from what is live rather than from a stale upload.
Build a research foundation. Turn a site's sitemap and page content into a bounded, attributable input for briefs, audits, catalog analysis or change review without granting the agent arbitrary web-fetch access.
Unlike Firecrawl and generic scraping APIs, this project is site-scoped,
robots-compliant, deterministic about change evidence, self-hosted and free.
It returns deterministic evidence only (hashes, validators, line diffs and
structured facts). No language model runs inside the server. Scheduling and
alerting stay outside it, and a companion watcher command is included: see
docs/watcher.md.
Related MCP server: mdb-mcp
Quick start: one minute to useful search
Use stdio for a local desktop agent. Add this to its MCP configuration:
{
"mcpServers": {
"website-content": {
"command": "npx",
"args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
"env": { "SITE_BASE_URL": "https://example.com" }
}
}
}Then ask the agent to call content_refresh once. When it finishes, ask it to
call content_search for a topic. The cache begins empty, so search has no
pages to search until you refresh it or fetch pages individually.
Use HTTP when several approved remote agents need a shared endpoint. The
public HTTP transport is deliberately read-only: populate search with
STARTUP_REFRESH_LIMIT or a restricted operator endpoint rather than exposing
content_refresh anonymously.
A real agent flow
Prompt: “Summarise what is new on astraeus.ie.”
The agent calls
content_list_pagesto discover the site structure.It calls
content_get_pagefor the relevant current pages.It summarises the returned Markdown and cites the source URLs, retaining
contentHash,ETagandfetchedAtfor the next comparison.
For a recurring change workflow, content_diff keeps the previous version of a
page and returns a bounded line diff with old and new hashes, and the
watcher runs a whole watchlist and writes a Markdown report
of only the pages that changed. Ask AI to classify or summarise a verified
difference, not to find it.
More examples
Prompt: “Has the Starter price on the competitor's pricing page changed since last week?”
The agent calls
content_difffor the page. On the first call it stores a baseline and says so. On later calls it reportschanged, the old and newcontentHashandfetchedAt, the added and removed lines, and a one-line summary such as “Amounts no longer shown: €49; now shown: €59.”
Prompt: “What does this page offer and what does it ask visitors to do?”
The agent calls
content_extract_factsand reads the outline, calls to action, pricing signals (amount, plan, billing period) and schema.org data. Anything the page does not state is left out rather than guessed.
Prompt: “Give me a site-health pass for my own site.”
The agent calls
content_site_healthand gets findings grouped as critical, high, medium and low: non-200 sitemap pages, broken internal links, missing or duplicate titles, canonicals that point elsewhere, missing H1s and more. It is bounded and polite, so keep it to sites you own.
Offline regression tests
Run npm ci followed by npm run check for typecheck, lint and tests. The
saved synthetic HTML under tests/fixtures/ covers article formatting, relative
links, code blocks, Unicode spacing, client-rendered shells and malformed
structured data. A fake fetcher also checks response truncation, content hashes
and cache reuse. No live website or API key is needed. The existing live-site
integration test remains opt-in and is skipped by default.
Features
Clean extraction: HTML -> markdown via Mozilla Readability + Turndown (real DOM parsing, never regex). Extraction runs once per page and is cached.
Deterministic change evidence: each page includes a SHA-256 of the complete normalised markdown plus upstream
ETag/Last-Modifiedvalidators when available.Extraction-quality evidence: results state whether content came from Readability, the cleaned body, plain text or metadata-only fallback, with actionable warnings for client-rendered shells.
Commerce-aware metadata: bounded schema.org
Product/OfferJSON-LD is returned as structured product, SKU, GTIN, brand, price, currency and availability facts.Change evidence:
content_diffstores the previous normalized Markdown per URL in a bounded store (retention and entry limits) and returns changed or not, old and new hashes and fetch times, a compact line diff and a deterministic summary. First sight, unchanged pages,304responses and stale-on-error fallbacks are each reported explicitly.Structured facts:
content_extract_factsreturns title, description, canonical, outline, calls to action, pricing signals, schema.org types and dated index items. Unknown values are omitted.Plan-level prices: pages with plan cards are read into a table of plans with price or range, currency, billing period, per-unit basis, free and trial flags and listed features. Renames are reported as renames, and prices carry absolute and percentage change. Currencies are never converted.
Price history and competitor intelligence: the watcher keeps a bounded price series per page, exports it as CSV, writes a living competitor comparison note and a weekly digest, and emits typed events for new posts, releases, roles, integrations, certifications and positioning changes. All deterministic. See
docs/watcher.md.Site health:
content_site_healthchecks titles, descriptions, canonicals, JSON-LD, headings, non-200 sitemap pages, broken internal links and pages missing from the sitemap, ranked by severity.Multiple sites, still allowlisted: tools accept an optional
sitefromSITE_ALLOWLIST. Each site has its own robots.txt rules, cache scope, search scope and request pacing, and redirects to a different site are refused.Discovery: page listing from
sitemap.xml, sitemaps advertised inrobots.txt, or a configured page list.Disk cache: fetched pages cached with a configurable TTL and a size bound; reads prefer cache, then a conditional revalidation, then stale-on-error.
Polite by default: respects
robots.txtdisallow rules, rate-limits to about 1 request/second, honoursRetry-After, and sendsIf-None-Match/If-Modified-Sinceso unchanged pages cost a304.Scoped to the configured sites: fetches are refused for any host outside the base site and the operator's allowlist.
Public-network safe by default: allowed hostnames are resolved before every request and redirect hop; private, loopback, link-local and special-use destinations require an explicit operator opt-in.
Two transports: Streamable HTTP and stdio.
What a page result looks like
content_get_page returns clean Markdown plus compact metadata that another
workflow can retain for comparison. Fields are omitted when the source does not
provide them.
{
"url": "https://shop.example/products/blue-widget",
"title": "Blue Widget",
"canonicalUrl": "https://shop.example/products/blue-widget",
"markdown": "# Blue Widget\n\nA useful blue widget.",
"contentHash": "8f3c...64-character-sha256...a91d",
"contentLength": 38,
"truncated": false,
"fetchedAt": "2026-08-16T16:00:00.000Z",
"fromCache": false,
"extractionMethod": "readability",
"extractionQuality": "full",
"sourceTrust": "untrusted-web-content",
"etag": "W/\"widget-v4\"",
"lastModified": "Sat, 16 Aug 2026 12:00:00 GMT",
"products": [{
"name": "Blue Widget",
"sku": "BW-1",
"brand": "Widget Co",
"offers": [{ "price": "19.99", "priceCurrency": "EUR", "availability": "https://schema.org/InStock" }]
}]
}Tools
Tool | Purpose |
| List discoverable pages (sitemap → robots.txt sitemaps → configured list). Returns URL, title, last-modified when known. |
| Walk the discoverable pages and warm the cache so |
| Fetch a page URL, strip to clean markdown, return content + metadata, content hash and any schema.org Product/Offer facts. Supports |
| Bounded structured facts from a page: title, meta description, canonical, H1 to H3 outline, calls to action, pricing signals, schema.org types and Product/Offer data, and dated blog or news items when the page is an index. Unknown values are omitted. |
| Compare a page with the version stored by the previous call: changed or not, old and new |
| Read the stored time series of plan prices for a page (date, plan, price, currency, billing period, unit, content hash) and the latest price of each plan. Never calls the network; history is written by |
| Bounded health check of a site's sitemap pages and internal links, grouped by severity. Available where refresh is allowed, because it fetches many pages. |
| Keyword search over cached pages. Returns URL, score, snippet and the last completed refresh time. |
| Return the raw sitemap structure (URLs + last-modified/priority/change-frequency when present). |
| Server status: configured site, allowed hosts, cache directory, cache size, last fetch time and last completed refresh time. |
On HTTP, up to 50 discoverable pages are warmed after startup and refreshed every
5 minutes by default. Each pass advances through the page list, so larger sites
are covered over successive passes. content_search reports last_refresh_at;
content_health reports lastRefreshAt. Set STARTUP_REFRESH_LIMIT=0 to disable
automatic HTTP refreshes. Stdio users can run content_refresh (about 1 request
per second) or fetch pages individually with content_get_page.
Requirements
Node.js 22+
Install
MCP clients can run the stdio transport without cloning the repository:
{
"mcpServers": {
"website-content": {
"command": "npx",
"args": ["-y", "-p", "@sarutobi-sasuke/website-content-mcp", "website-content-stdio"],
"env": { "SITE_BASE_URL": "https://example.com" }
}
}
}For source development:
git clone https://github.com/SarutobiSasuke8/website-content-mcp.git
cd website-content-mcp
npm install
npm run buildConfiguration
Configuration is via environment variables (see .env.example):
Variable | Required | Default | Description |
| ✅ | - | The site whose content is exposed. |
|
| Sitemap location. | |
| - | Comma-separated fallback page list (absolute or base-relative). | |
| - | Comma-separated hosts (or URLs) that tools may select with the optional | |
| - | Extra hosts that may be fetched. The base URL's host is always allowed. | |
|
| Disk cache directory. | |
|
| Cache TTL (0 disables caching). | |
|
| Cache size bound, evicting oldest-first (0 = unbounded). | |
|
| Where | |
|
| Previous versions and entries not compared for this long are removed. | |
|
| Maximum stored URLs, removing the least recently compared first. | |
|
| Maximum Markdown characters stored per version. The hash always covers the full text. | |
|
| Per-request timeout. | |
|
| Minimum spacing between fetches (about 1 request per second). | |
|
| Hard cap on a single response body. | |
|
| Retries on 429/503, honouring | |
|
| Permit configured hosts to resolve to private/special-use IPs. Use only for deliberate internal-site deployments. | |
|
| Outbound User-Agent. | |
|
| HTTP bind host. | |
|
| HTTP bind port. | |
|
| Expose | |
|
| Per-IP request window for | |
|
| Maximum | |
| empty | Comma-separated trusted proxy IPs, CIDRs or aliases such as | |
|
| Pages refreshed on startup and each scheduled HTTP pass. Set to | |
|
| Seconds between scheduled HTTP passes. Set to |
Run
Streamable HTTP (default transport):
# Using a .env file (recommended):
cp .env.example .env
npm start
# Or passing environment variables inline:
# POSIX / Bash:
SITE_BASE_URL=https://example.com npm start
# PowerShell:
$env:SITE_BASE_URL="https://example.com"; npm start
# -> website-content-mcp listening on http://127.0.0.1:3215/mcp (site: https://example.com/)The MCP endpoint is POST /mcp; a plain health probe is available at GET /healthz.
Streamable HTTP is deliberately public read-only by default. It omits
content_refresh, refuses content_get_page(refresh: true), and redacts the
local cache path from MCP health output. Expired pages are revalidated on
content_get_page reads. The bounded background refresh keeps search current
without exposing a public cache-warming tool. content_search describes the
actual HTTP refresh mode and returns the last completed refresh time.
Inbound /mcp requests are limited per client IP. The server uses the socket IP
unless HTTP_TRUST_PROXY explicitly lists the reverse proxy, so untrusted
X-Forwarded-For values cannot bypass the quota. Behind local nginx, set
HTTP_TRUST_PROXY=loopback; use the proxy's specific address or CIDR for other
topologies.
For a loopback Node process behind nginx/systemd, see
docs/production-deployment.md.
stdio (for local MCP clients):
# POSIX / Bash:
SITE_BASE_URL=https://example.com npm run start:stdio
# PowerShell:
$env:SITE_BASE_URL="https://example.com"; npm run start:stdioExample MCP client entry from a local source checkout (stdio):
{
"mcpServers": {
"website-content": {
"command": "node",
"args": ["/path/to/website-content-mcp/dist/src/stdio.js"],
"env": { "SITE_BASE_URL": "https://example.com" }
}
}
}Troubleshooting
Startup refresh fails or logs DNS errors (ENOTFOUND)
When starting the HTTP transport with STARTUP_REFRESH_LIMIT enabled (default 50), the server performs an initial cache-warming pass from the site sitemap. If SITE_BASE_URL cannot be resolved (for example when offline or targeting an unresolvable hostname), a warning is logged:
Startup cache refresh failed: getaddrinfo ENOTFOUND ...The server continues running and responds to HTTP and MCP requests. To disable automatic startup warming, set STARTUP_REFRESH_LIMIT=0 in .env.
Missing SITE_BASE_URL error on startup
SITE_BASE_URL is the only mandatory configuration setting. If omitted, startup fails with a validation error:
[ { "code": "invalid_type", "expected": "string", "received": "undefined", "path": [ "SITE_BASE_URL" ], "message": "Required" } ]Create a .env file from .env.example and configure SITE_BASE_URL before launching.
Keyword search returns empty results
content_search queries only pages present in the local disk cache. On a fresh server instance, the cache begins empty:
On stdio: run
content_refreshto warm the cache, or fetch individual target pages usingcontent_get_page.On HTTP: allow startup cache warming to complete (
STARTUP_REFRESH_LIMIT), or fetch pages individually withcontent_get_page.
Private IP address or loopback destination refused
Outbound network fetches to private, loopback (127.0.0.1, localhost), link-local, or cloud metadata IP ranges are blocked by default to prevent Server-Side Request Forgery (SSRF). When intentionally pointing the server at an internal service or local test environment, set FETCH_ALLOW_PRIVATE_NETWORK=true in .env.
Pages refused by robots.txt
If content_get_page or content_refresh reports RobotsDisallowedError, the requested URL is restricted by the remote site's robots.txt. The server strictly enforces origin robots policies and does not bypass crawl restrictions.
Development
npm run dev # HTTP transport with --watch
npm run dev:stdio # stdio transport
npm run check # typecheck + lint + build + testTesting
Contract suite:
npm run eval:contractchecks the tool list, input schemas and golden outputs against a local fixture site.Unit tests cover HTML→markdown conversion, the disk cache (TTL and eviction), robots parsing, sitemap parsing, the fetcher (size cap, conditional headers,
Retry-Afterretries, rate limiting) and the content service (host scoping, robots enforcement, truncation, cache warming, 304 revalidation, stale fallback). All offline, against a local fixture and a stubbed fetch.Round 2 tests cover the unavailable-versus-removed distinction (503, 429, timeouts, 404 once and twice), hostile text in tasks and notes, https-only extra sites, same-host sitemaps, plan extraction and matching, noise rules, confirmation runs, price thresholds, price history and CSV, typed events, optional rendering with a fake renderer, the comparison note and the digest.
Feature tests cover the multi-site allowlist (refusals, per-site robots, cache and pacing isolation, redirect refusal),
content_diff(first sight, unchanged,304, stale-on-error, bounds), fact extraction against saved HTML fixtures, site-health checks, the watcher (baseline, no-change idempotence, forced change, dry run, exit codes) and the vault feed against a temporary directory.A live integration test runs against
https://example.com, exercisinghealth,list_pages,get_page, andsearch.
npm testLive checks are opt-in. To validate real deployment targets, set
RUN_LIVE_TESTS=1 and provide a comma-separated LIVE_SITE_URLS list of
sites you own or operate before running npm test, for example:
RUN_LIVE_TESTS=1 LIVE_SITE_URLS=https://astraeus.ie npm testWithout LIVE_SITE_URLS, the live check falls back to https://example.com.
Dogfood monitoring
The source repository includes a bounded snapshot-and-diff proof harness (the older, hash-only precursor of the watcher). It stores hashes, validators, extraction quality and warnings without retaining page bodies:
npm run dogfood
npm run dogfoodThe first run establishes baselines and the second reports unchanged or
changed pages. See docs/dogfood-monitoring.md.
Scheduling, production retention and alert routing remain outside the core.
The published CLI can run the same bounded proof without a source checkout:
npx -y -p @sarutobi-sasuke/website-content-mcp website-content-dogfoodSecurity & etiquette
Binds to
127.0.0.1by default.No authentication and no API keys; intended for public content only.
Scoped to the configured site: a URL on any other host is refused, so the server cannot be used as a general-purpose fetcher. Widen deliberately with
SITE_ALLOWED_HOSTS.Redirects are followed manually and every destination is checked against the same host allowlist before a network request is made.
DNS results are checked before each request and redirect hop; private, loopback, link-local and special-use destinations are refused unless
FETCH_ALLOW_PRIVATE_NETWORK=trueis explicitly set by the operator.Returned page text is labelled
untrusted-web-content; clients must treat it as source data, never as instructions that override their own policy.Streamable HTTP defaults to a read-only tool surface. Keep
HTTP_ALLOW_REFRESH=falsefor anonymous deployments.Respects
robots.txt, fetched and enforced per origin; disallowed paths are refused.Rate-limited to about 1 request per second against the target site.
Response bodies are capped at
FETCH_MAX_BYTESand the cache atCACHE_MAX_ENTRIES.Never logs full page bodies (only URLs, status codes, and sizes).
What this server is not
The server is the content-access and evidence layer. It is not a scheduler or
an alerting service, and nothing in it calls a language model. content_diff
gives per-page change evidence and the companion watcher
adds a watchlist, snapshots and reports, but running it on a schedule, routing
alerts and deciding how long to keep evidence remain the operator's job. AI can
classify and summarise verified changes, but should not replace the underlying
hashes, fields and source records.
License
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Turn any public website into an MCP server for agents to search, read and navigate.
MCP server (stdio): fetch web pages as clean readable markdown via the AgentForge API
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Related MCP Servers
- FlicenseNot gradedqualityNot gradedmaintenanceAn MCP server for web content extraction that converts HTML pages into clean, LLM-optimized Markdown using Mozilla's Readability. It supports batch processing, intelligent multi-page crawling, and configurable caching while respecting robots.txt standards.48 npm-
- AlicenseAqualityCmaintenanceMCP server for mdbrowse that enables agents to fetch, search, and browse web pages as clean, token-efficient markdown with provenance, caching, and structured navigation.16MIT
- AlicenseAqualityCmaintenanceMCP server that turns web pages into clean Markdown, keeping tables, code blocks, and images intact. Agents can fetch pages, save docs to your project or a global library, and crawl whole doc sites.53Apache 2.0
- AlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch web pages and crawl sites locally, converting content into clean Markdown with preserved links, caching, robots.txt compliance, and minimal token usage.MIT