web-insights-mcp
# web-insights-mcp
**An MCP server that gives Claude (and any MCP client) safe, structured access to the web for research and SEO work.**
Read any page as clean Markdown, map a site, run on-page SEO audits with concrete fixes, pull out structured data and
compare a page against a competitor — with SSRF protection, timeouts, size limits and polite crawling built in.
   

## Why
LLM agents are good at reasoning about web content but bad at getting it: raw HTML wastes the context window, ad-hoc
`fetch` tools happily request `http://169.254.169.254/` (cloud metadata) or your intranet, and "SEO audits" written
by the model itself are inconsistent from run to run. This server does the deterministic parts in code and hands the
model compact, typed results it can act on:
- **Clean input for the model** — Mozilla Readability (the Firefox Reader View algorithm) plus Turndown turns a page into
Markdown with absolute links and GFM tables — usually a small fraction of the raw HTML size.
- **Repeatable audits** — the same page always gets the same checks, severities and score.
- **Safe by default** — private, loopback, link-local and metadata addresses are blocked after DNS resolution, on every
redirect hop and again at connect time.
## Tools
| Tool | What it does | Typical prompt |
|---|---|---|
| `fetch_page` | URL → main content as Markdown + title, description, language, canonical, byline, word count. `article` or `full` mode, `max_chars` cap, optional link list. | "Summarise this article and quote the key numbers." |
| `crawl_site` | Discover pages via sitemap.xml (incl. sitemap indexes) or same-origin BFS. Page/depth caps, path prefix, robots.txt + Crawl-delay honoured. Returns URL, status, title, redirects. | "List every blog post on this site and flag broken pages." |
| `seo_audit` | 25+ on-page checks: status/HTTPS/redirect chains, title & meta description length, H1 count, heading order, canonical, meta robots + `X-Robots-Tag`, viewport, lang, charset, Open Graph, Twitter card, image alt, JSON-LD presence/validity/`@context`/`@type`, internal links, empty and generic anchors, thin content. Each issue has severity + fix; 0-100 score. | "Audit our pricing page and give me a prioritised fix list." |
| `site_seo_audit` | Crawl + audit every page, then aggregate: most common issues (with page counts), duplicate titles/descriptions, broken pages, worst pages, average score. | "Do an SEO health check of the first 30 pages." |
| `extract_structured` | JSON-LD (parsed, with errors), microdata (nested items), Open Graph/Twitter, contacts (emails, phones, social profiles, contact pages), heading outline, HTML tables → JSON rows. | "Get the product prices and ratings from this page." |
| `compare_pages` | Two URLs side by side: title, description, H1, word count, sub-topics only in A / only in B, top keywords and phrases, structured-data types, link counts, plain-English observations. | "What does the competitor's guide cover that ours doesn't?" |
Every tool has zod-validated input, a JSON **output schema** (`structuredContent`) for programmatic clients, a readable
text rendering for chat clients, and `readOnlyHint`/`openWorldHint` annotations. Errors come back as tool errors
(`isError: true`) with a message the model can act on — never as a crashed session.
## Demo
All output below was recorded by `pnpm demo`, which spawns the **built server over stdio** with the official MCP
client and runs it against a fictional fixture site served locally. Full, unedited transcript:
[`docs/demo/transcript.md`](docs/demo/transcript.md).
| | |
|---|---|
|  |  |
|  |  |
Excerpt (`site_seo_audit`, text content as the model sees it):
```text
## Site SEO audit: http://127.0.0.1:3401/
8 page(s) audited (sitemap) · average score **79/100** · 1 broken.
### Most common issues
- [error] title — 1 page(s)
- [error] canonical — 1 page(s)
- [error] viewport — 1 page(s)
- [error] json-ld-valid — 1 page(s)
- [warning] https — 8 page(s)
- [warning] open-graph — 4 page(s)
- [warning] image-alt — 2 page(s)
...
### Broken pages
- http://127.0.0.1:3401/discontinued-product — 404
```
And the SSRF guard, from the same session:
```text
fetch_page(http://169.254.169.254/latest/meta-data/) failed: Blocked by SSRF protection:
169.254.169.254 is a linkLocal address, not a public internet address.
This URL cannot be fetched by design; do not retry it.
```
## Install
Requires **Node.js 22.19+**.
> The package is prepared for npm (`bin`, `files`, `prepublishOnly`) but is **not published yet**. Until it is, use the
> "from source" variants below; after publishing, the `npx -y web-insights-mcp` variants work as written.
```bash
git clone <this repo> web-insights-mcp && cd web-insights-mcp
pnpm install && pnpm build # produces dist/cli.js
node dist/cli.js --help
```
### Claude Code
```bash
# stdio (recommended)
claude mcp add web-insights -- npx -y web-insights-mcp
# from source
claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
# available in every project
claude mcp add --scope user web-insights -- npx -y web-insights-mcp
# with settings
claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
# over HTTP (start the server first: npx web-insights-mcp --http)
claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp
```
### Claude Desktop
Settings → Developer → Edit Config (`claude_desktop_config.json`), then restart Claude Desktop:
```json
{
"mcpServers": {
"web-insights": {
"command": "npx",
"args": ["-y", "web-insights-mcp"]
}
}
}
```
From source, use `"command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]`.
See [`examples/claude_desktop_config.json`](examples/claude_desktop_config.json) for a version with environment settings.
### Cursor
`~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (project) — same shape as above
([`examples/cursor-mcp.json`](examples/cursor-mcp.json)); for HTTP use `{"url": "http://127.0.0.1:3400/mcp"}`
([`examples/cursor-mcp-http.json`](examples/cursor-mcp-http.json)).
### Streamable HTTP
```bash
npx web-insights-mcp --http # http://127.0.0.1:3400/mcp, GET /healthz for probes
npx web-insights-mcp --http --port 8080
WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0 # public bind requires a token
```
The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally
behind a load balancer.
## Configuration
All optional, via environment variables:
| Variable | Default | Purpose |
|---|---|---|
| `WEB_INSIGHTS_TIMEOUT_MS` | `15000` | Per-request timeout (whole request incl. redirects and body). |
| `WEB_INSIGHTS_MAX_BYTES` | `5242880` | Max decoded response size; larger bodies are aborted mid-stream. |
| `WEB_INSIGHTS_MAX_REDIRECTS` | `5` | Redirect hops per request. |
| `WEB_INSIGHTS_CONCURRENCY` | `4` | Max outbound requests in flight, shared by all tools and sessions. |
| `WEB_INSIGHTS_CACHE_ENTRIES` / `_CACHE_TTL_MS` | `200` / `300000` | In-memory LRU cache (0 disables). |
| `WEB_INSIGHTS_MAX_CRAWL_PAGES` | `200` | Hard cap for `crawl_site` (and 100 for `site_seo_audit`). |
| `WEB_INSIGHTS_USER_AGENT` | `web-insights-mcp/0.1.0 (+…; polite research bot)` | Identify your bot honestly. |
| `WEB_INSIGHTS_ALLOW_LOCALHOST` | off | `1` allows **loopback only** (for testing against a local dev server). |
| `WEB_INSIGHTS_HTTP_TOKEN` | – | Require `Authorization: Bearer <token>` on `/mcp`. |
| `WEB_INSIGHTS_ALLOWED_HOSTS` | – | Extra `Host` header values to accept (reverse proxy / public hostname). |
## Security notes
- **SSRF protection.** Only `http:`/`https:` URLs without embedded credentials are fetched. Hostnames are resolved and
the request is refused if *any* answer is not a globally routable unicast address: RFC 1918 private ranges, loopback,
link-local (incl. `169.254.169.254` cloud metadata), CGNAT, unique-local IPv6, multicast, unspecified (`0.0.0.0`),
IPv4-mapped/NAT64/6to4 addresses that embed one of those, and odd IPv4 spellings like `2130706433` or `0x7f000001`.
- **No TOCTOU / DNS rebinding.** The same check runs inside the socket `lookup` used by the HTTP agent, so the address
actually connected to is the one that was validated. Every redirect hop is re-validated.
- **Resource limits.** Timeouts, streamed size cap (checked on decompressed bytes, so gzip bombs are cut off),
redirect cap, global concurrency limit, crawl page/depth caps, content-type allow-list (HTML/XML/text only).
- **Polite crawling.** robots.txt is always honoured (no override), Crawl-delay is respected (capped at 5 s, crawl drops
to 1 request at a time), tracking parameters are stripped to avoid duplicate fetches, and the User-Agent identifies the bot.
- **HTTP transport.** Binds to `127.0.0.1` by default with `Host`/`Origin` validation against DNS rebinding;
refuses to bind to a non-loopback interface unless a bearer token is set; token comparison is constant-time.
- **Read-only.** Tools only issue `GET` requests and never write anywhere. Page content is untrusted input: like any web
tool, a page can contain text aimed at manipulating the model (prompt injection), so keep a human in the loop for
consequential actions.
## Development
```bash
pnpm install
pnpm typecheck # tsc --noEmit, strict
pnpm test # build + 89 vitest tests (unit, tools, SSRF, stdio e2e, HTTP e2e)
pnpm demo # regenerate docs/demo/transcript.md and docs/screenshots (needs Playwright + Chromium)
```
Tests never touch the internet: `test/fixtures/server.ts` serves a fictional coffee-roaster site (sitemap index,
robots.txt, redirects, a deliberately broken legacy page, gzip/latin-1 pages, slow/huge/hostile routes) on a random
local port. The e2e suites spawn `dist/cli.js` exactly as an MCP client would — over stdio with the SDK's
`StdioClientTransport`, and over HTTP with `StreamableHTTPClientTransport`.
```
src/
cli.ts CLI entry (stdio | --http)
server.ts McpServer factory, tool registration
http.ts Streamable HTTP transport, host/origin/token checks
net/ssrf.ts IP classification, DNS validation, guarded lookup
net/fetcher.ts undici client: redirects, limits, cache, concurrency
net/crawl.ts robots.txt, sitemaps, BFS crawler
parse/ cheerio facts, Readability → Markdown, keyword stats
seo/audit.ts on-page checks, scoring, site aggregation
tools/ one file per MCP tool (schemas + handlers)
```
## Limitations (honest list)
- **No JavaScript rendering.** Pages are fetched as HTML; content that only appears after client-side rendering (some
SPAs) is not seen. Server-rendered sites, blogs, docs and most marketing sites work well.
- **On-page SEO only.** No Core Web Vitals/Lighthouse, backlinks, rankings, search volume or SERP data — those need
paid APIs or a browser. Keyword statistics are frequency-based (stopwords removed), not semantic.
- **Heuristic extraction.** Readability can misjudge unusual layouts; use `mode: "full"` when the article view drops
content. Microdata support covers the common cases, not every edge of the spec.
- **Crawl scope.** Same origin only; `.xml.gz` sitemaps are not decompressed; disallowed pages cannot be audited.
- **Networking.** Requests go direct (system HTTP proxies are not used). Loopback can be enabled for testing, but other
private networks cannot — auditing an intranet site is intentionally out of scope.
- **Verification scope.** The automated tests run against local fixtures only (CI has no live-internet tests), so
real-world sites with unusual markup may expose parser edge cases. Bug reports with a URL are welcome.
- **HTTP auth** is a static bearer token, not OAuth. Put it behind your own gateway for multi-tenant use.
## License
MIT © Cuong Nguyen
---
Built by Cuong Nguyen — AI developer. Available for hire on Fiverr: https://www.fiverr.com/cuongnqxx
TDQS
Scored across 6 tools
Each tool targets a clearly distinct operation: fetch_page returns content for one URL, crawl_site returns a URL inventory without bodies, seo_audit scores one page while site_seo_audit aggregates across a site, and extract_structured/compare_pages handle data extraction and pairwise diffing. The descriptions explicitly disambiguate the closest pairs (seo_audit vs site_seo_audit, crawl_site vs fetch_page). No two tools appear to do the same thing.
Four tools follow a clean verb_noun pattern (fetch_page, crawl_site, extract_structured, compare_pages), but seo_audit and site_seo_audit are noun phrases that break the verb-led convention. The deviation is minor and the names remain self-explanatory, so the set is mostly predictable.
Six tools is well within the sweet spot for a web-insights server, and each maps to a genuinely different workflow stage (discover, read, extract, audit, compare). Nothing feels padded or redundant.
The surface covers the core lifecycle: discover URLs, fetch content, extract structured data, audit single pages and whole sites, and compare two pages. Minor gaps exist (e.g. no dedicated link/redirect graph reporting or performance/accessibility checks), but agents can work around these with the existing tools.