Skip to main content
Glama
web-insights-mcp
by nqcthedev

web-insights-mcp

An MCP server that gives Claude (and any MCP client) safe, structured access to the web for research and SEO work. Read any page as clean Markdown, map a site, run on-page SEO audits with concrete fixes, pull out structured data and compare a page against a competitor — with SSRF protection, timeouts, size limits and polite crawling built in.

CI TypeScript MCP License

seo_audit result

Why

LLM agents are good at reasoning about web content but bad at getting it: raw HTML wastes the context window, ad-hoc fetch tools happily request http://169.254.169.254/ (cloud metadata) or your intranet, and "SEO audits" written by the model itself are inconsistent from run to run. This server does the deterministic parts in code and hands the model compact, typed results it can act on:

  • Clean input for the model — Mozilla Readability (the Firefox Reader View algorithm) plus Turndown turns a page into Markdown with absolute links and GFM tables — usually a small fraction of the raw HTML size.

  • Repeatable audits — the same page always gets the same checks, severities and score.

  • Safe by default — private, loopback, link-local and metadata addresses are blocked after DNS resolution, on every redirect hop and again at connect time.

Related MCP server: read-website-fast

Tools

Tool

What it does

Typical prompt

fetch_page

URL → main content as Markdown + title, description, language, canonical, byline, word count. article or full mode, max_chars cap, optional link list.

"Summarise this article and quote the key numbers."

crawl_site

Discover pages via sitemap.xml (incl. sitemap indexes) or same-origin BFS. Page/depth caps, path prefix, robots.txt + Crawl-delay honoured. Returns URL, status, title, redirects.

"List every blog post on this site and flag broken pages."

seo_audit

25+ on-page checks: status/HTTPS/redirect chains, title & meta description length, H1 count, heading order, canonical, meta robots + X-Robots-Tag, viewport, lang, charset, Open Graph, Twitter card, image alt, JSON-LD presence/validity/@context/@type, internal links, empty and generic anchors, thin content. Each issue has severity + fix; 0-100 score.

"Audit our pricing page and give me a prioritised fix list."

site_seo_audit

Crawl + audit every page, then aggregate: most common issues (with page counts), duplicate titles/descriptions, broken pages, worst pages, average score.

"Do an SEO health check of the first 30 pages."

extract_structured

JSON-LD (parsed, with errors), microdata (nested items), Open Graph/Twitter, contacts (emails, phones, social profiles, contact pages), heading outline, HTML tables → JSON rows.

"Get the product prices and ratings from this page."

compare_pages

Two URLs side by side: title, description, H1, word count, sub-topics only in A / only in B, top keywords and phrases, structured-data types, link counts, plain-English observations.

"What does the competitor's guide cover that ours doesn't?"

Every tool has zod-validated input, a JSON output schema (structuredContent) for programmatic clients, a readable text rendering for chat clients, and readOnlyHint/openWorldHint annotations. Errors come back as tool errors (isError: true) with a message the model can act on — never as a crashed session.

Demo

All output below was recorded by pnpm demo, which spawns the built server over stdio with the official MCP client and runs it against a fictional fixture site served locally. Full, unedited transcript: docs/demo/transcript.md.

tools/list

fetch_page

compare_pages

seo_audit

Excerpt (site_seo_audit, text content as the model sees it):

## Site SEO audit: http://127.0.0.1:3401/
8 page(s) audited (sitemap) · average score **79/100** · 1 broken.

### Most common issues
- [error] title — 1 page(s)
- [error] canonical — 1 page(s)
- [error] viewport — 1 page(s)
- [error] json-ld-valid — 1 page(s)
- [warning] https — 8 page(s)
- [warning] open-graph — 4 page(s)
- [warning] image-alt — 2 page(s)
...
### Broken pages
- http://127.0.0.1:3401/discontinued-product — 404

And the SSRF guard, from the same session:

fetch_page(http://169.254.169.254/latest/meta-data/) failed: Blocked by SSRF protection:
169.254.169.254 is a linkLocal address, not a public internet address.
This URL cannot be fetched by design; do not retry it.

Install

Requires Node.js 22.19+.

The package is prepared for npm (bin, files, prepublishOnly) but is not published yet. Until it is, use the "from source" variants below; after publishing, the npx -y web-insights-mcp variants work as written.

git clone <this repo> web-insights-mcp && cd web-insights-mcp
pnpm install && pnpm build          # produces dist/cli.js
node dist/cli.js --help

Claude Code

# stdio (recommended)
claude mcp add web-insights -- npx -y web-insights-mcp
# from source
claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
# available in every project
claude mcp add --scope user web-insights -- npx -y web-insights-mcp
# with settings
claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
# over HTTP (start the server first: npx web-insights-mcp --http)
claude mcp add --transport http web-insights http://127.0.0.1:3400/mcp

Claude Desktop

Settings → Developer → Edit Config (claude_desktop_config.json), then restart Claude Desktop:

{
  "mcpServers": {
    "web-insights": {
      "command": "npx",
      "args": ["-y", "web-insights-mcp"]
    }
  }
}

From source, use "command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"]. See examples/claude_desktop_config.json for a version with environment settings.

Cursor

~/.cursor/mcp.json (global) or .cursor/mcp.json (project) — same shape as above (examples/cursor-mcp.json); for HTTP use {"url": "http://127.0.0.1:3400/mcp"} (examples/cursor-mcp-http.json).

Streamable HTTP

npx web-insights-mcp --http                    # http://127.0.0.1:3400/mcp, GET /healthz for probes
npx web-insights-mcp --http --port 8080
WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0   # public bind requires a token

The HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally behind a load balancer.

Configuration

All optional, via environment variables:

Variable

Default

Purpose

WEB_INSIGHTS_TIMEOUT_MS

15000

Per-request timeout (whole request incl. redirects and body).

WEB_INSIGHTS_MAX_BYTES

5242880

Max decoded response size; larger bodies are aborted mid-stream.

WEB_INSIGHTS_MAX_REDIRECTS

5

Redirect hops per request.

WEB_INSIGHTS_CONCURRENCY

4

Max outbound requests in flight, shared by all tools and sessions.

WEB_INSIGHTS_CACHE_ENTRIES / _CACHE_TTL_MS

200 / 300000

In-memory LRU cache (0 disables).

WEB_INSIGHTS_MAX_CRAWL_PAGES

200

Hard cap for crawl_site (and 100 for site_seo_audit).

WEB_INSIGHTS_USER_AGENT

web-insights-mcp/0.1.0 (+…; polite research bot)

Identify your bot honestly.

WEB_INSIGHTS_ALLOW_LOCALHOST

off

1 allows loopback only (for testing against a local dev server).

WEB_INSIGHTS_HTTP_TOKEN

–

Require Authorization: Bearer <token> on /mcp.

WEB_INSIGHTS_ALLOWED_HOSTS

–

Extra Host header values to accept (reverse proxy / public hostname).

Security notes

  • SSRF protection. Only http:/https: URLs without embedded credentials are fetched. Hostnames are resolved and the request is refused if any answer is not a globally routable unicast address: RFC 1918 private ranges, loopback, link-local (incl. 169.254.169.254 cloud metadata), CGNAT, unique-local IPv6, multicast, unspecified (0.0.0.0), IPv4-mapped/NAT64/6to4 addresses that embed one of those, and odd IPv4 spellings like 2130706433 or 0x7f000001.

  • No TOCTOU / DNS rebinding. The same check runs inside the socket lookup used by the HTTP agent, so the address actually connected to is the one that was validated. Every redirect hop is re-validated.

  • Resource limits. Timeouts, streamed size cap (checked on decompressed bytes, so gzip bombs are cut off), redirect cap, global concurrency limit, crawl page/depth caps, content-type allow-list (HTML/XML/text only).

  • Polite crawling. robots.txt is always honoured (no override), Crawl-delay is respected (capped at 5 s, crawl drops to 1 request at a time), tracking parameters are stripped to avoid duplicate fetches, and the User-Agent identifies the bot.

  • HTTP transport. Binds to 127.0.0.1 by default with Host/Origin validation against DNS rebinding; refuses to bind to a non-loopback interface unless a bearer token is set; token comparison is constant-time.

  • Read-only. Tools only issue GET requests and never write anywhere. Page content is untrusted input: like any web tool, a page can contain text aimed at manipulating the model (prompt injection), so keep a human in the loop for consequential actions.

Development

pnpm install
pnpm typecheck      # tsc --noEmit, strict
pnpm test           # build + 89 vitest tests (unit, tools, SSRF, stdio e2e, HTTP e2e)
pnpm demo           # regenerate docs/demo/transcript.md and docs/screenshots (needs Playwright + Chromium)

Tests never touch the internet: test/fixtures/server.ts serves a fictional coffee-roaster site (sitemap index, robots.txt, redirects, a deliberately broken legacy page, gzip/latin-1 pages, slow/huge/hostile routes) on a random local port. The e2e suites spawn dist/cli.js exactly as an MCP client would — over stdio with the SDK's StdioClientTransport, and over HTTP with StreamableHTTPClientTransport.

src/
  cli.ts            CLI entry (stdio | --http)
  server.ts         McpServer factory, tool registration
  http.ts           Streamable HTTP transport, host/origin/token checks
  net/ssrf.ts       IP classification, DNS validation, guarded lookup
  net/fetcher.ts    undici client: redirects, limits, cache, concurrency
  net/crawl.ts      robots.txt, sitemaps, BFS crawler
  parse/            cheerio facts, Readability → Markdown, keyword stats
  seo/audit.ts      on-page checks, scoring, site aggregation
  tools/            one file per MCP tool (schemas + handlers)

Limitations (honest list)

  • No JavaScript rendering. Pages are fetched as HTML; content that only appears after client-side rendering (some SPAs) is not seen. Server-rendered sites, blogs, docs and most marketing sites work well.

  • On-page SEO only. No Core Web Vitals/Lighthouse, backlinks, rankings, search volume or SERP data — those need paid APIs or a browser. Keyword statistics are frequency-based (stopwords removed), not semantic.

  • Heuristic extraction. Readability can misjudge unusual layouts; use mode: "full" when the article view drops content. Microdata support covers the common cases, not every edge of the spec.

  • Crawl scope. Same origin only; .xml.gz sitemaps are not decompressed; disallowed pages cannot be audited.

  • Networking. Requests go direct (system HTTP proxies are not used). Loopback can be enabled for testing, but other private networks cannot — auditing an intranet site is intentionally out of scope.

  • Verification scope. The automated tests run against local fixtures only (CI has no live-internet tests), so real-world sites with unusual markup may expose parser edge cases. Bug reports with a URL are welcome.

  • HTTP auth is a static bearer token, not OAuth. Put it behind your own gateway for multi-tenant use.

License

MIT © Cuong Nguyen


Built by Cuong Nguyen — AI developer. Available for hire on Fiverr: https://www.fiverr.com/cuongnqxx

Available Tools

6 tools
compare_pagesCompare two pagesA
Read-onlyIdempotent

Fetch two URLs and compare them side by side: title, meta description, H1, word count, heading outline (sub-topics covered only by A, only by B, or both), top keywords and two-word phrases, structured-data types and link counts. Use it for competitor content-gap analysis (your page vs. a ranking competitor), before/after checks of a redesign or migration, or comparing two versions of a landing page.

Inputs: url_a (required), url_b (required), keyword_limit. Returns: a, b, differences, observations

crawl_siteCrawl websiteA
Read-onlyIdempotent

Discover the pages of a website and return each page's URL, HTTP status and . Reads sitemap.xml (and sitemap indexes) when present, otherwise follows same-origin links breadth-first. Always honours robots.txt (disallowed URLs are listed under skipped) and Crawl-delay. Use it to map a site, find broken pages, list blog posts, or pick URLs to read with fetch_page. Page bodies are not returned — call fetch_page for content.

Inputs: url (required), strategy, max_depth, max_pages, +1 more. Returns: pages, stats, robots, sitemap, +5 more

extract_structuredExtract structured dataA
Read-onlyIdempotent

Extract machine-readable data from a page as JSON: schema.org JSON-LD blocks (parsed, with validity errors), microdata items, Open Graph and Twitter card tags, contact details (mailto emails, tel phone numbers, social profile links, contact pages), the heading outline (h1-h6) and HTML tables (headers + rows). Use it to read product/price/rating/event/organization data, collect a company's public contact channels, or turn a pricing or comparison table into rows. Choose sections with include to keep the response small.

Inputs: url (required), include, max_table_rows. Returns: url, meta, jsonLd, status, +7 more

fetch_pageFetch page as MarkdownA
Read-onlyIdempotent

Download a web page and return its main readable content as clean Markdown (navigation, ads, scripts and boilerplate removed, links made absolute) plus title, meta description, language, canonical URL and word count. Use this whenever you need to read, summarise, quote or fact-check a specific URL. mode "article" (default) uses Mozilla Readability for articles/blog posts/docs; use mode "full" for home pages, landing pages or listings where the article detector would drop content. Output is truncated to max_chars.

Inputs: url (required), mode, max_chars, include_links. Returns: url, lang, bytes, links, +15 more

seo_auditSEO audit (single page)A
Read-onlyIdempotent

Run an on-page SEO audit of one URL. Checks HTTP status/HTTPS/redirects, title and meta description length, H1 count and heading order, canonical, meta robots/X-Robots-Tag, viewport, lang, charset, Open Graph and Twitter tags, image alt text, JSON-LD presence and validity, internal/external links and anchor text, and thin content. Returns a 0-100 score and a prioritised issue list where every issue has a severity (error/warning/info) and a concrete fix. Use site_seo_audit for a whole site.

Inputs: url (required). Returns: url, facts, score, issues, +4 more

site_seo_auditSEO audit (whole site)A
Read-onlyIdempotent

Crawl a website (sitemap.xml or same-origin links, robots.txt respected) and run seo_audit on every page, then aggregate: average score, the most common issues with how many pages each affects, duplicate titles and meta descriptions, broken pages (4xx/5xx/unreachable) and the worst-scoring pages. Use it for site health reports and to prioritise fixes across many pages. Keep max_pages small (10-30) for a quick overview.

Inputs: url (required), strategy, max_depth, max_pages, +1 more. Returns: crawl, pages, worstPages, brokenPages, +5 more

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool targets a clearly distinct operation: fetch_page returns content for one URL, crawl_site returns a URL inventory without bodies, seo_audit scores one page while site_seo_audit aggregates across a site, and extract_structured/compare_pages handle data extraction and pairwise diffing. The descriptions explicitly disambiguate the closest pairs (seo_audit vs site_seo_audit, crawl_site vs fetch_page). No two tools appear to do the same thing.

    Naming Consistency4/5

    Four tools follow a clean verb_noun pattern (fetch_page, crawl_site, extract_structured, compare_pages), but seo_audit and site_seo_audit are noun phrases that break the verb-led convention. The deviation is minor and the names remain self-explanatory, so the set is mostly predictable.

    Tool Count5/5

    Six tools is well within the sweet spot for a web-insights server, and each maps to a genuinely different workflow stage (discover, read, extract, audit, compare). Nothing feels padded or redundant.

    Completeness4/5

    The surface covers the core lifecycle: discover URLs, fetch content, extract structured data, audit single pages and whole sites, and compare two pages. Minor gaps exist (e.g. no dedicated link/redirect graph reporting or performance/accessibility checks), but agents can work around these with the existing tools.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables MCP clients to scrape and crawl websites into clean markdown, map site URLs, run browser interactions, extract structured fields, and monitor projects for page changes over time. It also supports searching stored crawl data and following long-running jobs.
      MIT
    • F
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.
      -
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables MCP clients and AI agents to fetch individual webpages or crawl sites and receive clean, link-preserving Markdown with caching and robots.txt-aware rate limiting, reducing token usage for downstream pipelines.
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables MCP clients and AI agents to fetch webpages and crawl sites, strip noise, and convert content into clean, link-preserving Markdown with caching and configurable multi-page depth while minimizing token usage.
      MIT