web-insights-mcp
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@web-insights-mcp Audit our pricing page and give me a prioritized fix list."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
web-insights-mcp
An MCP server that gives Claude (and any MCP client) safe, structured access to the web for research and SEO work. Read any page as clean Markdown, map a site, run on-page SEO audits with concrete fixes, pull out structured data and compare a page against a competitor — with SSRF protection, timeouts, size limits and polite crawling built in.

Why
LLM agents are good at reasoning about web content but bad at getting it: raw HTML wastes the context window, ad-hoc
fetch tools happily request http://169.254.169.254/ (cloud metadata) or your intranet, and "SEO audits" written
by the model itself are inconsistent from run to run. This server does the deterministic parts in code and hands the
model compact, typed results it can act on:
Clean input for the model — Mozilla Readability (the Firefox Reader View algorithm) plus Turndown turns a page into Markdown with absolute links and GFM tables — usually a small fraction of the raw HTML size.
Repeatable audits — the same page always gets the same checks, severities and score.
Safe by default — private, loopback, link-local and metadata addresses are blocked after DNS resolution, on every redirect hop and again at connect time.
Related MCP server: read-website-fast
Tools
Tool | What it does | Typical prompt |
| URL → main content as Markdown + title, description, language, canonical, byline, word count. | "Summarise this article and quote the key numbers." |
| Discover pages via sitemap.xml (incl. sitemap indexes) or same-origin BFS. Page/depth caps, path prefix, robots.txt + Crawl-delay honoured. Returns URL, status, title, redirects. | "List every blog post on this site and flag broken pages." |
| 25+ on-page checks: status/HTTPS/redirect chains, title & meta description length, H1 count, heading order, canonical, meta robots + | "Audit our pricing page and give me a prioritised fix list." |
| Crawl + audit every page, then aggregate: most common issues (with page counts), duplicate titles/descriptions, broken pages, worst pages, average score. | "Do an SEO health check of the first 30 pages." |
| JSON-LD (parsed, with errors), microdata (nested items), Open Graph/Twitter, contacts (emails, phones, social profiles, contact pages), heading outline, HTML tables → JSON rows. | "Get the product prices and ratings from this page." |
| Two URLs side by side: title, description, H1, word count, sub-topics only in A / only in B, top keywords and phrases, structured-data types, link counts, plain-English observations. | "What does the competitor's guide cover that ours doesn't?" |
Every tool has zod-validated input, a JSON output schema (structuredContent) for programmatic clients, a readable
text rendering for chat clients, and readOnlyHint/openWorldHint annotations. Errors come back as tool errors
(isError: true) with a message the model can act on — never as a crashed session.
Demo
All output below was recorded by pnpm demo, which spawns the built server over stdio with the official MCP
client and runs it against a fictional fixture site served locally. Full, unedited transcript:
docs/demo/transcript.md.
|
|
|
|
Excerpt (site_seo_audit, text content as the model sees it):
## Site SEO audit: http://127.0.0.1:3401/
8 page(s) audited (sitemap) · average score **79/100** · 1 broken.
### Most common issues
- [error] title — 1 page(s)
- [error] canonical — 1 page(s)
- [error] viewport — 1 page(s)
- [error] json-ld-valid — 1 page(s)
- [warning] https — 8 page(s)
- [warning] open-graph — 4 page(s)
- [warning] image-alt — 2 page(s)
...
### Broken pages
- http://127.0.0.1:3401/discontinued-product — 404And the SSRF guard, from the same session:
fetch_page(http://169.254.169.254/latest/meta-data/) failed: Blocked by SSRF protection:
169.254.169.254 is a linkLocal address, not a public internet address.
This URL cannot be fetched by design; do not retry it.Install
Requires Node.js 22.19+.
The package is prepared for npm (
bin,files,prepublishOnly) but is not published yet. Until it is, use the "from source" variants below; after publishing, thenpx -y web-insights-mcpvariants work as written.
git clone <this repo> web-insights-mcp && cd web-insights-mcp
pnpm install && pnpm build # produces dist/cli.js
node dist/cli.js --helpClaude Code
# stdio (recommended)
claude mcp add web-insights -- npx -y web-insights-mcp
# from source
claude mcp add web-insights -- node /absolute/path/to/web-insights-mcp/dist/cli.js
# available in every project
claude mcp add --scope user web-insights -- npx -y web-insights-mcp
# with settings
claude mcp add web-insights -e WEB_INSIGHTS_TIMEOUT_MS=20000 -- npx -y web-insights-mcp
# over HTTP (start the server first: npx web-insights-mcp --http)
claude mcp add --transport http web-insights http://127.0.0.1:3400/mcpClaude Desktop
Settings → Developer → Edit Config (claude_desktop_config.json), then restart Claude Desktop:
{
"mcpServers": {
"web-insights": {
"command": "npx",
"args": ["-y", "web-insights-mcp"]
}
}
}From source, use "command": "node", "args": ["/absolute/path/to/web-insights-mcp/dist/cli.js"].
See examples/claude_desktop_config.json for a version with environment settings.
Cursor
~/.cursor/mcp.json (global) or .cursor/mcp.json (project) — same shape as above
(examples/cursor-mcp.json); for HTTP use {"url": "http://127.0.0.1:3400/mcp"}
(examples/cursor-mcp-http.json).
Streamable HTTP
npx web-insights-mcp --http # http://127.0.0.1:3400/mcp, GET /healthz for probes
npx web-insights-mcp --http --port 8080
WEB_INSIGHTS_HTTP_TOKEN=change-me npx web-insights-mcp --http --host 0.0.0.0 # public bind requires a tokenThe HTTP transport is stateless (one MCP session per request; cache and limits are shared), so it scales horizontally behind a load balancer.
Configuration
All optional, via environment variables:
Variable | Default | Purpose |
|
| Per-request timeout (whole request incl. redirects and body). |
|
| Max decoded response size; larger bodies are aborted mid-stream. |
|
| Redirect hops per request. |
|
| Max outbound requests in flight, shared by all tools and sessions. |
|
| In-memory LRU cache (0 disables). |
|
| Hard cap for |
|
| Identify your bot honestly. |
| off |
|
| – | Require |
| – | Extra |
Security notes
SSRF protection. Only
http:/https:URLs without embedded credentials are fetched. Hostnames are resolved and the request is refused if any answer is not a globally routable unicast address: RFC 1918 private ranges, loopback, link-local (incl.169.254.169.254cloud metadata), CGNAT, unique-local IPv6, multicast, unspecified (0.0.0.0), IPv4-mapped/NAT64/6to4 addresses that embed one of those, and odd IPv4 spellings like2130706433or0x7f000001.No TOCTOU / DNS rebinding. The same check runs inside the socket
lookupused by the HTTP agent, so the address actually connected to is the one that was validated. Every redirect hop is re-validated.Resource limits. Timeouts, streamed size cap (checked on decompressed bytes, so gzip bombs are cut off), redirect cap, global concurrency limit, crawl page/depth caps, content-type allow-list (HTML/XML/text only).
Polite crawling. robots.txt is always honoured (no override), Crawl-delay is respected (capped at 5 s, crawl drops to 1 request at a time), tracking parameters are stripped to avoid duplicate fetches, and the User-Agent identifies the bot.
HTTP transport. Binds to
127.0.0.1by default withHost/Originvalidation against DNS rebinding; refuses to bind to a non-loopback interface unless a bearer token is set; token comparison is constant-time.Read-only. Tools only issue
GETrequests and never write anywhere. Page content is untrusted input: like any web tool, a page can contain text aimed at manipulating the model (prompt injection), so keep a human in the loop for consequential actions.
Development
pnpm install
pnpm typecheck # tsc --noEmit, strict
pnpm test # build + 89 vitest tests (unit, tools, SSRF, stdio e2e, HTTP e2e)
pnpm demo # regenerate docs/demo/transcript.md and docs/screenshots (needs Playwright + Chromium)Tests never touch the internet: test/fixtures/server.ts serves a fictional coffee-roaster site (sitemap index,
robots.txt, redirects, a deliberately broken legacy page, gzip/latin-1 pages, slow/huge/hostile routes) on a random
local port. The e2e suites spawn dist/cli.js exactly as an MCP client would — over stdio with the SDK's
StdioClientTransport, and over HTTP with StreamableHTTPClientTransport.
src/
cli.ts CLI entry (stdio | --http)
server.ts McpServer factory, tool registration
http.ts Streamable HTTP transport, host/origin/token checks
net/ssrf.ts IP classification, DNS validation, guarded lookup
net/fetcher.ts undici client: redirects, limits, cache, concurrency
net/crawl.ts robots.txt, sitemaps, BFS crawler
parse/ cheerio facts, Readability → Markdown, keyword stats
seo/audit.ts on-page checks, scoring, site aggregation
tools/ one file per MCP tool (schemas + handlers)Limitations (honest list)
No JavaScript rendering. Pages are fetched as HTML; content that only appears after client-side rendering (some SPAs) is not seen. Server-rendered sites, blogs, docs and most marketing sites work well.
On-page SEO only. No Core Web Vitals/Lighthouse, backlinks, rankings, search volume or SERP data — those need paid APIs or a browser. Keyword statistics are frequency-based (stopwords removed), not semantic.
Heuristic extraction. Readability can misjudge unusual layouts; use
mode: "full"when the article view drops content. Microdata support covers the common cases, not every edge of the spec.Crawl scope. Same origin only;
.xml.gzsitemaps are not decompressed; disallowed pages cannot be audited.Networking. Requests go direct (system HTTP proxies are not used). Loopback can be enabled for testing, but other private networks cannot — auditing an intranet site is intentionally out of scope.
Verification scope. The automated tests run against local fixtures only (CI has no live-internet tests), so real-world sites with unusual markup may expose parser edge cases. Bug reports with a URL are welcome.
HTTP auth is a static bearer token, not OAuth. Put it behind your own gateway for multi-tenant use.
License
MIT © Cuong Nguyen
Built by Cuong Nguyen — AI developer. Available for hire on Fiverr: https://www.fiverr.com/cuongnqxx
Available Tools
6 toolscompare_pagesCompare two pagesARead-onlyIdempotent
Fetch two URLs and compare them side by side: title, meta description, H1, word count, heading outline (sub-topics covered only by A, only by B, or both), top keywords and two-word phrases, structured-data types and link counts. Use it for competitor content-gap analysis (your page vs. a ranking competitor), before/after checks of a redesign or migration, or comparing two versions of a landing page.
Inputs: url_a (required), url_b (required), keyword_limit. Returns: a, b, differences, observations
crawl_siteCrawl websiteARead-onlyIdempotent
Discover the pages of a website and return each page's URL, HTTP status and . Reads sitemap.xml (and sitemap indexes) when present, otherwise follows same-origin links breadth-first. Always honours robots.txt (disallowed URLs are listed under skipped) and Crawl-delay. Use it to map a site, find broken pages, list blog posts, or pick URLs to read with fetch_page. Page bodies are not returned — call fetch_page for content.
Inputs: url (required), strategy, max_depth, max_pages, +1 more. Returns: pages, stats, robots, sitemap, +5 more
extract_structuredExtract structured dataARead-onlyIdempotent
Extract machine-readable data from a page as JSON: schema.org JSON-LD blocks (parsed, with validity errors), microdata items, Open Graph and Twitter card tags, contact details (mailto emails, tel phone numbers, social profile links, contact pages), the heading outline (h1-h6) and HTML tables (headers + rows). Use it to read product/price/rating/event/organization data, collect a company's public contact channels, or turn a pricing or comparison table into rows. Choose sections with include to keep the response small.
Inputs: url (required), include, max_table_rows. Returns: url, meta, jsonLd, status, +7 more
fetch_pageFetch page as MarkdownARead-onlyIdempotent
Download a web page and return its main readable content as clean Markdown (navigation, ads, scripts and boilerplate removed, links made absolute) plus title, meta description, language, canonical URL and word count. Use this whenever you need to read, summarise, quote or fact-check a specific URL. mode "article" (default) uses Mozilla Readability for articles/blog posts/docs; use mode "full" for home pages, landing pages or listings where the article detector would drop content. Output is truncated to max_chars.
Inputs: url (required), mode, max_chars, include_links. Returns: url, lang, bytes, links, +15 more
seo_auditSEO audit (single page)ARead-onlyIdempotent
Run an on-page SEO audit of one URL. Checks HTTP status/HTTPS/redirects, title and meta description length, H1 count and heading order, canonical, meta robots/X-Robots-Tag, viewport, lang, charset, Open Graph and Twitter tags, image alt text, JSON-LD presence and validity, internal/external links and anchor text, and thin content. Returns a 0-100 score and a prioritised issue list where every issue has a severity (error/warning/info) and a concrete fix. Use site_seo_audit for a whole site.
Inputs: url (required). Returns: url, facts, score, issues, +4 more
site_seo_auditSEO audit (whole site)ARead-onlyIdempotent
Crawl a website (sitemap.xml or same-origin links, robots.txt respected) and run seo_audit on every page, then aggregate: average score, the most common issues with how many pages each affects, duplicate titles and meta descriptions, broken pages (4xx/5xx/unreachable) and the worst-scoring pages. Use it for site health reports and to prioritise fixes across many pages. Keep max_pages small (10-30) for a quick overview.
Inputs: url (required), strategy, max_depth, max_pages, +1 more. Returns: crawl, pages, worstPages, brokenPages, +5 more
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 6 tools
Each tool targets a clearly distinct operation: fetch_page returns content for one URL, crawl_site returns a URL inventory without bodies, seo_audit scores one page while site_seo_audit aggregates across a site, and extract_structured/compare_pages handle data extraction and pairwise diffing. The descriptions explicitly disambiguate the closest pairs (seo_audit vs site_seo_audit, crawl_site vs fetch_page). No two tools appear to do the same thing.
Four tools follow a clean verb_noun pattern (fetch_page, crawl_site, extract_structured, compare_pages), but seo_audit and site_seo_audit are noun phrases that break the verb-led convention. The deviation is minor and the names remain self-explanatory, so the set is mostly predictable.
Six tools is well within the sweet spot for a web-insights server, and each maps to a genuinely different workflow stage (discover, read, extract, audit, compare). Nothing feels padded or redundant.
The surface covers the core lifecycle: discover URLs, fetch content, extract structured data, audit single pages and whole sites, and compare two pages. Minor gaps exist (e.g. no dedicated link/redirect graph reporting or performance/accessibility checks), but agents can work around these with the existing tools.
Maintenance
Related MCP Connectors
Markdown scrape, site map, page metadata, screenshots, web search and PDF to text. Pay per call
Web pages to clean Markdown, metadata, links, tech stack and scored SEO audits. Pay per call.
Web scraping for agents. Point it at a URL and it returns the page as clean markdown, JavaScript-rendered pages included. Point it at a site and it maps the URLs or crawls the section you need in the background, a few pages at a time so results fit in the conversation. Search the web and read full pages, extract fields with a JSON schema you define (validated, never invented), read a store's catalogue or a blog's posts from the platform's own feed, and check whether a page has changed. Failed requests cost nothing. The free plan includes 1,500 credits a month.
Turns any URL into SEO metadata, contacts, tech stack, and AI-ready Markdown, in one call.
Related MCP Servers
AlicenseNot gradedqualityBmaintenanceEnables MCP clients to scrape and crawl websites into clean markdown, map site URLs, run browser interactions, extract structured fields, and monitor projects for page changes over time. It also supports searching stored crawl data and following long-running jobs.MIT- FlicenseNot gradedqualityAmaintenanceEnables AI agents and MCP clients to fetch webpages and crawl sites, extract main content, and convert it into clean Markdown with preserved links for fast, token-efficient scraping.-
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients and AI agents to fetch individual webpages or crawl sites and receive clean, link-preserving Markdown with caching and robots.txt-aware rate limiting, reducing token usage for downstream pipelines.MIT
- AlicenseNot gradedqualityAmaintenanceEnables MCP clients and AI agents to fetch webpages and crawl sites, strip noise, and convert content into clean, link-preserving Markdown with caching and configurable multi-page depth while minimizing token usage.MIT


