web-insights-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| WEB_INSIGHTS_MAX_BYTES | No | Max decoded response size; larger bodies are aborted mid-stream. | 5242880 |
| WEB_INSIGHTS_HTTP_TOKEN | No | Require Authorization: Bearer <token> on /mcp. | |
| WEB_INSIGHTS_TIMEOUT_MS | No | Per-request timeout (whole request incl. redirects and body). | 15000 |
| WEB_INSIGHTS_USER_AGENT | No | User-Agent used for outbound requests; identify your bot honestly. | web-insights-mcp/0.1.0 (+…; polite research bot) |
| WEB_INSIGHTS_CONCURRENCY | No | Max outbound requests in flight, shared by all tools and sessions. | 4 |
| WEB_INSIGHTS_CACHE_TTL_MS | No | In-memory LRU cache TTL in milliseconds (0 disables). | 300000 |
| WEB_INSIGHTS_ALLOWED_HOSTS | No | Extra Host header values to accept (reverse proxy / public hostname). | |
| WEB_INSIGHTS_CACHE_ENTRIES | No | In-memory LRU cache entry limit (0 disables). | 200 |
| WEB_INSIGHTS_MAX_REDIRECTS | No | Redirect hops per request. | 5 |
| WEB_INSIGHTS_ALLOW_LOCALHOST | No | Set to 1 to allow loopback only (for testing against a local dev server). | off |
| WEB_INSIGHTS_MAX_CRAWL_PAGES | No | Hard cap for crawl_site (and 100 for site_seo_audit). | 200 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Tools for researching and auditing public web pages.
fetch_page: read one URL as clean Markdown.
crawl_site: list a site's pages (sitemap or links; robots.txt respected).
seo_audit / site_seo_audit: on-page SEO checks with severities and fixes, for one page or a crawl.
extract_structured: JSON-LD, microdata, Open Graph, contacts, headings and tables as JSON.
compare_pages: content-gap comparison of two URLs. Only public http(s) URLs are reachable; private and internal network addresses are blocked by design.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| fetch_pageA | Download a web page and return its main readable content as clean Markdown (navigation, ads, scripts and boilerplate removed, links made absolute) plus title, meta description, language, canonical URL and word count. Use this whenever you need to read, summarise, quote or fact-check a specific URL. mode "article" (default) uses Mozilla Readability for articles/blog posts/docs; use mode "full" for home pages, landing pages or listings where the article detector would drop content. Output is truncated to max_chars. |
| crawl_siteA | Discover the pages of a website and return each page's URL, HTTP status and . Reads sitemap.xml (and sitemap indexes) when present, otherwise follows same-origin links breadth-first. Always honours robots.txt (disallowed URLs are listed under |
| seo_auditA | Run an on-page SEO audit of one URL. Checks HTTP status/HTTPS/redirects, title and meta description length, H1 count and heading order, canonical, meta robots/X-Robots-Tag, viewport, lang, charset, Open Graph and Twitter tags, image alt text, JSON-LD presence and validity, internal/external links and anchor text, and thin content. Returns a 0-100 score and a prioritised issue list where every issue has a severity (error/warning/info) and a concrete fix. Use site_seo_audit for a whole site. |
| site_seo_auditA | Crawl a website (sitemap.xml or same-origin links, robots.txt respected) and run seo_audit on every page, then aggregate: average score, the most common issues with how many pages each affects, duplicate titles and meta descriptions, broken pages (4xx/5xx/unreachable) and the worst-scoring pages. Use it for site health reports and to prioritise fixes across many pages. Keep max_pages small (10-30) for a quick overview. |
| extract_structuredA | Extract machine-readable data from a page as JSON: schema.org JSON-LD blocks (parsed, with validity errors), microdata items, Open Graph and Twitter card tags, contact details (mailto emails, tel phone numbers, social profile links, contact pages), the heading outline (h1-h6) and HTML tables (headers + rows). Use it to read product/price/rating/event/organization data, collect a company's public contact channels, or turn a pricing or comparison table into rows. Choose sections with |
| compare_pagesA | Fetch two URLs and compare them side by side: title, meta description, H1, word count, heading outline (sub-topics covered only by A, only by B, or both), top keywords and two-word phrases, structured-data types and link counts. Use it for competitor content-gap analysis (your page vs. a ranking competitor), before/after checks of a redesign or migration, or comparing two versions of a landing page. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 6 tools
Each tool targets a clearly distinct operation: fetch_page returns content for one URL, crawl_site returns a URL inventory without bodies, seo_audit scores one page while site_seo_audit aggregates across a site, and extract_structured/compare_pages handle data extraction and pairwise diffing. The descriptions explicitly disambiguate the closest pairs (seo_audit vs site_seo_audit, crawl_site vs fetch_page). No two tools appear to do the same thing.
Four tools follow a clean verb_noun pattern (fetch_page, crawl_site, extract_structured, compare_pages), but seo_audit and site_seo_audit are noun phrases that break the verb-led convention. The deviation is minor and the names remain self-explanatory, so the set is mostly predictable.
Six tools is well within the sweet spot for a web-insights server, and each maps to a genuinely different workflow stage (discover, read, extract, audit, compare). Nothing feels padded or redundant.
The surface covers the core lifecycle: discover URLs, fetch content, extract structured data, audit single pages and whole sites, and compare two pages. Minor gaps exist (e.g. no dedicated link/redirect graph reporting or performance/accessibility checks), but agents can work around these with the existing tools.