EDS Site Analyser (MCP Server)

MCP server built with Mastra that analyses all URLs of a website from a given URL and groups pages by templates.

Tools

1. discover_site_urls – Discover internal URLs of a site starting from a given URL (same-origin crawl, optional limit).

2. analyse_site_and_group_by_templates – Full workflow: crawl the site, fetch each page, compute a DOM-structure template signature, and group pages by identical template.

Related MCP server: crawl4ai-mcp

Setup

cd eds-site-analyser
npm install
npm run build

URL discovery uses a headless browser (Playwright) so JS-rendered sites (e.g. SPAs) are fully crawled: each page is loaded in Chromium, links are extracted from the DOM after a short wait. Sitemaps are fetched first to seed the crawl. If Playwright fails (e.g. no browser), discovery falls back to sitemap + plain fetch. Install Chromium once if needed: npx playwright install chromium.

Run as MCP server (stdio)

node dist/stdio.js

Or from project root after build:

npm run dev

Use in Cursor / MCP clients

Add to your MCP config. For 403 or inaccessible pages, add the google-search server (see mcp-config.example.json):

{
  "mcpServers": {
    "eds-site-analyser": {
      "command": "node",
      "args": ["/path/to/mastra/mcp-servers/eds-site-analyser/dist/stdio.js"]
    },
    "google-search": {
      "command": "npx",
      "args": ["https://github.com/ACSGenUI/mcp-google-search#release"]
    }
  }
}

When step 1 or 2 report HTTP 403 or failed URLs, the LLM can use the google-search tool to try to get page content for those URLs.

Google-search configured inside this server (optional)

You can configure the google-search MCP server inside eds-site-analyser so that 403s and failed fetches are handled automatically (no separate client config). Set environment variables when starting the server:

Example (in-process google-search for 403 fallback):

EDS_GOOGLE_SEARCH_ENABLED=1 node dist/stdio.js

Or in your MCP client config, pass env when spawning the server, e.g.:

{
  "mcpServers": {
    "eds-site-analyser": {
      "command": "node",
      "args": ["/path/to/eds-site-analyser/dist/stdio.js"],
      "env": {
        "EDS_GOOGLE_SEARCH_ENABLED": "1"
      }
    }
  }
}

When enabled, discover_site_urls and analyse_site_and_group_by_templates will call the configured google-search MCP server for any URL that returns HTTP 403 or fails to fetch; if the search returns HTML-like content, it is used for link discovery and template grouping.

Or with npx for eds-site-analyser only (if published):

{
  "mcpServers": {
    "eds-site-analyser": {
      "command": "npx",
      "args": ["-y", "eds-site-analyser"]
    }
  }
}

Template grouping

Templates are derived from a DOM skeleton of each page (tag names and key ids/classes, no text). Pages with the same skeleton are grouped into one template. You can tune maxDepth (default 4) to make grouping coarser (smaller depth) or finer (larger depth).