Crawlmind

AI-native SEO audits from your terminal or any MCP client. Crawlmind crawls a website, finds technical SEO problems, and goes further than other crawlers: it suggests the internal links you should add, measures page speed per template instead of per page, and shows what changed between two crawls.

What makes it different

1. Internal link suggestions. Local multilingual embeddings find pages that are about the same thing but not linked, and point to the paragraph where a link fits and the words that could become its anchor. They also detect keyword cannibalization.

2. Speed by template. Every crawled page gets TTFB and download timings. Pages are grouped into templates, and Lighthouse (via the PageSpeed Insights API) runs only on one or two representatives of each, so one fix covers thousands of pages.

3. Crawl diffs. Compare two crawls of the same site: new 404s, pages that lost inlinks, TTFB regressions, changed titles and canonicals.

Related MCP server: SEO Audit MCP

Install

Requires Node.js 22 or newer.

As an MCP server

Claude Code:

claude mcp add crawlmind -- npx -y crawlmind-mcp

Claude Desktop, Cursor and other clients (JSON config):

{
  "mcpServers": {
    "crawlmind": {
      "command": "npx",
      "args": ["-y", "crawlmind-mcp"]
    }
  }
}

crawlmind-mcp is a shortcut for crawlmind mcp; both start the same server over stdio. For internal link suggestions, add the embeddings add-on (see below).

Then ask your AI client:

• "Audit https://example.com and give me the top 10 fixes."

• "Which internal links should I add to my blog?"

• "Which templates on my shop are slow on mobile?"

• "Crawl the site again and tell me what broke since last week."

• "Export an HTML report of the last crawl to ~/Desktop/audit.html."

As a CLI

npx crawlmind --help
# or
npm install -g crawlmind

crawlmind crawl https://example.com          # crawl with live progress, then a summary
crawlmind crawl https://example.com -n 2000  # up to 2,000 pages
crawlmind summary                            # key numbers and top issues of the latest crawl
crawlmind issues --severity critical         # issues with examples
crawlmind issues --check links.broken_internal   # every affected URL of one check
crawlmind page /pricing                      # everything about one URL
crawlmind links /pricing --content-only      # who links here, from the main content
crawlmind top --by underlinked               # important pages that need more links
crawlmind suggest                            # internal links to add (needs the embeddings add-on)
crawlmind suggest --target /pricing          # links that should point to one page
crawlmind cannibalization                    # pages competing for the same topic
crawlmind templates                          # page templates with TTFB per template
crawlmind speed                              # PageSpeed Insights on template representatives
crawlmind diff                               # what changed between the two latest crawls
crawlmind report -o audit.html               # self-contained HTML report (or -f md)
crawlmind sql "SELECT url, ttfb_ms FROM pages ORDER BY ttfb_ms DESC LIMIT 10"
crawlmind crawls                             # past crawls

Every command takes --json for machine-readable output (progress goes to stderr). In CI, crawlmind crawl <url> --fail-on critical exits with code 1 when critical issues are found.

The CLI and the MCP server share one local database, so you can crawl from the terminal and then ask your AI client about the result.

Packages

MCP tools

Prompts: seo_audit runs the full audit workflow and ends with a prioritized fix list; internal_linking_plan turns link suggestions into a table of links to add.

Resources: crawlmind://checks/{id} documents each check (why it matters, how to fix), and crawlmind://crawls/{id}/summary gives the summary of a crawl.

Checks

Every issue has the same shape: id, severity (critical, warning, info), title, count, up to five examples and a recommendation.

Generic anchors ("click here", "read more", "подробнее", "mehr erfahren", …) are listed per language in generic-anchors.ts.

Link graph

Inlinks and outlinks are counted per page and by placement (nav, header, footer, main, other), with links through redirects credited to the final page. Click depth is a breadth-first search from the start URL. Internal PageRank uses damping 0.85, spreads the rank of pages without outlinks evenly, ignores nofollow, and iterates until the total change is below 1e-6; scores are shown 0–100 relative to the strongest page.

Internal link suggestions

Crawlmind embeds every indexable page (title, H1 and the start of the main content) with a local model and looks for pages that are about the same topic as a target but do not link to it yet. For each pair it returns:

• the paragraph of the source page that fits the target best (paragraphs of the main content are embedded separately),

• anchor candidates: phrases from that paragraph that share key words with the target's title and H1, matched by word prefixes so inflected forms count (беговые кроссовки for "Лучшие беговые кроссовки"),

• the similarity and the target's content inlinks and PageRank.

Without a target, the most link-worthy pages come first: rank = similarity × (1 + importance / (1 + content inlinks)), where importance is the larger of the page's relative PageRank and its sitemap priority. Crawlmind never writes the final sentence; your AI client does, from this context. No API keys are involved.

Cannibalization: pairs of indexable pages whose embeddings are at least 0.92 similar and whose title/H1 key words overlap by at least 50%.

The embeddings add-on

The model runs locally with transformers.js. Its ONNX runtime is several hundred megabytes, so it is an optional peer dependency and everything else works without it. To enable link suggestions:

npm install -g crawlmind @huggingface/transformers
# or, without installing:
npx -p crawlmind -p @huggingface/transformers crawlmind suggest

For MCP clients:

claude mcp add crawlmind -- npx -y -p crawlmind-mcp -p @huggingface/transformers crawlmind-mcp

The model is downloaded once (about 120 MB) into ~/.crawlmind/models. Page vectors are stored per crawl; paragraph vectors are computed only for pages that appear in suggestions.

Model choice. Xenova/multilingual-e5-small (8-bit quantized, 384 dimensions) covers about 100 languages, which matters for sites that are not in English, and is strong at retrieval for its size: it is the smallest of the e5 family that clearly beats older multilingual MiniLM models on retrieval benchmarks, while still embedding hundreds of pages per minute on a laptop CPU. Its cosine similarities cluster high, so the default thresholds (0.82 for suggestions, 0.92 for cannibalization) are tuned for it.

Speed by template

Every crawled page gets its time to first byte and download time. Pages are then grouped into templates, and Lighthouse runs only on one or two representatives per template, so a site with thousands of product pages needs a handful of PageSpeed Insights calls instead of thousands:

3,400 pages use template /product/* with mobile LCP 4.2 s (real-user, poor), score 38. Fixing this one template improves all of them.

How pages are grouped:

1. URL pattern. Path segments are compared level by level. Numbers, dates, UUIDs, hex ids and long slugs become *; siblings that differ only in digits (post-1, post-2) become *; below the first level, four or more distinct sibling values become *. First-level sections (/blog, /shop) are never merged by count alone.

2. DOM structure. Each page's skeleton (the set of tag paths below <body>, six levels deep, ignoring text and attributes) is reduced to a 32-value MinHash signature. Within one URL pattern, pages whose estimated Jaccard similarity is at least 0.8 form one template, so /p/* pages with two different layouts become two templates.

3. Representatives are the pages whose TTFB is closest to the template median: typical, not the best or worst case.

PageSpeed Insights runs on demand (measure_templates / crawlmind speed), mobile by default, with a 60 s timeout and retries with backoff. Results (performance score, LCP, CLS, TBT, FCP, page weight and CrUX field data when Google has it) are stored per crawl and reused. Without PSI_API_KEY Crawlmind uses Google's shared keyless quota one request at a time, which is often exhausted; a free API key raises it.

Crawl diffs and reports

Crawls are kept, so any two crawls of a site can be compared (compare_crawls, crawlmind diff): new and fixed 4xx/5xx pages, new and fixed broken links, pages added and removed, the pages that gained or lost the most inlinks, TTFB regressions (at least 200 ms and 50% slower), title and canonical changes, and how each issue count moved. A warning is added when the crawls differ much in size, since "removed" pages may just not have been reached.

Reports (export_report, crawlmind report) contain the summary, every issue with up to ten examples and its fix, under-linked important pages, speed by template, and the strongest and slowest pages. The HTML version is a single file with inline CSS, no scripts, light and dark themes, and works on phones. A report never replaces an existing file unless asked to, and never writes through a symbolic link. MCP clients write only inside ~/.crawlmind/reports unless the server runs with CRAWLMIND_ALLOW_REPORT_PATHS=1, so instructions hidden in a crawled page cannot make the AI client overwrite your files.

Configuration

Crawling rules

• Identifies itself as CrawlmindBot/<version> (+https://github.com/Everhard/crawlmind).

• Follows robots.txt, including Crawl-delay (capped at 10 s), unless told not to.

• Defaults: 500 pages (hard cap 10,000), depth 10, 4 parallel requests, 250 ms between requests to the same host, 15 s timeout, 5 MB per page.

• Only text/html is parsed. Sitemaps (from robots.txt and /sitemap.xml, indexes and gzip included) add URLs that links do not reach.

• Refuses private, loopback and link-local addresses, checked on every connection and redirect hop (DNS rebinding included). For intranet or local sites use --allow-private in the CLI. MCP clients may only set allowPrivateNetworks when the server runs with CRAWLMIND_ALLOW_PRIVATE_NETWORKS=1, so a prompt injection on a crawled page cannot turn the AI client into a local network scanner.

Limits

• No JavaScript rendering: pages are analyzed as the server sends them. Sites that build their content in the browser show up nearly empty.

• One site per crawl (optionally with subdomains), up to 10,000 pages.

• Crawlmind only reads. No tool changes the audited site.

Privacy

Everything stays on your machine, in ~/.crawlmind (override with CRAWLMIND_HOME). Crawlmind contacts only the site you crawl, the PageSpeed Insights API when you measure templates, and Hugging Face once to download the embedding model when you first use link suggestions. No telemetry.

Development

npm install
npm run build
npm run lint && npm run typecheck && npm test

Try the MCP server in the official inspector:

npx @modelcontextprotocol/inspector node packages/crawlmind/dist/cli.js mcp

Layout: packages/crawlmind/src/core is the engine (crawler, storage, analysis, speed, embeddings, report, diff) and never imports the MCP SDK or CLI libraries; src/mcp and src/cli are thin layers over it.

See CONTRIBUTING.md.

License

MIT © 2026 Andrew Dorokhov