Skip to main content
Glama
Sign In
Crawlmind

Crawlmind

by Everhard
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

Crawlmind

AI-native SEO audits from your terminal or any MCP client. Crawlmind crawls a website, finds technical SEO problems, and goes further than other crawlers: it suggests the internal links you should add and shows what changed between two crawls.

CI npm MCP License: MIT

💬 Just ask your AI

Connect Crawlmind to Claude, Cursor or any MCP client (install) and talk to it:

You ask

Crawlmind does

🩺 "Audit https://example.com and give me the top 10 fixes."

Crawls the site, ranks the issues and explains how to fix each one

🔗 "Which internal links should I add to my blog?"

Finds related pages that don't link to each other, with anchor ideas

🥊 "Do any of my pages compete for the same keywords?"

Lists page pairs that cover the same topic

🐢 "Which pages on my site respond slowest?"

Ranks pages by server response time (TTFB)

🔄 "Crawl again and tell me what broke since last week."

Compares two crawls: new 404s, lost links, slower pages

📄 "Make an HTML report of the last crawl."

Writes a single-file report you can send to a client or a team

Related MCP server: SEO Audit MCP

What it can do

Feature

What you get

CLI

MCP tool

🕷️ Site crawl

Follows links and sitemaps, respects robots.txt, shows live progress

crawl

start_crawl

🩺 Technical SEO audit

Broken links, redirect chains, canonical errors, missing and duplicate titles, thin content, orphan pages, slow pages, each with a fix

summary, issues

get_summary, get_issues

🔎 Page inspector

Everything about one URL: status, redirects, metadata, timings, link metrics

page

get_page

🕸️ Link graph

Inlinks and outlinks by placement, click depth, internal PageRank

links, top

get_links, get_top_pages

📉 Under-linked pages

Important pages that get too few links from content

top --by underlinked

get_top_pages

🔗 Internal link suggestions ★

Which page should link where, the paragraph that fits and anchor candidates

suggest

suggest_links

🥊 Keyword cannibalization ★

Page pairs that compete for the same topic

cannibalization

find_cannibalization

🔄 Crawl diffs ★

What changed since the last crawl: new 404s, lost inlinks, TTFB regressions, changed titles and canonicals

diff

compare_crawls

📄 Reports

A single-file HTML report (light and dark, works on phones) or Markdown

report

export_report

🧮 SQL access

Read-only SELECT queries on the crawl database

sql

query_data

🚦 CI gate

--fail-on critical exits with code 1 when critical issues are found

crawl --fail-on

—

🤖 Ready-made AI workflows

The seo_audit prompt ends with a prioritized fix list; internal_linking_plan builds a table of links to add

—

prompts

★ Other crawlers don't do this. Link suggestions and cannibalization use local multilingual embeddings: no API keys, nothing leaves your machine. The embeddings add-on (about 800 MB) installs itself the first time you use these features, after you confirm.

Crawlmind analyzes pages as the server sends them and does not run JavaScript, so sites that build their content in the browser show up nearly empty.

More detail: all audit checks, MCP tools and how it works.

Install

Requires Node.js 22 or newer.

As an MCP server

Claude Code:

claude mcp add crawlmind -- npx -y crawlmind-mcp

For Claude Desktop, Cursor and other clients, see the JSON config.

Then ask your AI client anything, for example the questions above.

As a CLI

npx crawlmind --help
# or
npm install -g crawlmind

🕷️ Crawl

crawlmind crawl https://example.com          # crawl with live progress, then a summary
crawlmind crawl https://example.com -n 2000  # up to 2,000 pages (default 1,000)
crawlmind crawl https://example.com -n 0     # the whole site, no page limit
crawlmind crawls                             # past crawls

🩺 Audit

crawlmind summary                               # key numbers and top issues of the latest crawl
crawlmind issues --severity critical            # issues with examples
crawlmind issues --check links.broken_internal  # every affected URL of one check
crawlmind page /pricing                         # everything about one URL

🔗 Links

crawlmind links /pricing --content-only  # who links here, from the main content
crawlmind top --by underlinked           # important pages that need more links
crawlmind suggest                        # internal links to add (installs embeddings on first use)
crawlmind suggest --target /pricing      # links that should point to one page
crawlmind cannibalization                # pages competing for the same topic

🔄 Compare and export

crawlmind diff                  # what changed between the two latest crawls
crawlmind report -o audit.html  # self-contained HTML report (or -f md)

🧮 Advanced

# read-only SQL on the crawl database
crawlmind sql "SELECT url, ttfb_ms FROM pages ORDER BY ttfb_ms DESC LIMIT 10"
crawlmind addon status  # is the embeddings add-on installed?

Every command takes --json for machine-readable output (progress goes to stderr). In CI, crawlmind crawl <url> --fail-on critical exits with code 1 when critical issues are found.

The CLI and the MCP server share one local database, so you can crawl from the terminal and then ask your AI client about the result.

Privacy

Everything stays on your machine, in ~/.crawlmind (override with CRAWLMIND_HOME). Crawlmind contacts only the site you crawl. When you install the embeddings add-on, it also contacts the npm registry and Hugging Face once, to download the packages and the model. No telemetry.

Crawlmind only reads; it never changes the audited site. What the crawler sends and which limits it respects: see the crawling rules.

Development

See CONTRIBUTING.md.

License

MIT © 2026 Andrew Dorokhov

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    This server cannot be deployed

    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Resources

    Related MCP Connectors

    • crawlinx-mcpA
      io.github.im-ff

      Free technical-SEO audit MCP: crawl a site, run checks, return an LLM-ready shareable report.

      Web ScrapingBrowser AutomationMarketing
      3
      https://crawlinx.com/mcp
    • AIsa On-Page AuditB
      one.aisa

      Your agent needs to crawl a site and say what is wrong with it — broken tags, duplicate content, pages nothing can index, resources that never load. **What you can ask for** • "Crawl this site and list every page with a duplicate title or missing description." • "Which pages are non-indexable, and why?" • "Run Lighthouse on these URLs and give me the failing audits." • "Show the internal link graph and the orphan pages." • "Give me this page's raw HTML and its microdata." **How to use it** Point any MCP client at https://mcp.aisa.one/seo-onpage/mcp and sign in with OAuth — there is no key to create or paste. 20 tools: submit a crawl and read its summary, pages, resources, links and waterfall; duplicate content and duplicate tags; keyword density; non-indexable and uncrawlable resources; parsed content, raw HTML, microdata, screenshots and Lighthouse. **Why this rather than the source** A crawler you drive from the agent, with the audit results as structured data rather than a PDF. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Find the broken pages here, then ask the same agent what those pages used to rank for — without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo/mcp for all of it at once — rankings, keywords, backlinks, site health and AI-answer visibility across DataForSEO, Semrush and Ahrefs.

      Browser AutomationWeb Scraping
      261
      https://mcp.aisa.one/seo-onpage/mcp?from=glama
    • SEOReport
      dev.seoreport

      Run full technical SEO audits of any website from your AI agent and get structured findings back.

      MarketingBrowser AutomationWeb Scraping
      1
      https://seoreport.dev/mcp
    • Agent Audit AI
      io.github.michaelvereb

      Autonomous AI agent readiness, crawler reachability, and 30-point audit specification engine.

      Autonomous AgentsTesting & QA ToolsBrowser Automation
      1
      https://www.agentauditai.com/mcp

    Related MCP Servers

    • SEO Audit MCP

      ahmaddeveloper17
      A
      license
      Not graded
      quality
      D
      maintenance
      Enables running comprehensive SEO audits on websites, checking meta tags, sitemaps, robots.txt, broken links, headings, duplicate titles, and keyword density through MCP tools.
      7 npm
      MIT
    • Keytomic

      Keytomic

      epicx-labs
      A
      license
      Not graded
      quality
      B
      maintenance
      Remote MCP server for Keytomic, the AI SEO agent: SEO audits, Search Console and GA4 data, keyword research, articles, CMS publishing, indexing, backlinks, Reddit and AI visibility. You sign in with your Keytomic account.
      1
      MIT