CrawlBit MCP
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@CrawlBit MCP Can AI crawlers read my site example.com?"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
CrawlBit MCP
Four free tools that answer one question: can AI engines find, read and recognise a website?
No account. No API key. No signup. Nothing here runs a language model, so nothing here costs you or us anything.
"Can ChatGPT read stripe.com?"
"Why does AI never mention my store?"
"Where is my brand missing outside my own site?"Ask in plain language. Your AI client picks the right tool.
Install
Requires Node.js 18+.
Claude Code
claude mcp add crawlbit -- npx -y crawlbit-mcpClaude Desktop
Add to claude_desktop_config.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}Restart your client. Four tools appear.
Related MCP server: seo-audit-mcp
The four tools
Tool | What it answers |
| Can GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended read this site? |
| Does this brand read as a clear entity AI can recognise? |
| Where is this brand missing outside its own website? |
| Can AI shopping agents understand and recommend these products? |
Try these
Check whether AI crawlers can read example.com
Does example.com read as a clear brand entity to AI?
Where is example.com missing off-page, and which gap should we close first?
Compare example.com and competitor.com on AI crawler accessThe tools compose. A useful sequence is: crawler access first (a blocked crawler makes everything else pointless), then the entity check, then the off-page gaps.
What this does NOT do
Written plainly, because an SEO tool that overstates itself is worse than no tool.
It does not test whether AI actually cites a brand. That means running real buyer questions against ChatGPT, Perplexity and Claude, reading the answers, and reporting who gets named instead of you. It costs real money per run and is not part of this free surface. These four tools measure the conditions that make a citation possible, which is a smaller claim.
crawler_watchreadsrobots.txtonly. It cannot see blocking done at the CDN, firewall or rate-limit layer. A site can pass here and still refuse the crawler in practice.entity_checkreads what the site publishes about itself. It says nothing about the brand's reputation across the rest of the web.shoppinginspects published structured data. It cannot see a merchant feed submitted privately to a platform.
If a site scores well on all four and still is not cited, the answer is almost always the same and it is not technical: too few third-party pages mention it. These tools will tell you that honestly rather than sell you a fix that does not exist.
Why only four tools
CrawlBit also runs a full technical audit and generates llms.txt. Both use a language model,
so both cost money per run, and neither is exposed here.
That is a deliberate line rather than a teaser. A free tool whose bill grows with its popularity
gets rate-limited, degraded or withdrawn the moment it succeeds, and the people who installed it
are the ones who pay for that. Everything in this server costs nothing to run, so nothing here
has to be taken back later. The audit and the llms.txt generator live at
crawlbit.app if you want them.
Rate limit
The API rate-limits per IP. Normal conversational use does not come close, and the server runs on your machine, so the budget is yours rather than shared. If you do hit it, you get a plain sentence rather than a stack trace.
Configuration
Variable | Default | Purpose |
|
| Point the server at another instance |
Privacy
The server runs locally and stores nothing. The only data leaving your machine is the URL you ask about, sent to the CrawlBit API to be analysed. There is no account, so there is nothing to tie a request to a person.
About
Built by CrawlBit, which measures AI visibility and does the off-page work that earns citations. The free tools here are the measurement half. The paid work is the other half, and you are not required to look at it to use this.
MIT licensed. Issues and pull requests welcome.
Available Tools
4 toolscrawlbit_crawler_watchAI crawler accessA
Checks whether AI crawlers are allowed to read a site: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. Reads the site's robots.txt and reports, per crawler, whether it is fully allowed, partially blocked or blocked entirely. Use this when someone asks why AI never mentions their site, or before any other AI-visibility work: a blocked crawler makes everything else pointless. Reflects robots.txt only, so it cannot detect blocking done at the CDN or firewall layer.
Inputs: url (required).
crawlbit_entity_checkBrand entity checkA
Checks whether a brand reads as a clear, consistent entity that AI can recognise: Organization schema, a consistent name, a description, a logo and linked profiles (sameAs). AI engines struggle to name a brand they cannot identify as a distinct thing. Use this when a brand is confused with another, or described vaguely, in AI answers. Measures the signals published on the site itself, not the brand's reputation across the web.
Inputs: url (required).
crawlbit_offpage_gapsOff-page gap finderA
Finds where a brand is missing OUTSIDE its own website: review platforms, directories, communities and 'best of' roundups relevant to its category. Off-page mentions on third-party pages are the strongest known predictor of AI citations, because engines weight what others say about a brand far above what the brand says about itself. Use this when a site is technically perfect yet still never cited. Returns the gaps and priorities; it does not publish anything or contact anyone.
Inputs: url (required).
crawlbit_shoppingAI shopping readinessA
Checks whether AI shopping agents such as ChatGPT Shopping can find, understand and recommend a store's products: product schema completeness, price and availability signals, and the attributes those agents read. Use this for e-commerce sites, especially Shopify, when products never surface in AI recommendations. Inspects published structured data, so it cannot see a merchant feed submitted privately to a platform.
Inputs: url (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool targets a distinct aspect of AI visibility: off-page mentions, product schema for shopping agents, crawler access, and entity recognition. There is no overlap, so an agent can unambiguously select the right tool for a given diagnostic need.
All tools share the 'crawlbit_' prefix followed by a descriptive snake_case suffix (offpage_gaps, shopping, crawler_watch, entity_check). The naming convention is consistent and readable, with only a minor variation in compound-word structure that does not cause confusion.
Four tools is an ideal size for a focused diagnostic server. Each tool addresses a separate facet of AI visibility, keeping the scope tight without unnecessary bloat or overwhelming the agent.
The tool set covers the major pillars of AI visibility: being crawlable (crawler_watch), being recognizable (entity_check), having external validation (offpage_gaps), and being suitable for AI shopping (shopping). This is a complete diagnostic suite for the domain, with no obvious missing operations.
Maintenance
Related MCP Connectors
Free, no-auth tools for AI search visibility: robots.txt, schema, rendering, sitemap, PageSpeed.
Free AI visibility checks for any website: AI crawler access, llms.txt, schema, business info.
Free AI SEO audit, product page check, agent readiness check and site comparison. No key.
Free MCP tools for AI-search visibility: crawler checks, page audits, and llms.txt generation.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEvaluates any website's AI visibility with 15 checks across crawlability, structure, content, and connectivity, and provides actionable fixes.MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to perform instant SEO audits, check robots.txt, sitemaps, and AI crawler access for any URL without API keys.MIT
- AlicenseNot gradedqualityDmaintenanceEnables auditing AI search visibility: checks site readiness for AI crawlers and measures whether ChatGPT, Gemini, and Perplexity recommend your site, including verbatim answers and citation gap analysis.4AGPL 3.0
- AlicenseNot gradedqualityDmaintenanceEnables AI agents to perform SEO tasks like keyword research, rank tracking, backlink analysis, site audits, and competitor insights using free data sources, with no subscription or per-call charges.4MIT