CrawlBit MCP
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| CRAWLBIT_BASE_URL | No | Point the server at another instance. | https://www.crawlbit.app |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": true
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| crawlbit_crawler_watchA | Checks whether AI crawlers are allowed to read a site: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. Reads the site's robots.txt and reports, per crawler, whether it is fully allowed, partially blocked or blocked entirely. Use this when someone asks why AI never mentions their site, or before any other AI-visibility work: a blocked crawler makes everything else pointless. Reflects robots.txt only, so it cannot detect blocking done at the CDN or firewall layer. |
| crawlbit_entity_checkA | Checks whether a brand reads as a clear, consistent entity that AI can recognise: Organization schema, a consistent name, a description, a logo and linked profiles (sameAs). AI engines struggle to name a brand they cannot identify as a distinct thing. Use this when a brand is confused with another, or described vaguely, in AI answers. Measures the signals published on the site itself, not the brand's reputation across the web. |
| crawlbit_offpage_gapsA | Finds where a brand is missing OUTSIDE its own website: review platforms, directories, communities and 'best of' roundups relevant to its category. Off-page mentions on third-party pages are the strongest known predictor of AI citations, because engines weight what others say about a brand far above what the brand says about itself. Use this when a site is technically perfect yet still never cited. Returns the gaps and priorities; it does not publish anything or contact anyone. |
| crawlbit_shoppingA | Checks whether AI shopping agents such as ChatGPT Shopping can find, understand and recommend a store's products: product schema completeness, price and availability signals, and the attributes those agents read. Use this for e-commerce sites, especially Shopify, when products never surface in AI recommendations. Inspects published structured data, so it cannot see a merchant feed submitted privately to a platform. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Each tool targets a distinct aspect of AI visibility: off-page mentions, product schema for shopping agents, crawler access, and entity recognition. There is no overlap, so an agent can unambiguously select the right tool for a given diagnostic need.
All tools share the 'crawlbit_' prefix followed by a descriptive snake_case suffix (offpage_gaps, shopping, crawler_watch, entity_check). The naming convention is consistent and readable, with only a minor variation in compound-word structure that does not cause confusion.
Four tools is an ideal size for a focused diagnostic server. Each tool addresses a separate facet of AI visibility, keeping the scope tight without unnecessary bloat or overwhelming the agent.
The tool set covers the major pillars of AI visibility: being crawlable (crawler_watch), being recognizable (entity_check), having external validation (offpage_gaps), and being suitable for AI shopping (shopping). This is a complete diagnostic suite for the domain, with no obvious missing operations.