CrawlBit MCP
Server Quality Checklist
Latest release: v1.0.1
- Disambiguation5/5
Each tool targets a distinct aspect of AI visibility: off-page mentions, product schema for shopping agents, crawler access, and entity recognition. There is no overlap, so an agent can unambiguously select the right tool for a given diagnostic need.
Naming Consistency5/5All tools share the 'crawlbit_' prefix followed by a descriptive snake_case suffix (offpage_gaps, shopping, crawler_watch, entity_check). The naming convention is consistent and readable, with only a minor variation in compound-word structure that does not cause confusion.
Tool Count5/5Four tools is an ideal size for a focused diagnostic server. Each tool addresses a separate facet of AI visibility, keeping the scope tight without unnecessary bloat or overwhelming the agent.
Completeness5/5The tool set covers the major pillars of AI visibility: being crawlable (crawler_watch), being recognizable (entity_check), having external validation (offpage_gaps), and being suitable for AI shopping (shopping). This is a complete diagnostic suite for the domain, with no obvious missing operations.
Average 4.4/5 across 4 of 4 tools scored.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 3 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI status not available
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add
glama.jsonto the root of your repository:{ "$schema": "https://glama.ai/mcp/schemas/server.json", "maintainers": [ "your-github-username" ] }Then . Browse examples.
Add related servers to improve discoverability.
How to sync the server with GitHub?
Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.
To manually sync the server, click the "Sync Server" button in the MCP server admin interface.
How is the quality score calculated?
The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).
Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.
Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).
Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.
Tool Scores
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden and does well by disclosing that it 'Reads the site's robots.txt' and reports per-crawler statuses, while also stating the limitation that it only reflects robots.txt and not CDN/firewall blocking. This gives the agent a clear picture of what the tool does and cannot do, though details like response format or rate limits are omitted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences long and front-loaded with the core function, followed by usage guidance and a limitation. Every sentence contributes meaningful information with no redundancy or filler, making it highly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (single parameter, no output schema, no annotations), the description is reasonably complete. It covers what the tool does, when to use it, and a key limitation. While it does not explicitly detail the return format, the phrase 'reports, per crawler, whether it is fully allowed, partially blocked or blocked entirely' gives the agent enough context. A small gap is the lack of mention of any network or rate-limit behaviors, but this is minor for a read-only check.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides a full description for the single 'url' parameter ('The site to analyse, e.g. example.com or https://example.com'), giving 100% schema coverage. The tool description does not add additional parameter-level guidance beyond what the schema offers, so a baseline score of 3 is appropriate as per the calibration rules.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Checks') and resource ('whether AI crawlers are allowed to read a site'), enumerates the exact crawlers covered, and clearly differentiates itself from sibling tools like offpage_gaps, shopping, and entity_check by focusing on robots.txt-based AI visibility. This leaves no ambiguity about the tool's function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use it ('when someone asks why AI never mentions their site, or before any other AI-visibility work') and provides a contextual exclusion ('cannot detect blocking done at the CDN or firewall layer'). It does not name alternative tools, but the 'before any other AI-visibility work' phrasing implies this is a prerequisite rather than a competing option, which is sufficient guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses that the tool measures on-site signals only and not web-wide reputation, which is a key behavioral trait. It could add more about return format or side effects, but for a read-only check this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, front-loaded with the purpose, and every sentence serves a purpose: purpose, context, usage, scope. No fluff or redundant repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity (one parameter, no output schema, no annotations), the description adequately covers what the tool does, when to use it, and its limitations. The only gap is not describing the output format, but that is not required for a straightforward check tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single url parameter, with a clear description in the schema. The tool description adds minimal semantics beyond implying the URL is the site to analyse. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool checks whether a brand reads as a clear, consistent AI-recognizable entity, listing specific signals (Organization schema, name, description, logo, sameAs). This is a specific verb+resource that clearly distinguishes it from sibling tools like offpage gaps or shopping.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit usage guidance is given: 'Use this when a brand is confused with another, or described vaguely, in AI answers.' It also excludes external reputation ('not the brand's reputation across the web'), clarifying its scope versus other possible tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of disclosure. It clearly states the tool inspects published structured data and cannot see private feeds, which is a key limitation. It also names what it evaluates (schema completeness, price, availability, attributes), giving a solid behavioral overview.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: the first states purpose, the second gives usage context, and the third describes a limitation. The description is front-loaded and concise without unnecessary fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description explains what the tool checks and its limitation, which sets appropriate expectations. It doesn't describe return format, but the scope of analysis is clear. For a diagnostic tool with one parameter and clear use cases, this is reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The single parameter url is fully described in the schema with examples (example.com or https://example.com), achieving 100% schema coverage. The description adds no extra parameter-level detail beyond what the schema provides, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks whether AI shopping agents can find, understand, and recommend products, focusing on schema completeness, price/availability signals, and attributes. This specific verb+resource scope distinguishes it from sibling tools like offpage_gaps or crawler_watch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use for e-commerce sites, especially Shopify, when products never surface in AI recommendations. Also provides exclusion: it cannot see privately submitted merchant feeds, which helps avoid misuse. No explicit alternatives are named, but the when-to-use guidance is strong.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states that the tool is non-destructive ('does not publish anything or contact anyone') and describes what it returns ('gaps and priorities'). This adds useful context beyond the schema, though it could mention whether any crawling or network requests are made.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is four sentences long, each earning its place: main action, rationale, when-to-use, and behavioral boundary. It is front-loaded with the core verb and resource, with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple single-parameter tool with no output schema, the description covers all essential decision-making information: what it finds, why it matters, when to use it, and what it returns. The high-level return phrase 'gaps and priorities' is adequate for an agent to select and invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%: the single required `url` parameter already has a clear description and example. The tool description adds no additional parameter-level semantics, but the parameter is self-explanatory, so the schema alone is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with the specific verb 'Finds' and names concrete resource types (review platforms, directories, communities, 'best of' roundups), making it unmistakably distinct from sibling tools like shopping or crawler watch. It also clarifies the scope as 'OUTSIDE its own website,' further sharpening the purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an explicit trigger: 'Use this when a site is technically perfect yet still never cited.' It also provides a when-not by stating that the tool 'does not publish anything or contact anyone,' so the agent knows it is only for analysis, not action.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
GitHub Badge
Glama performs regular codebase and documentation scans to:
- Confirm that the MCP server is working as expected.
- Confirm that there are no obvious security issues.
- Evaluate tool definition quality.
Our badge communicates server capabilities, safety, and installation instructions.
Card Badge
Copy to your README.md:
Score Badge
Copy to your README.md:
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/amati032-dev/crawlbit-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server