mcp-common-crawl
Server Quality Checklist
Latest release: v1.0.0
- Disambiguation4/5
Tools have distinct purposes, but discover_backlinks and competitor_gap both involve backlink queries, while find_expired and check_domain both assess domain status. Descriptions clarify the differences, so ambiguity is low but not zero.
Naming Consistency2/5Naming is inconsistent: discover_backlinks and check_domain use verb_noun, find_expired uses verb_adjective, and competitor_gap is a noun phrase. All use lowercase underscores, but the patterns are mixed.
Tool Count5/5Four tools is well-scoped for a Common Crawl domain analysis server. Each tool addresses a distinct task without redundancy, and the count fits the niche purpose perfectly.
Completeness4/5The surface covers core domain-backlink workflows: backlink discovery, expired domain hunting, domain status checks, and competitor gap analysis. A general URL search or content fetch tool is missing, but agents can work around it.
Average 4/5 across 4 of 4 tools scored.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 0 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI status not available
Add a LICENSE file by following GitHub's guide. Once GitHub recognizes the license, the system will automatically detect it within a few hours.
If the license does not appear after some time, you can manually trigger a new scan using the MCP server admin interface.
MCP servers without a LICENSE cannot be installed.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add
glama.jsonto the root of your repository:{ "$schema": "https://glama.ai/mcp/schemas/server.json", "maintainers": [ "your-github-username" ] }Then . Browse examples.
Add related servers to improve discoverability.
How to sync the server with GitHub?
Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.
To manually sync the server, click the "Sync Server" button in the MCP server admin interface.
How is the quality score calculated?
The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).
Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.
Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).
Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.
Tool Scores
- Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full transparency burden. It discloses the internal process (searches CC CDX, checks liveness) and the output (domain status), but it omits potential side behaviors like network latency, external dependency on Common Crawl, or how 'live' status is determined. This is adequate but not richly detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the main purpose and then adding process and output details. Every word contributes value, with no redundant information or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description must explain return values, and it does by specifying 'domain status (expired/parked/live)'. The tool is simple (2 params, no nested objects), and the description covers input, process, and output adequately for an agent to invoke it. Minor gaps like exact result structure or edge cases keep it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for both parameters, so the baseline is 3. The description references keywords in the context of niche matching but does not add deeper semantics about how limit is applied or how keywords are combined. It does not compensate beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds expired or parked domains in a niche using Common Crawl, which is a specific verb+resource+method. It distinguishes from siblings like check_domain (which likely checks a single domain) and discover_backlinks by focusing on finding multiple status-classified domains.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for niche-based searches of expired/parked domains, but it does not explicitly state when to prefer this over alternatives like check_domain. No exclusions or alternative tool references are provided, so guidance is only implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It clearly discloses the tool's behavior by listing the specific checks performed (liveness, parked page, Common Crawl count, last seen date). This gives a concrete picture of the tool's operation, though it does not mention potential side effects or rate limits, which are likely minimal for a check operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently conveys the tool's scope and key features. It contains no filler and every word adds value, making it appropriately concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema), the description covers the main functional aspects by listing the checks performed. However, it does not explicitly describe the return format, which would be helpful in the absence of an output schema. Still, the listed checks implicitly indicate the output, making it reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents the 'domain' parameter as 'Domain to check' with 100% coverage. The description adds no additional parameter semantics beyond reiterating it is a single domain, which is already implied by the singular noun in the schema. Thus, the description contributes no meaningful extra meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Deep check' with a clear resource 'a single domain' and enumerates concrete checks: liveness, parked page detection, Common Crawl page count, last seen date. This distinguishes it from sibling tools like discover_backlinks, find_expired, and competitor_gap, which focus on different analyses.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for analyzing a single domain's health and historical presence, but it does not explicitly state when to use this tool over alternatives or mention exclusions. The sibling tool names provide context, but the description itself lacks direct guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full transparency burden. It discloses the underlying API and the number of indexes, but it doesn't mention potential limitations of Common Crawl's data (e.g., crawl date coverage, incomplete index of the web) or rate limits. The claim 'Free alternative to Ahrefs/Semrush' could overpromise without caveats about data coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with no filler. The first sentence states what and how; the second gives return values and positioning. Every word contributes value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a simple parameter set and no output schema, and the description covers the main mental model: input domain, output backlink details, and free access. It could be slightly more complete by noting that results are limited to Common Crawl's stored data, but the mention of 'Common Crawl CDX API (3 indexes)' already hints at this. Overall, it is sufficient for selecting and using the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes both parameters (domain and limit with default 50), and description coverage is 100%. The description adds no additional semantic detail beyond schema, but that is acceptable because the schema is self-explanatory. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb phrase 'Discover backlinks to a domain' and specifies the data source (Common Crawl CDX API) and return values (source URLs, anchor text, link context). This clearly distinguishes it from sibling tools (find_expired, check_domain, competitor_gap) which address different use cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5Does the description explain when to use this tool, when not to, or what alternatives exist?
It positions the tool as a 'Free alternative to Ahrefs/Semrush backlink API,' indicating when to use it (when backlink data is needed without paid services). It doesn't explicitly exclude other tools, but its unique purpose and sibling names make the use case clear. However, it lacks explicit comparisons to alternatives like competitor_gap for competitor backlink analysis.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
- Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses key behavioral traits: it queries Common Crawl (an external data source) and performs a diff against your domain. This gives the agent insight into data dependencies and the underlying mechanism, though it doesn't mention limitations like data freshness or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose, and every word earns its place. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a relatively simple tool with two parameters and no output schema, the description provides sufficient context: what it does, the data source, and the diff logic. It could mention return format, but the phrase 'find domains' reasonably implies the result is a set of domains, so the description is complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with both parameters already described in the input schema. The description does not add additional parameter-level meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds domains linking to competitors but not to you, using a specific verb (Find) and resource (domains). It also explains the method (queries Common Crawl, diffs), and this distinguishes it from sibling tools like discover_backlinks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it: when you want to identify link gap opportunities against competitors. It doesn't explicitly contrast with alternatives like discover_backlinks, but the use case is clear enough without exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
GitHub Badge
Glama performs regular codebase and documentation scans to:
- Confirm that the MCP server is working as expected.
- Confirm that there are no obvious security issues.
- Evaluate tool definition quality.
Our badge communicates server capabilities, safety, and installation instructions.
Card Badge
Copy to your README.md:
Score Badge
Copy to your README.md:
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/arturseo-geo/mcp-common-crawl'
If you have feedback or need assistance with the MCP directory API, please join our Discord server