mcp-common-crawl
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@mcp-common-crawlfind expired domains in the marketing niche"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
mcp-common-crawl
Built by Artur Ferreira @ The GEO Lab ยท ๐ @TheGEO_Lab ยท LinkedIn ยท Reddit
MCP server for Common Crawl CDX โ backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).
Tools
Tool | Description |
| Find backlinks to any domain across 3 CC indexes |
| Search for expired/parked domains in a niche via CC CDX |
| Deep single domain check โ live/expired/parked + CC page count |
| Find domains linking to competitors but not to you |
Related MCP server: crawlgraph-mcp
Features
โ Production-tested โ patterns used in production at TheGEOLab
Install
# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl
# Or in .mcp.json
{
"mcpServers": {
"common-crawl": {
"command": "npx",
"args": ["mcp-common-crawl"]
}
}
}No API Keys Required
Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.
Usage
> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitorsImportant Notes
Uses native
fetch()for CC CDX (axios returns 404 on CC CDX โ known issue)Queries the 3 most recent CC indexes for best coverage
Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains
Attributions & Licence
Built and maintained by Artur Ferreira @ TheGEOLab.
Email: artur@thegeolab.net
Best Practice Attribution
This MCP server was built following the open source Best Practice Approach โ reading community work for inspiration, then writing original content, and crediting every source.
Based on:
Model Context Protocol specification by Anthropic
MCP SDK (MIT)
Data source:
Common Crawl โ free, open web archive (non-profit)
Common Crawl CDX API โ index search endpoint
Backlink analysis concepts inspired by:
Ahrefs โ backlink discovery and competitor gap methodology
Semrush โ backlink analytics and domain comparison
Majestic โ historic backlink index concepts
Technical decisions:
Native
fetch()used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express โ persistent debugging issue documented in geolab-backlinks)
All server code is original writing. No files were copied or adapted from any source. MIT licence.
Found this useful? โญ Star the repo and connect: ๐ thegeolab.net ยท ๐ @TheGEO_Lab ยท LinkedIn ยท Reddit
Related Repos
claude-code-mcps โ All 5 MCP servers in one collection
mcp-seo-auditor โ On-page SEO audit + JSON-LD validation
mcp-serp-intel โ SERP weak spots, PAA trees, intent comparison
mcp-common-crawl โ Free backlink discovery via Common Crawl
mcp-gsc-advanced โ GSC cannibalization, rank changes
mcp-wordpress-setup โ WordPress MCP server setup guide
Licence
MIT โ see LICENSE
Built and maintained by Artur Ferreira @ TheGEOLab ยท MIT License
Available Tools
4 toolscheck_domainA
Deep check a single domain โ liveness, parked page detection, Common Crawl page count, last seen date.
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Domain to check |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It clearly discloses the tool's behavior by listing the specific checks performed (liveness, parked page, Common Crawl count, last seen date). This gives a concrete picture of the tool's operation, though it does not mention potential side effects or rate limits, which are likely minimal for a check operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that efficiently conveys the tool's scope and key features. It contains no filler and every word adds value, making it appropriately concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema), the description covers the main functional aspects by listing the checks performed. However, it does not explicitly describe the return format, which would be helpful in the absence of an output schema. Still, the listed checks implicitly indicate the output, making it reasonably complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents the 'domain' parameter as 'Domain to check' with 100% coverage. The description adds no additional parameter semantics beyond reiterating it is a single domain, which is already implied by the singular noun in the schema. Thus, the description contributes no meaningful extra meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'Deep check' with a clear resource 'a single domain' and enumerates concrete checks: liveness, parked page detection, Common Crawl page count, last seen date. This distinguishes it from sibling tools like discover_backlinks, find_expired, and competitor_gap, which focus on different analyses.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for analyzing a single domain's health and historical presence, but it does not explicitly state when to use this tool over alternatives or mention exclusions. The sibling tool names provide context, but the description itself lacks direct guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
competitor_gapA
Find domains linking to competitors but not to you. Queries Common Crawl for backlinks to each competitor, diffs against your domain.
| Name | Required | Description | Default |
|---|---|---|---|
| competitors | Yes | Competitor domains (max 5) | |
| your_domain | Yes | Your domain |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It discloses key behavioral traits: it queries Common Crawl (an external data source) and performs a diff against your domain. This gives the agent insight into data dependencies and the underlying mechanism, though it doesn't mention limitations like data freshness or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core purpose, and every word earns its place. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a relatively simple tool with two parameters and no output schema, the description provides sufficient context: what it does, the data source, and the diff logic. It could mention return format, but the phrase 'find domains' reasonably implies the result is a set of domains, so the description is complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with both parameters already described in the input schema. The description does not add additional parameter-level meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds domains linking to competitors but not to you, using a specific verb (Find) and resource (domains). It also explains the method (queries Common Crawl, diffs), and this distinguishes it from sibling tools like discover_backlinks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use it: when you want to identify link gap opportunities against competitors. It doesn't explicitly contrast with alternatives like discover_backlinks, but the use case is clear enough without exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
discover_backlinksA
Discover backlinks to a domain using Common Crawl CDX API (3 indexes). Returns source URLs, anchor text, link context. Free alternative to Ahrefs/Semrush backlink API.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max results per index (default 50) | |
| domain | Yes | Target domain (e.g. thegeolab.net) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full transparency burden. It discloses the underlying API and the number of indexes, but it doesn't mention potential limitations of Common Crawl's data (e.g., crawl date coverage, incomplete index of the web) or rate limits. The claim 'Free alternative to Ahrefs/Semrush' could overpromise without caveats about data coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with no filler. The first sentence states what and how; the second gives return values and positioning. Every word contributes value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a simple parameter set and no output schema, and the description covers the main mental model: input domain, output backlink details, and free access. It could be slightly more complete by noting that results are limited to Common Crawl's stored data, but the mention of 'Common Crawl CDX API (3 indexes)' already hints at this. Overall, it is sufficient for selecting and using the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes both parameters (domain and limit with default 50), and description coverage is 100%. The description adds no additional semantic detail beyond schema, but that is acceptable because the schema is self-explanatory. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb phrase 'Discover backlinks to a domain' and specifies the data source (Common Crawl CDX API) and return values (source URLs, anchor text, link context). This clearly distinguishes it from sibling tools (find_expired, check_domain, competitor_gap) which address different use cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It positions the tool as a 'Free alternative to Ahrefs/Semrush backlink API,' indicating when to use it (when backlink data is needed without paid services). It doesn't explicitly exclude other tools, but its unique purpose and sibling names make the use case clear. However, it lacks explicit comparisons to alternatives like competitor_gap for competitor backlink analysis.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
find_expiredA
Find expired or parked domains in a niche using Common Crawl. Searches CC CDX for domains matching keywords, checks liveness. Returns domain status (expired/parked/live).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | Max domains to check (default 20) | |
| keywords | Yes | Niche keywords to search for |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full transparency burden. It discloses the internal process (searches CC CDX, checks liveness) and the output (domain status), but it omits potential side behaviors like network latency, external dependency on Common Crawl, or how 'live' status is determined. This is adequate but not richly detailed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the main purpose and then adding process and output details. Every word contributes value, with no redundant information or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, so the description must explain return values, and it does by specifying 'domain status (expired/parked/live)'. The tool is simple (2 params, no nested objects), and the description covers input, process, and output adequately for an agent to invoke it. Minor gaps like exact result structure or edge cases keep it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for both parameters, so the baseline is 3. The description references keywords in the context of niche matching but does not add deeper semantics about how limit is applied or how keywords are combined. It does not compensate beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool finds expired or parked domains in a niche using Common Crawl, which is a specific verb+resource+method. It distinguishes from siblings like check_domain (which likely checks a single domain) and discover_backlinks by focusing on finding multiple status-classified domains.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for niche-based searches of expired/parked domains, but it does not explicitly state when to prefer this over alternatives like check_domain. No exclusions or alternative tool references are provided, so guidance is only implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
4 tool updates
v1.0.0- First observed
check_domain - First observed
competitor_gap - First observed
discover_backlinks - First observed
find_expired
TDQS
Tools have distinct purposes, but discover_backlinks and competitor_gap both involve backlink queries, while find_expired and check_domain both assess domain status. Descriptions clarify the differences, so ambiguity is low but not zero.
Naming is inconsistent: discover_backlinks and check_domain use verb_noun, find_expired uses verb_adjective, and competitor_gap is a noun phrase. All use lowercase underscores, but the patterns are mixed.
Four tools is well-scoped for a Common Crawl domain analysis server. Each tool addresses a distinct task without redundancy, and the count fits the niche purpose perfectly.
The surface covers core domain-backlink workflows: backlink discovery, expired domain hunting, domain status checks, and competitor gap analysis. A general URL search or content fetch tool is missing, but agents can work around it.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Related MCP Connectors
SEO MCP server โ backlinks, domain authority, tech stack, and 18+ tools via Common Crawl.
- CalmSEOOAuthcom.calmseo
SEO MCP server for keyword research, SERP analysis, audits, and Search Console workflows.
- RampifyOAuthdev.rampify
SEO MCP server: crawl your site, find AI-visibility gaps, and ship the fix from your coding agent.
Related MCP Servers
- AlicenseBqualityDmaintenanceA MCP server for retrieving backlink information for any domain(SEO).4257MIT
- AlicenseAqualityBmaintenanceMCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).4275MIT
- AlicenseBqualityDmaintenanceMCP server & CLI for keyword research, domain analytics, backlinks, traffic analysis, and competitive intelligence using Semrush API data.773739MIT
- AlicenseAqualityCmaintenanceMCP server and CLI for querying Common Crawl via AWS Athena, enabling users to check a domain's presence in crawls, analyze dynamics, compare domains, and inspect page text.13MIT
Latest Blog Posts
- Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)By Om-Shree-0709 on .mcpAgent IdentityOAuth 2.1
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/arturseo-geo/mcp-common-crawl'
If you have feedback or need assistance with the MCP directory API, please join our Discord server