mcp-common-crawl
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@mcp-common-crawl find expired domains in the marketing niche"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
mcp-common-crawl
Built by Artur Ferreira @ The GEO Lab Β· π @TheGEO_Lab Β· LinkedIn Β· Reddit
MCP server for Common Crawl CDX β backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).
Tools
Tool | Description |
| Find backlinks to any domain across 3 CC indexes |
| Search for expired/parked domains in a niche via CC CDX |
| Deep single domain check β live/expired/parked + CC page count |
| Find domains linking to competitors but not to you |
Related MCP server: crawlgraph-mcp
Features
β Production-tested β patterns used in production at TheGEOLab
Install
# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl
# Or in .mcp.json
{
"mcpServers": {
"common-crawl": {
"command": "npx",
"args": ["mcp-common-crawl"]
}
}
}No API Keys Required
Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.
Usage
> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitorsImportant Notes
Uses native
fetch()for CC CDX (axios returns 404 on CC CDX β known issue)Queries the 3 most recent CC indexes for best coverage
Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains
Attributions & Licence
Built and maintained by Artur Ferreira @ TheGEOLab.
Email: artur@thegeolab.net
Best Practice Attribution
This MCP server was built following the open source Best Practice Approach β reading community work for inspiration, then writing original content, and crediting every source.
Based on:
Model Context Protocol specification by Anthropic
MCP SDK (MIT)
Data source:
Common Crawl β free, open web archive (non-profit)
Common Crawl CDX API β index search endpoint
Backlink analysis concepts inspired by:
Ahrefs β backlink discovery and competitor gap methodology
Semrush β backlink analytics and domain comparison
Majestic β historic backlink index concepts
Technical decisions:
Native
fetch()used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express β persistent debugging issue documented in geolab-backlinks)
All server code is original writing. No files were copied or adapted from any source. MIT licence.
Found this useful? β Star the repo and connect: π thegeolab.net Β· π @TheGEO_Lab Β· LinkedIn Β· Reddit
Related Repos
claude-code-mcps β All 5 MCP servers in one collection
mcp-seo-auditor β On-page SEO audit + JSON-LD validation
mcp-serp-intel β SERP weak spots, PAA trees, intent comparison
mcp-common-crawl β Free backlink discovery via Common Crawl
mcp-gsc-advanced β GSC cannibalization, rank changes
mcp-wordpress-setup β WordPress MCP server setup guide
Licence
MIT β see LICENSE
Built and maintained by Artur Ferreira @ TheGEOLab Β· MIT License
Available Tools
4 toolscheck_domainA
Deep check a single domain β liveness, parked page detection, Common Crawl page count, last seen date.
Inputs: domain (required).
competitor_gapA
Find domains linking to competitors but not to you. Queries Common Crawl for backlinks to each competitor, diffs against your domain.
Inputs: competitors (required), your_domain (required).
discover_backlinksA
Discover backlinks to a domain using Common Crawl CDX API (3 indexes). Returns source URLs, anchor text, link context. Free alternative to Ahrefs/Semrush backlink API.
Inputs: domain (required), limit.
find_expiredA
Find expired or parked domains in a niche using Common Crawl. Searches CC CDX for domains matching keywords, checks liveness. Returns domain status (expired/parked/live).
Inputs: keywords (required), limit.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
TDQS
Scored across 4 tools
Tools have distinct purposes, but discover_backlinks and competitor_gap both involve backlink queries, while find_expired and check_domain both assess domain status. Descriptions clarify the differences, so ambiguity is low but not zero.
Naming is inconsistent: discover_backlinks and check_domain use verb_noun, find_expired uses verb_adjective, and competitor_gap is a noun phrase. All use lowercase underscores, but the patterns are mixed.
Four tools is well-scoped for a Common Crawl domain analysis server. Each tool addresses a distinct task without redundancy, and the count fits the niche purpose perfectly.
The surface covers core domain-backlink workflows: backlink discovery, expired domain hunting, domain status checks, and competitor gap analysis. A general URL search or content fetch tool is missing, but agents can work around it.
Related MCP Connectors
SEO MCP server β backlinks, domain authority, tech stack, and 18+ tools via Common Crawl.
SEO MCP server for keyword research, live SERPs, backlinks, site audits and AI visibility.
Free MCP server that audits any public URL: technical SEO, Core Web Vitals, indexing checks.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Related MCP Servers
- AlicenseBqualityDmaintenanceA MCP server for retrieving backlink information for any domain(SEO).461 PyPI259MIT
- AlicenseAqualityCmaintenanceMCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).416 npm8MIT
- AlicenseBqualityCmaintenanceMCP server & CLI for keyword research, domain analytics, backlinks, traffic analysis, and competitive intelligence using Semrush API data.7740MIT
- AlicenseAqualityDmaintenanceMCP server and CLI for querying Common Crawl via AWS Athena, enabling users to check a domain's presence in crawls, analyze dynamics, compare domains, and inspect page text.13MIT