mcp-common-crawl

Built by Artur Ferreira @ The GEO Lab · 𝕏 @TheGEO_Lab · LinkedIn · Reddit

MCP server for Common Crawl CDX — backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).

Tools

Related MCP server: AgentWebSearch-MCP

Features

✅ Production-tested — patterns used in production at TheGEOLab

Install

# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl

# Or in .mcp.json
{
  "mcpServers": {
    "common-crawl": {
      "command": "npx",
      "args": ["mcp-common-crawl"]
    }
  }
}

No API Keys Required

Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.

Usage

> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitors

Important Notes

• Uses native fetch() for CC CDX (axios returns 404 on CC CDX — known issue)

• Queries the 3 most recent CC indexes for best coverage

• Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains

Attributions & Licence

Built and maintained by Artur Ferreira @ TheGEOLab.

Email: artur@thegeolab.net

Best Practice Attribution

This MCP server was built following the open source Best Practice Approach — reading community work for inspiration, then writing original content, and crediting every source.

Based on:

• Model Context Protocol specification by Anthropic

• MCP SDK (MIT)

Data source:

• Common Crawl — free, open web archive (non-profit)

• Common Crawl CDX API — index search endpoint

Backlink analysis concepts inspired by:

• Ahrefs — backlink discovery and competitor gap methodology

• Semrush — backlink analytics and domain comparison

• Majestic — historic backlink index concepts

Technical decisions:

• Native fetch() used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express — persistent debugging issue documented in geolab-backlinks)

All server code is original writing. No files were copied or adapted from any source. MIT licence.

Found this useful? ⭐ Star the repo and connect: 🌐 thegeolab.net · 𝕏 @TheGEO_Lab · LinkedIn · Reddit

Related Repos

• claude-code-mcps — All 5 MCP servers in one collection

• mcp-seo-auditor — On-page SEO audit + JSON-LD validation

• mcp-serp-intel — SERP weak spots, PAA trees, intent comparison

• mcp-common-crawl — Free backlink discovery via Common Crawl

• mcp-gsc-advanced — GSC cannibalization, rank changes

• mcp-wordpress-setup — WordPress MCP server setup guide

Licence

MIT — see LICENSE

Built and maintained by Artur Ferreira @ TheGEOLab · MIT License