Skip to main content
Glama
mcp-common-crawl

mcp-common-crawl

by arturseo-geo

mcp-common-crawl

Built by Artur Ferreira @ The GEO Lab Β· 𝕏 @TheGEO_Lab Β· LinkedIn Β· Reddit

Version Licence Claude Code

MCP server for Common Crawl CDX β€” backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).

Tools

Tool

Description

discover_backlinks

Find backlinks to any domain across 3 CC indexes

find_expired

Search for expired/parked domains in a niche via CC CDX

check_domain

Deep single domain check β€” live/expired/parked + CC page count

competitor_gap

Find domains linking to competitors but not to you

Related MCP server: crawlgraph-mcp

Features

βœ… Production-tested β€” patterns used in production at TheGEOLab

Install

# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl

# Or in .mcp.json
{
  "mcpServers": {
    "common-crawl": {
      "command": "npx",
      "args": ["mcp-common-crawl"]
    }
  }
}

No API Keys Required

Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.

Usage

> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitors

Important Notes

  • Uses native fetch() for CC CDX (axios returns 404 on CC CDX β€” known issue)

  • Queries the 3 most recent CC indexes for best coverage

  • Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains


Attributions & Licence

Built and maintained by Artur Ferreira @ TheGEOLab.

Email: artur@thegeolab.net

Best Practice Attribution

This MCP server was built following the open source Best Practice Approach β€” reading community work for inspiration, then writing original content, and crediting every source.

Based on:

Data source:

Backlink analysis concepts inspired by:

  • Ahrefs β€” backlink discovery and competitor gap methodology

  • Semrush β€” backlink analytics and domain comparison

  • Majestic β€” historic backlink index concepts

Technical decisions:

  • Native fetch() used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express β€” persistent debugging issue documented in geolab-backlinks)

All server code is original writing. No files were copied or adapted from any source. MIT licence.


Found this useful? ⭐ Star the repo and connect: 🌐 thegeolab.net Β· 𝕏 @TheGEO_Lab Β· LinkedIn Β· Reddit

Licence

MIT β€” see LICENSE


Built and maintained by Artur Ferreira @ TheGEOLab Β· MIT License

Available Tools

4 tools
check_domainA

Deep check a single domain β€” liveness, parked page detection, Common Crawl page count, last seen date.

Inputs: domain (required).

competitor_gapA

Find domains linking to competitors but not to you. Queries Common Crawl for backlinks to each competitor, diffs against your domain.

Inputs: competitors (required), your_domain (required).

find_expiredA

Find expired or parked domains in a niche using Common Crawl. Searches CC CDX for domains matching keywords, checks liveness. Returns domain status (expired/parked/live).

Inputs: keywords (required), limit.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 4 tools

    Disambiguation4/5

    Tools have distinct purposes, but discover_backlinks and competitor_gap both involve backlink queries, while find_expired and check_domain both assess domain status. Descriptions clarify the differences, so ambiguity is low but not zero.

    Naming Consistency2/5

    Naming is inconsistent: discover_backlinks and check_domain use verb_noun, find_expired uses verb_adjective, and competitor_gap is a noun phrase. All use lowercase underscores, but the patterns are mixed.

    Tool Count5/5

    Four tools is well-scoped for a Common Crawl domain analysis server. Each tool addresses a distinct task without redundancy, and the count fits the niche purpose perfectly.

    Completeness4/5

    The surface covers core domain-backlink workflows: backlink discovery, expired domain hunting, domain status checks, and competitor gap analysis. A general URL search or content fetch tool is missing, but agents can work around it.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      A MCP server for retrieving backlink information for any domain(SEO).
      4
      61 PyPI
      259
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      MCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).
      4
      16 npm
      8
      MIT
    • A
      license
      B
      quality
      C
      maintenance
      MCP server & CLI for keyword research, domain analytics, backlinks, traffic analysis, and competitive intelligence using Semrush API data.
      77
      40
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      MCP server and CLI for querying Common Crawl via AWS Athena, enabling users to check a domain's presence in crawls, analyze dynamics, compare domains, and inspect page text.
      13
      MIT