Skip to main content
Glama
arturseo-geo

mcp-common-crawl

by arturseo-geo

mcp-common-crawl

Built by Artur Ferreira @ The GEO Lab ยท ๐• @TheGEO_Lab ยท LinkedIn ยท Reddit

Version Licence Claude Code

MCP server for Common Crawl CDX โ€” backlink discovery, expired domain finder, competitor gap analysis. Free alternative to Ahrefs/Semrush backlink APIs ($100+/month).

Tools

Tool

Description

discover_backlinks

Find backlinks to any domain across 3 CC indexes

find_expired

Search for expired/parked domains in a niche via CC CDX

check_domain

Deep single domain check โ€” live/expired/parked + CC page count

competitor_gap

Find domains linking to competitors but not to you

Related MCP server: crawlgraph-mcp

Features

โœ… Production-tested โ€” patterns used in production at TheGEOLab

Install

# Claude Code
claude mcp add common-crawl -- npx mcp-common-crawl

# Or in .mcp.json
{
  "mcpServers": {
    "common-crawl": {
      "command": "npx",
      "args": ["mcp-common-crawl"]
    }
  }
}

No API Keys Required

Common Crawl is a free, open web archive. No API keys, no rate limits, no paid tiers.

Usage

> find backlinks to thegeolab.net using Common Crawl
> search for expired domains in the "seo tools" niche
> check if example.com is expired or parked
> find link gap between my site and competitors

Important Notes

  • Uses native fetch() for CC CDX (axios returns 404 on CC CDX โ€” known issue)

  • Queries the 3 most recent CC indexes for best coverage

  • Expired domain detection: ECONNREFUSED/ENOTFOUND = expired, parked page pattern matching for parked domains


Attributions & Licence

Built and maintained by Artur Ferreira @ TheGEOLab.

Email: artur@thegeolab.net

Best Practice Attribution

This MCP server was built following the open source Best Practice Approach โ€” reading community work for inspiration, then writing original content, and crediting every source.

Based on:

Data source:

Backlink analysis concepts inspired by:

  • Ahrefs โ€” backlink discovery and competitor gap methodology

  • Semrush โ€” backlink analytics and domain comparison

  • Majestic โ€” historic backlink index concepts

Technical decisions:

  • Native fetch() used instead of axios for CC CDX queries (axios returns 404 on CC CDX from inside Express โ€” persistent debugging issue documented in geolab-backlinks)

All server code is original writing. No files were copied or adapted from any source. MIT licence.


Found this useful? โญ Star the repo and connect: ๐ŸŒ thegeolab.net ยท ๐• @TheGEO_Lab ยท LinkedIn ยท Reddit

Licence

MIT โ€” see LICENSE


Built and maintained by Artur Ferreira @ TheGEOLab ยท MIT License

Available Tools

4 tools
check_domainA

Deep check a single domain โ€” liveness, parked page detection, Common Crawl page count, last seen date.

ParametersJSON Schema
NameRequiredDescriptionDefault
domainYesDomain to check

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden. It clearly discloses the tool's behavior by listing the specific checks performed (liveness, parked page, Common Crawl count, last seen date). This gives a concrete picture of the tool's operation, though it does not mention potential side effects or rate limits, which are likely minimal for a check operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, front-loaded sentence that efficiently conveys the tool's scope and key features. It contains no filler and every word adds value, making it appropriately concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter, no output schema), the description covers the main functional aspects by listing the checks performed. However, it does not explicitly describe the return format, which would be helpful in the absence of an output schema. Still, the listed checks implicitly indicate the output, making it reasonably complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema already documents the 'domain' parameter as 'Domain to check' with 100% coverage. The description adds no additional parameter semantics beyond reiterating it is a single domain, which is already implied by the singular noun in the schema. Thus, the description contributes no meaningful extra meaning.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'Deep check' with a clear resource 'a single domain' and enumerates concrete checks: liveness, parked page detection, Common Crawl page count, last seen date. This distinguishes it from sibling tools like discover_backlinks, find_expired, and competitor_gap, which focus on different analyses.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for analyzing a single domain's health and historical presence, but it does not explicitly state when to use this tool over alternatives or mention exclusions. The sibling tool names provide context, but the description itself lacks direct guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

competitor_gapA

Find domains linking to competitors but not to you. Queries Common Crawl for backlinks to each competitor, diffs against your domain.

ParametersJSON Schema
NameRequiredDescriptionDefault
competitorsYesCompetitor domains (max 5)
your_domainYesYour domain

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It discloses key behavioral traits: it queries Common Crawl (an external data source) and performs a diff against your domain. This gives the agent insight into data dependencies and the underlying mechanism, though it doesn't mention limitations like data freshness or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loaded with the core purpose, and every word earns its place. No superfluous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a relatively simple tool with two parameters and no output schema, the description provides sufficient context: what it does, the data source, and the diff logic. It could mention return format, but the phrase 'find domains' reasonably implies the result is a set of domains, so the description is complete enough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, with both parameters already described in the input schema. The description does not add additional parameter-level meaning beyond what the schema already provides, so a baseline score of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool finds domains linking to competitors but not to you, using a specific verb (Find) and resource (domains). It also explains the method (queries Common Crawl, diffs), and this distinguishes it from sibling tools like discover_backlinks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use it: when you want to identify link gap opportunities against competitors. It doesn't explicitly contrast with alternatives like discover_backlinks, but the use case is clear enough without exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

find_expiredA

Find expired or parked domains in a niche using Common Crawl. Searches CC CDX for domains matching keywords, checks liveness. Returns domain status (expired/parked/live).

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoMax domains to check (default 20)
keywordsYesNiche keywords to search for

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full transparency burden. It discloses the internal process (searches CC CDX, checks liveness) and the output (domain status), but it omits potential side behaviors like network latency, external dependency on Common Crawl, or how 'live' status is determined. This is adequate but not richly detailed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences, front-loading the main purpose and then adding process and output details. Every word contributes value, with no redundant information or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description must explain return values, and it does by specifying 'domain status (expired/parked/live)'. The tool is simple (2 params, no nested objects), and the description covers input, process, and output adequately for an agent to invoke it. Minor gaps like exact result structure or edge cases keep it from a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for both parameters, so the baseline is 3. The description references keywords in the context of niche matching but does not add deeper semantics about how limit is applied or how keywords are combined. It does not compensate beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool finds expired or parked domains in a niche using Common Crawl, which is a specific verb+resource+method. It distinguishes from siblings like check_domain (which likely checks a single domain) and discover_backlinks by focusing on finding multiple status-classified domains.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for niche-based searches of expired/parked domains, but it does not explicitly state when to prefer this over alternatives like check_domain. No exclusions or alternative tool references are provided, so guidance is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.

  1. 4 tool updatesv1.0.0
    • First observedcheck_domain
    • First observedcompetitor_gap
    • First observeddiscover_backlinks
    • First observedfind_expired

TDQS

A3.9/5.0
Disambiguation4/5

Tools have distinct purposes, but discover_backlinks and competitor_gap both involve backlink queries, while find_expired and check_domain both assess domain status. Descriptions clarify the differences, so ambiguity is low but not zero.

Naming Consistency2/5

Naming is inconsistent: discover_backlinks and check_domain use verb_noun, find_expired uses verb_adjective, and competitor_gap is a noun phrase. All use lowercase underscores, but the patterns are mixed.

Tool Count5/5

Four tools is well-scoped for a Common Crawl domain analysis server. Each tool addresses a distinct task without redundancy, and the count fits the niche purpose perfectly.

Completeness4/5

The surface covers core domain-backlink workflows: backlink discovery, expired domain hunting, domain status checks, and competitor gap analysis. A general URL search or content fetch tool is missing, but agents can work around it.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    B
    quality
    D
    maintenance
    A MCP server for retrieving backlink information for any domain(SEO).
    4
    257
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    MCP server for the CrawlGraph backlink-intelligence API. Gives any MCP client - Claude Desktop, Claude Code, Cursor, Cline, Zed, Windsurf - backlink lookups and competitor gap analysis built on the public Common Crawl webgraph (4.4B edges, 120M domains).
    4
    27
    5
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    MCP server & CLI for keyword research, domain analytics, backlinks, traffic analysis, and competitive intelligence using Semrush API data.
    77
    37
    39
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    MCP server and CLI for querying Common Crawl via AWS Athena, enabling users to check a domain's presence in crawls, analyze dynamics, compare domains, and inspect page text.
    13
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/arturseo-geo/mcp-common-crawl'

If you have feedback or need assistance with the MCP directory API, please join our Discord server