URL Reputation and Validity Checker

A FASTMCP 2.0-based service that validates URLs and checks their reputation to help identify AI hallucinations and verify web page authenticity.

Features

URL Validation: Verify that URLs resolve to actual web pages
Reputation Scoring: 0-100 score based on domain age, web archive presence, and technical factors
Link Extraction: Extract and validate links from HTML or text content
Historical Analysis: Check domain age via WHOIS and Wayback Machine
AI Hallucination Detection: Identify patterns common in AI-generated URLs
Caching: Redis-based caching for improved performance
Docker Support: Easy deployment with Docker and docker-compose

Quick Start

Using Docker (Recommended)

Clone the repository:

git clone <repository-url>
cd reputation-and-validity-checker

Start the services:

docker-compose up -d

The MCP server will be available at http://localhost:5000.

Local Installation

Install dependencies:

pip install -r requirements.txt

Start Redis (optional, for caching):

docker run -d -p 6379:6379 redis:7-alpine

Run the server:

python -m url_reputation_checker.server

MCP Tools

`check_links_reputation`

Check reputation for a list of URLs.

Parameters:

urls: List of URLs to validate

Returns:

List of validation results with reputation scores

Example:

result = await check_links_reputation([
    "https://example.com",
    "https://github.com"
])

`extract_and_check_links`

Extract links from content and check their reputation.

Parameters:

content: HTML or text content
content_type: "html", "text", or "auto" (default: "auto")

Returns:

Extracted links with validation results and summary

Example:

result = await extract_and_check_links(
    "<a href='https://example.com'>Link</a>",
    content_type="html"
)

`validate_url`

Validate a single URL.

Parameters:

url: URL to validate

Returns:

Detailed validation information

`get_domain_history`

Get historical information about a domain.

Parameters:

domain: Domain name or URL

Returns:

Domain creation date, WHOIS info, and Wayback Machine data

Reputation Scoring

The reputation score (0-100) is calculated based on:

Domain Age (0-30 points):
- 5+ years: 30 points
- 2-5 years: 20 points
- 1-2 years: 15 points
- 6-12 months: 10 points
- 3-6 months: 5 points
- <3 months: 2 points
Web Archive Presence (0-20 points):
- 100+ snapshots: 20 points
- 50-99 snapshots: 15 points
- 20-49 snapshots: 10 points
- 5-19 snapshots: 5 points
- 1-4 snapshots: 2 points
Technical Factors (0-25 points):
- Valid SSL: 10 points
- Fast response (<1s): 10 points
- HTTP 200 status: 5 points
Consistency (0-25 points):
- No warnings: 25 points
- 1 warning: 15 points
- 2 warnings: 10 points
- 3 warnings: 5 points

AI Hallucination Detection

The service checks for patterns commonly seen in AI-generated URLs:

Overly specific blog paths (e.g., /blog/2023/03/15/specific-topic)
Version-specific API documentation URLs
Excessive path depth (>6 levels)
Over-hyphenated product names
Typosquatting attempts
Excessive subdomains

Configuration

Environment variables:

REDIS_URL: Redis connection URL (default: redis://localhost:6379)
MCP_SERVER_HOST: Server host (default: 0.0.0.0)
MCP_SERVER_PORT: Server port (default: 5000)

Development

Install development dependencies:

pip install -e ".[dev]"

Run tests:

pytest

Format code:

black url_reputation_checker/

Lint code:

ruff url_reputation_checker/

Docker Build

Build the Docker image:

docker build -t url-reputation-checker .

Run with docker-compose:

docker-compose up -d

View logs:

docker-compose logs -f

Example Usage

Python Client Example

import asyncio
from fastmcp import FastMCPClient

async def main():
    async with FastMCPClient("http://localhost:5000") as client:
        # Check a list of URLs
        results = await client.call_tool(
            "check_links_reputation",
            urls=[
                "https://github.com",
                "https://example-fake-site-12345.com"
            ]
        )
        
        for result in results:
            print(f"{result['url']}: {result['reputation_score']}/100")
        
        # Extract and check links from HTML
        html_content = """
        <html>
            <body>
                <a href="https://google.com">Google</a>
                <a href="https://fake-news-site.com">Fake News</a>
            </body>
        </html>
        """
        
        extraction_result = await client.call_tool(
            "extract_and_check_links",
            content=html_content,
            content_type="html"
        )
        
        print(f"Found {extraction_result['summary']['total_links']} links")
        print(f"Average reputation: {extraction_result['summary']['average_reputation_score']}")

if __name__ == "__main__":
    asyncio.run(main())

API Endpoints

The MCP server exposes tools that can be called via the MCP protocol. Additionally, it provides resources:

/url_validation_report: Get a formatted report of all validated URLs
/cache_stats: Get cache statistics

Troubleshooting

Redis Connection Issues

If Redis is not available, the service will continue to work without caching. To disable Redis entirely, set:

export REDIS_URL=""

WHOIS Lookup Failures

Some domains may not have WHOIS information available due to privacy protection or registry limitations. The service will continue with other checks.

Wayback Machine Rate Limiting

The Wayback Machine API may rate limit requests. The service handles this gracefully and will skip historical data if unavailable.

License

MIT License - see LICENSE file for details.

This server cannot be installed

security - not tested

license - permissive license

quality - not tested

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

Validates URLs and checks their reputation to help identify AI hallucinations and verify web page authenticity.

Related MCP Servers

WordPress MCP Server
emzimmer
-
security
A
license
-
quality
Enables AI assistants to interact with WordPress sites through the WordPress REST API. Supports multiple WordPress sites with secure authentication, enabling content management, post operations, and site configuration through natural language.
Last updated -
18
31
MIT License
Dub.co MCP Server
Gitmaxd
A
security
A
license
A
quality
Enables AI agents to create, update, and manage short links through your Dub.co account, allowing for creation, modification, and deletion of custom shortened URLs.
Last updated -
4
11
JavaScript
MIT License
WordPress MCP Server
LiamCioci
-
security
A
license
-
quality
Enables AI assistants to interact with WordPress sites through the REST API. Supports multiple WordPress sites with secure authentication, enabling content management, post operations, and site configuration through natural language.
Last updated -
18
MIT License
WordPress MCP Server
prathammanocha
A
security
A
license
A
quality
Enables AI assistants to interact with WordPress sites through REST APIs, allowing programmatic management of posts, users, comments, categories, and tags with secure authentication.
Last updated -
29
32
JavaScript
MIT License

View all related MCP servers

URL Reputation and Validity Checker