Skip to main content
Glama

๐ŸŽฏ Competitor Hunter

AI-Powered Competitor Analysis Agent | Automated web scraping and structured data extraction using MCP, LangGraph, and Playwright

License: MIT Python 3.10+ Code style: black

Language | ่ฏญ่จ€: English | ไธญๆ–‡


๐Ÿ“– Introduction

Competitor Hunter is a production-ready AI agent that automates competitor analysis by scraping product pages and extracting structured information using Large Language Models. Built on the Model Context Protocol (MCP), it seamlessly integrates with Claude Desktop and other MCP-compatible clients.

Key Capabilities

  • ๐Ÿ” Intelligent Web Scraping: Automated browser-based content extraction with anti-detection features

  • ๐Ÿค– LLM-Powered Extraction: Structured data extraction using OpenAI-compatible APIs

  • ๐Ÿ“Š Structured Output: Pydantic-validated product information (pricing, features, SWOT analysis)

  • ๐Ÿ”„ LangGraph Workflow: Robust state management and error handling

  • ๐Ÿ”Œ MCP Integration: Native support for Claude Desktop and MCP clients


Related MCP server: ZapFetch MCP Server

๐Ÿ—๏ธ Architecture

The system follows Hexagonal Architecture with clear separation of concerns. Workflow: User Request โ†’ MCP Server โ†’ LangGraph Workflow โ†’ Browser Scraping โ†’ LLM Extraction โ†’ Structured Data Response.


โœจ Core Features

  • ๐Ÿค– AI-Powered: Intelligent extraction using LLM with automatic SWOT analysis

  • ๐Ÿ“Š Structured Output: Pydantic-validated data models (pricing, features, summary)

  • ๐Ÿ›ก๏ธ Anti-Detection: Random User-Agents, intelligent scrolling, auto-screenshots

  • ๐Ÿ”Œ MCP Native: Seamless integration with Claude Desktop and Cursor IDE

  • ๐Ÿ“ฆ CLI Tool: Professional command-line interface via competitor-hunter command

  • Async/Await: Full asynchronous programming for optimal performance


๐Ÿš€ Quick Start

Prerequisites

  • Python 3.10+ (3.11 or 3.12 recommended)

  • UV or Poetry (dependency manager)

  • Playwright browsers (installed automatically)

Installation

  1. Clone the repository:

    git clone https://github.com/your-username/competitor-hunter.git
    cd competitor-hunter
  2. Install dependencies (using UV):

    uv sync

    Or using Poetry:

    poetry install

    Or using pip:

    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install -e ".[dev]"
  3. Install Playwright browsers:

    playwright install chromium

Configuration

Create a .env file in the project root:

# OpenAI API Configuration
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1  # Optional: for custom endpoints
OPENAI_MODEL_NAME=gpt-4o                    # Optional: default is gpt-4o

# Browser Configuration
HEADLESS_MODE=true                          # Set to false for debugging

# Database Configuration
DB_PATH=data/competitors.db                 # SQLite database path

๐Ÿ’ก Tip: Copy .env.example to .env and fill in your values:

cp .env.example .env

๐Ÿ“ธ Screenshots & Examples

Analysis Results

Notion Pricing Analysis

Screenshot of Notion pricing page analysis

CLI Output Example

$ competitor-hunter https://www.notion.so/pricing
๐Ÿ” ๆญฃๅœจๅˆ†ๆž: https://www.notion.so/pricing

โœ… ๅˆ†ๆžๅฎŒๆˆ๏ผ

======================================================================
๐Ÿ“ฆ ไบงๅ“ๅ็งฐ: Notion
๐Ÿ”— URL: https://www.notion.so/pricing
๐Ÿ•’ ๆ›ดๆ–ฐๆ—ถ้—ด: 2024-06-13 00:00:00+00:00
======================================================================

๐Ÿ’ฐ ๅฎšไปทๆ–นๆกˆ (4 ไธช):
   โ€ข Free: 0 USD / monthly
   โ€ข Plus: 10 USD / monthly
   โ€ข Business: 20 USD / monthly
   โ€ข Enterprise: Custom USD / custom

โœจ ๆ ธๅฟƒๅŠŸ่ƒฝ (13 ไธช):
   1. AI automation
   2. Enterprise search
   3. Meeting notes
   ...

๐Ÿ’พ ็ป“ๆžœๅทฒไฟๅญ˜ๅˆฐ: reports/product_Notion.json

JSON Output Structure

The analysis results are saved as structured JSON files:

{
  "product_name": "Notion",
  "url": "https://www.notion.so/pricing",
  "pricing_tiers": [
    {
      "name": "Free",
      "price": "0",
      "currency": "USD",
      "billing_cycle": "monthly"
    },
    {
      "name": "Plus",
      "price": "10",
      "currency": "USD",
      "billing_cycle": "monthly"
    }
  ],
  "core_features": [
    "AI automation",
    "Docs",
    "Knowledge Base"
  ],
  "summary": "## ไบงๅ“ๆฆ‚่ฟฐ\nNotion ๆ˜ฏไธ€ๆฌพ้›†ๆ–‡ๆกฃ็ผ–่พ‘...",
  "last_updated": "2024-06-13T00:00:00Z"
}

๐Ÿ“š Usage

Method 1: CLI Command (Easiest)

After installation, use the competitor-hunter command:

# Analyze a single website
competitor-hunter https://www.notion.so/pricing

# Specify output file
competitor-hunter https://example.com output.json

# Batch analysis
competitor-hunter https://site1.com https://site2.com https://site3.com

Results are automatically saved to the reports/ directory with proper UTF-8 encoding.

Run the MCP server to enable integration with Claude Desktop or Cursor:

python -m src.competitor_hunter.interface.mcp_server.server

Claude Desktop Integration

Add the following configuration to your Claude Desktop claude_desktop_config.json:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "competitor-hunter": {
      "command": "python",
      "args": [
        "-m",
        "src.competitor_hunter.interface.mcp_server.server"
      ],
      "cwd": "/path/to/competitor-hunter"
    }
  }
}

Cursor IDE Integration

Create .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "competitor-hunter": {
      "command": "python",
      "args": [
        "-m",
        "src.competitor_hunter.interface.mcp_server.server"
      ],
      "cwd": "${workspaceFolder}"
    }
  }
}

After restarting, you can use the tool directly in chat:

Analyze this competitor: https://www.notion.so/pricing

Method 3: Python Library

Use the LangGraph workflow directly in your Python code:

import asyncio
from competitor_hunter.core import graph, AgentState, cleanup_resources

async def analyze(url: str):
    # Initialize state
    initial_state: AgentState = {
        "url": url,
        "scraped_content": None,
        "product": None,
        "error": None,
    }
    
    # Run workflow
    result = await graph.ainvoke(initial_state)
    
    # Check results
    if result.get("error"):
        print(f"Error: {result['error']}")
        return None
    
    product = result["product"]
    print(f"Product: {product.product_name}")
    print(f"Pricing Tiers: {len(product.pricing_tiers)}")
    print(f"Features: {product.core_features}")
    
    return product

# Use
product = await analyze("https://www.notion.so/pricing")
await cleanup_resources()

Output Structure

All analysis results are saved to the reports/ directory:

reports/
โ”œโ”€โ”€ product_Notion.json
โ”œโ”€โ”€ product_Example_Domain.json
โ””โ”€โ”€ ...

Each JSON file contains:

  • Product name and URL

  • Pricing tiers (name, price, currency, billing cycle)

  • Core features list

  • Markdown-formatted summary with SWOT analysis

  • Last updated timestamp


๐Ÿงช Development

Running Tests

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_crawler.py -v

# Run with coverage
pytest tests/ --cov=src/competitor_hunter --cov-report=html

Code Quality

# Format code
black src/ tests/

# Lint code
ruff check src/ tests/

# Type checking (if using mypy)
mypy src/

Project Structure

competitor-hunter/
โ”œโ”€โ”€ src/
โ”‚   โ””โ”€โ”€ competitor_hunter/
โ”‚       โ”œโ”€โ”€ cli.py             # CLI command-line interface
โ”‚       โ”œโ”€โ”€ main.py            # Application entry point
โ”‚       โ”œโ”€โ”€ config.py          # Configuration management
โ”‚       โ”œโ”€โ”€ core/               # Domain models & LangGraph workflow
โ”‚       โ”‚   โ”œโ”€โ”€ models.py       # Pydantic models (CompetitorProduct, etc.)
โ”‚       โ”‚   โ””โ”€โ”€ graph.py        # LangGraph workflow definition
โ”‚       โ”œโ”€โ”€ infrastructure/     # External services
โ”‚       โ”‚   โ”œโ”€โ”€ browser/        # Playwright browser service
โ”‚       โ”‚   โ””โ”€โ”€ llm/            # LLM extractor service
โ”‚       โ””โ”€โ”€ interface/          # Entry points
โ”‚           โ””โ”€โ”€ mcp_server/     # MCP server implementation
โ”œโ”€โ”€ config/                     # Configuration files
โ”‚   โ””โ”€โ”€ app.yaml.example        # Configuration template
โ”œโ”€โ”€ docker/                     # Docker configuration
โ”‚   โ”œโ”€โ”€ Dockerfile              # Docker image definition
โ”‚   โ””โ”€โ”€ docker-compose.yml     # Docker Compose configuration
โ”œโ”€โ”€ examples/                   # Example scripts
โ”œโ”€โ”€ tests/                      # Test suite
โ”œโ”€โ”€ reports/                    # Analysis results (gitignored)
โ”œโ”€โ”€ data/                       # SQLite database (gitignored)
โ”œโ”€โ”€ logs/                       # Screenshots & logs (gitignored)
โ”œโ”€โ”€ pyproject.toml              # Project dependencies & CLI entry points
โ””โ”€โ”€ README.md                   # This file

๐Ÿ“ฆ Dependencies

Core Dependencies

  • mcp: Model Context Protocol server implementation

  • langgraph: Workflow orchestration

  • langchain: LLM integration framework

  • playwright: Browser automation

  • pydantic: Data validation and serialization

  • html2text: HTML to Markdown conversion

  • loguru: Structured logging

Development Dependencies

  • pytest: Testing framework

  • pytest-asyncio: Async test support

  • ruff: Fast Python linter

  • black: Code formatter


๐Ÿค Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository

  2. Create your feature branch (git checkout -b feature/amazing-feature)

  3. Commit your changes (git commit -m 'Add some amazing feature')

  4. Push to the branch (git push origin feature/amazing-feature)

  5. Open a Pull Request


๐Ÿ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.


๐Ÿ™ Acknowledgments


๐Ÿ“ž Support

For issues, questions, or contributions, please open an issue on GitHub.


Made with โค๏ธ for competitive intelligence

A
license - permissive license
-
quality - not tested
D
maintenance

Maintenance

โ€“Maintainers
โ€“Response time
โ€“Release cycle
โ€“Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Duang777/competitor-hunter'

If you have feedback or need assistance with the MCP directory API, please join our Discord server