Skip to main content
Glama

๐ŸŽฏ Competitor Hunter

AI-Powered Competitor Analysis Agent | Automated web scraping and structured data extraction using MCP, LangGraph, and Playwright

License: MIT Python 3.10+ Code style: black

Language | ่ฏญ่จ€: English | ไธญๆ–‡


๐Ÿ“– Introduction

Competitor Hunter is a production-ready AI agent that automates competitor analysis by scraping product pages and extracting structured information using Large Language Models. Built on the Model Context Protocol (MCP), it seamlessly integrates with Claude Desktop and other MCP-compatible clients.

Key Capabilities

  • ๐Ÿ” Intelligent Web Scraping: Automated browser-based content extraction with anti-detection features

  • ๐Ÿค– LLM-Powered Extraction: Structured data extraction using OpenAI-compatible APIs

  • ๐Ÿ“Š Structured Output: Pydantic-validated product information (pricing, features, SWOT analysis)

  • ๐Ÿ”„ LangGraph Workflow: Robust state management and error handling

  • ๐Ÿ”Œ MCP Integration: Native support for Claude Desktop and MCP clients


Related MCP server: merch-connector

๐Ÿ—๏ธ Architecture

The system follows Hexagonal Architecture with clear separation of concerns. Workflow: User Request โ†’ MCP Server โ†’ LangGraph Workflow โ†’ Browser Scraping โ†’ LLM Extraction โ†’ Structured Data Response.


โœจ Core Features

  • ๐Ÿค– AI-Powered: Intelligent extraction using LLM with automatic SWOT analysis

  • ๐Ÿ“Š Structured Output: Pydantic-validated data models (pricing, features, summary)

  • ๐Ÿ›ก๏ธ Anti-Detection: Random User-Agents, intelligent scrolling, auto-screenshots

  • ๐Ÿ”Œ MCP Native: Seamless integration with Claude Desktop and Cursor IDE

  • ๐Ÿ“ฆ CLI Tool: Professional command-line interface via competitor-hunter command

  • Async/Await: Full asynchronous programming for optimal performance


๐Ÿš€ Quick Start

Prerequisites

  • Python 3.10+ (3.11 or 3.12 recommended)

  • UV or Poetry (dependency manager)

  • Playwright browsers (installed automatically)

Installation

  1. Clone the repository:

    git clone https://github.com/your-username/competitor-hunter.git
    cd competitor-hunter
  2. Install dependencies (using UV):

    uv sync

    Or using Poetry:

    poetry install

    Or using pip:

    python -m venv venv
    source venv/bin/activate  # On Windows: venv\Scripts\activate
    pip install -e ".[dev]"
  3. Install Playwright browsers:

    playwright install chromium

Configuration

Create a .env file in the project root:

# OpenAI API Configuration
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1  # Optional: for custom endpoints
OPENAI_MODEL_NAME=gpt-4o                    # Optional: default is gpt-4o

# Browser Configuration
HEADLESS_MODE=true                          # Set to false for debugging

# Database Configuration
DB_PATH=data/competitors.db                 # SQLite database path

๐Ÿ’ก Tip: Copy .env.example to .env and fill in your values:

cp .env.example .env

๐Ÿ“ธ Screenshots & Examples

Analysis Results

Notion Pricing Analysis

Screenshot of Notion pricing page analysis

CLI Output Example

$ competitor-hunter https://www.notion.so/pricing
๐Ÿ” ๆญฃๅœจๅˆ†ๆž: https://www.notion.so/pricing

โœ… ๅˆ†ๆžๅฎŒๆˆ๏ผ

======================================================================
๐Ÿ“ฆ ไบงๅ“ๅ็งฐ: Notion
๐Ÿ”— URL: https://www.notion.so/pricing
๐Ÿ•’ ๆ›ดๆ–ฐๆ—ถ้—ด: 2024-06-13 00:00:00+00:00
======================================================================

๐Ÿ’ฐ ๅฎšไปทๆ–นๆกˆ (4 ไธช):
   โ€ข Free: 0 USD / monthly
   โ€ข Plus: 10 USD / monthly
   โ€ข Business: 20 USD / monthly
   โ€ข Enterprise: Custom USD / custom

โœจ ๆ ธๅฟƒๅŠŸ่ƒฝ (13 ไธช):
   1. AI automation
   2. Enterprise search
   3. Meeting notes
   ...

๐Ÿ’พ ็ป“ๆžœๅทฒไฟๅญ˜ๅˆฐ: reports/product_Notion.json

JSON Output Structure

The analysis results are saved as structured JSON files:

{
  "product_name": "Notion",
  "url": "https://www.notion.so/pricing",
  "pricing_tiers": [
    {
      "name": "Free",
      "price": "0",
      "currency": "USD",
      "billing_cycle": "monthly"
    },
    {
      "name": "Plus",
      "price": "10",
      "currency": "USD",
      "billing_cycle": "monthly"
    }
  ],
  "core_features": [
    "AI automation",
    "Docs",
    "Knowledge Base"
  ],
  "summary": "## ไบงๅ“ๆฆ‚่ฟฐ\nNotion ๆ˜ฏไธ€ๆฌพ้›†ๆ–‡ๆกฃ็ผ–่พ‘...",
  "last_updated": "2024-06-13T00:00:00Z"
}

๐Ÿ“š Usage

Method 1: CLI Command (Easiest)

After installation, use the competitor-hunter command:

# Analyze a single website
competitor-hunter https://www.notion.so/pricing

# Specify output file
competitor-hunter https://example.com output.json

# Batch analysis
competitor-hunter https://site1.com https://site2.com https://site3.com

Results are automatically saved to the reports/ directory with proper UTF-8 encoding.

Run the MCP server to enable integration with Claude Desktop or Cursor:

python -m src.competitor_hunter.interface.mcp_server.server

Claude Desktop Integration

Add the following configuration to your Claude Desktop claude_desktop_config.json:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "competitor-hunter": {
      "command": "python",
      "args": [
        "-m",
        "src.competitor_hunter.interface.mcp_server.server"
      ],
      "cwd": "/path/to/competitor-hunter"
    }
  }
}

Cursor IDE Integration

Create .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "competitor-hunter": {
      "command": "python",
      "args": [
        "-m",
        "src.competitor_hunter.interface.mcp_server.server"
      ],
      "cwd": "${workspaceFolder}"
    }
  }
}

After restarting, you can use the tool directly in chat:

Analyze this competitor: https://www.notion.so/pricing

Method 3: Python Library

Use the LangGraph workflow directly in your Python code:

import asyncio
from competitor_hunter.core import graph, AgentState, cleanup_resources

async def analyze(url: str):
    # Initialize state
    initial_state: AgentState = {
        "url": url,
        "scraped_content": None,
        "product": None,
        "error": None,
    }
    
    # Run workflow
    result = await graph.ainvoke(initial_state)
    
    # Check results
    if result.get("error"):
        print(f"Error: {result['error']}")
        return None
    
    product = result["product"]
    print(f"Product: {product.product_name}")
    print(f"Pricing Tiers: {len(product.pricing_tiers)}")
    print(f"Features: {product.core_features}")
    
    return product

# Use
product = await analyze("https://www.notion.so/pricing")
await cleanup_resources()

Output Structure

All analysis results are saved to the reports/ directory:

reports/
โ”œโ”€โ”€ product_Notion.json
โ”œโ”€โ”€ product_Example_Domain.json
โ””โ”€โ”€ ...

Each JSON file contains:

  • Product name and URL

  • Pricing tiers (name, price, currency, billing cycle)

  • Core features list

  • Markdown-formatted summary with SWOT analysis

  • Last updated timestamp


๐Ÿงช Development

Running Tests

# Run all tests
pytest tests/ -v

# Run specific test file
pytest tests/test_crawler.py -v

# Run with coverage
pytest tests/ --cov=src/competitor_hunter --cov-report=html

Code Quality

# Format code
black src/ tests/

# Lint code
ruff check src/ tests/

# Type checking (if using mypy)
mypy src/

Project Structure

competitor-hunter/
โ”œโ”€โ”€ src/
โ”‚   โ””โ”€โ”€ competitor_hunter/
โ”‚       โ”œโ”€โ”€ cli.py             # CLI command-line interface
โ”‚       โ”œโ”€โ”€ main.py            # Application entry point
โ”‚       โ”œโ”€โ”€ config.py          # Configuration management
โ”‚       โ”œโ”€โ”€ core/               # Domain models & LangGraph workflow
โ”‚       โ”‚   โ”œโ”€โ”€ models.py       # Pydantic models (CompetitorProduct, etc.)
โ”‚       โ”‚   โ””โ”€โ”€ graph.py        # LangGraph workflow definition
โ”‚       โ”œโ”€โ”€ infrastructure/     # External services
โ”‚       โ”‚   โ”œโ”€โ”€ browser/        # Playwright browser service
โ”‚       โ”‚   โ””โ”€โ”€ llm/            # LLM extractor service
โ”‚       โ””โ”€โ”€ interface/          # Entry points
โ”‚           โ””โ”€โ”€ mcp_server/     # MCP server implementation
โ”œโ”€โ”€ config/                     # Configuration files
โ”‚   โ””โ”€โ”€ app.yaml.example        # Configuration template
โ”œโ”€โ”€ docker/                     # Docker configuration
โ”‚   โ”œโ”€โ”€ Dockerfile              # Docker image definition
โ”‚   โ””โ”€โ”€ docker-compose.yml     # Docker Compose configuration
โ”œโ”€โ”€ examples/                   # Example scripts
โ”œโ”€โ”€ tests/                      # Test suite
โ”œโ”€โ”€ reports/                    # Analysis results (gitignored)
โ”œโ”€โ”€ data/                       # SQLite database (gitignored)
โ”œโ”€โ”€ logs/                       # Screenshots & logs (gitignored)
โ”œโ”€โ”€ pyproject.toml              # Project dependencies & CLI entry points
โ””โ”€โ”€ README.md                   # This file

๐Ÿ“ฆ Dependencies

Core Dependencies

  • mcp: Model Context Protocol server implementation

  • langgraph: Workflow orchestration

  • langchain: LLM integration framework

  • playwright: Browser automation

  • pydantic: Data validation and serialization

  • html2text: HTML to Markdown conversion

  • loguru: Structured logging

Development Dependencies

  • pytest: Testing framework

  • pytest-asyncio: Async test support

  • ruff: Fast Python linter

  • black: Code formatter


๐Ÿค Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository

  2. Create your feature branch (git checkout -b feature/amazing-feature)

  3. Commit your changes (git commit -m 'Add some amazing feature')

  4. Push to the branch (git push origin feature/amazing-feature)

  5. Open a Pull Request


๐Ÿ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.


๐Ÿ™ Acknowledgments


๐Ÿ“ž Support

For issues, questions, or contributions, please open an issue on GitHub.


Made with โค๏ธ for competitive intelligence

Related MCP Connectors

  • Social media analytics, video analysis, and competitor intel for any MCP-compatible AI agent.

  • AMZScout Skill + MCP gives AI agents live access to real Amazon marketplace data across 14 Amazon marketplaces. Analyze any ASIN, validate product ideas, research niches, compare competitors, discover profitable keywords, and build data-driven PPC strategies using trusted Amazon insights instead of AI assumptions. Works with Claude, ChatGPT, Cursor, and any other MCP-compatible AI client. To connect, you'll need an AMZScout API plan and authorize your account. Get access and view pricing here: https://learn.amzscout.net/amazon-product-api-for-ai-agents

  • The Google for AI agents โ€” company intel, competitor tracking, market research via MCP. JSON output

  • Your agent needs marketplace data โ€” what a product costs on Amazon and Google Shopping, who the sellers are, what reviewers actually complain about. **What you can ask for** โ€ข "What is this ASIN's price history, rating and seller list?" โ€ข "Who else sells this product, and at what price?" โ€ข "Pull the reviews for this product and group the complaints." โ€ข "What comes up on Google Shopping for this query in the UK?" โ€ข "Compare these products across both marketplaces." **How to use it** Point any MCP client at https://mcp.aisa.one/seo-merchant/mcp and sign in with OAuth โ€” there is no key to create or paste. 22 tools: Amazon products, ASIN detail and sellers; Google Shopping products, product info, sellers and reviews; live and queued forms, with raw HTML where you need it. **It is also a door to the rest** The same login reaches 26 sources and 580+ operations. Price the product here, then ask the same agent what the brand's site traffic or ad spend looks like โ€” without adding a second server. **What it costs** Finding and inspecting an operation is free. Running one is billed per call at API prices, with no seat and no monthly minimum, and every call takes max_price_usd so an agent cannot overspend by accident. **Where else it reaches** https://mcp.aisa.one/seo/mcp for all of it at once โ€” rankings, keywords, backlinks, site health and AI-answer visibility across DataForSEO, Semrush and Ahrefs.

Related MCP Servers