MCP Web Research Agent

A powerful MCP (Model Context Protocol) tool for automated web research, scraping, and intelligence gathering.

A sophisticated web research automation tool that converts your existing scraper into an MCP-compatible agent for enhanced AI workflows. Perfect for competitive intelligence, market research, and automated data collection.

🚀 Features

• 🔍 Intelligent Scraping: Recursive web crawling with configurable depth

• 🔎 Search Integration: Multi-engine search with result processing

• 💾 Database Storage: Persistent SQLite storage with advanced querying

• 📊 Multiple Export Formats: JSON, Markdown, and CSV exports

• 🤖 MCP Integration: Seamless integration with AI assistants

• ⚡ Async Ready: Built for concurrent operations

• 🔧 Configurable: Adjustable settings for any use case

Related MCP server: RivalSearchMCP

🛠️ Installation

Prerequisites

• Python 3.8+

• MCP-compatible client (Claude Desktop, etc.)

Quick Install

# Clone the repository
git clone https://github.com/yourusername/mcp-web-research-agent.git
cd mcp-web-research-agent

# Install dependencies
pip install -e .

MCP Client Configuration

Add to your MCP client configuration:

{
  "mcpServers": {
    "web-research-agent": {
      "command": "python",
      "args": ["/path/to/mcp-web-research-agent/server.py"]
    }
  }
}

📖 Usage

Available Tools

scrape_url

Scrape a single URL for specific keywords

result = await scrape_url(
    url="https://example.com",
    keywords=["python", "automation", "scraping"],
    extract_links=False,
    max_depth=1
)

search_and_scrape

Search the web and automatically scrape results

result = await search_and_scrape(
    query="web scraping best practices",
    keywords=["python", "beautifulsoup", "requests"],
    search_engine_url="https://searx.gophernuttz.us/search/",
    max_results=10
)

get_scraping_results

Query the database for previous scraping results

result = await get_scraping_results(
    keyword_filter="python",
    limit=50
)

export_results

Export results to various formats

result = await export_results(
    format="markdown",
    keyword_filter="python",
    output_path="/path/to/output.md"
)

get_scraping_stats

Get current statistics and status

result = await get_scraping_stats()

🗃️ Database Schema

The agent uses SQLite with the following structure:

-- URLs table
CREATE TABLE urls (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url TEXT UNIQUE NOT NULL,
    title TEXT,
    content TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Keywords table  
CREATE TABLE keywords (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    keyword TEXT UNIQUE NOT NULL
);

-- URL-Keyword relationships
CREATE TABLE url_keywords (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    url_id INTEGER,
    keyword_id INTEGER,
    matches INTEGER DEFAULT 1,
    context TEXT,
    FOREIGN KEY (url_id) REFERENCES urls (id),
    FOREIGN KEY (keyword_id) REFERENCES keywords (id),
    UNIQUE(url_id, keyword_id)
);

🔧 Configuration

Default Settings

• Max Depth: 3 levels of recursive crawling

• Request Delay: 1 second between requests

• User Agent: Modern Chrome browser simulation

• Database: scraper_results.db (auto-created)

Customization

Modify settings in the MCPWebScraper constructor:

scraper = MCPWebScraper(
    db_manager=db_manager,
    max_depth=5,      # Increase crawl depth
    delay=0.5         # Faster requests
)

🧪 Development

Running Tests

python test_mcp_scraper.py

Example Usage

python example_usage.py

Project Structure

mcp-web-research-agent/
├── server.py              # MCP server implementation
├── scraper.py             # Core scraping logic
├── database.py            # Database management
├── requirements.txt       # Python dependencies
├── pyproject.toml         # Package configuration
├── test_mcp_scraper.py    # Unit tests
├── example_usage.py       # Usage examples
└── README.md              # This file

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository

2. Create your feature branch (git checkout -b feature/amazing-feature)

3. Commit your changes (git commit -m 'Add some amazing feature')

4. Push to the branch (git push origin feature/amazing-feature)

5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built on the Model Context Protocol

• Inspired by modern web scraping best practices

• Thanks to the open-source community for amazing tools

Built with ❤️ for the MCP ecosystem