MCP Web Research Agent
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
MCP Web Research Agent
A powerful MCP (Model Context Protocol) tool for automated web research, scraping, and intelligence gathering.
A sophisticated web research automation tool that converts your existing scraper into an MCP-compatible agent for enhanced AI workflows. Perfect for competitive intelligence, market research, and automated data collection.
π Features
π Intelligent Scraping: Recursive web crawling with configurable depth
π Search Integration: Multi-engine search with result processing
πΎ Database Storage: Persistent SQLite storage with advanced querying
π Multiple Export Formats: JSON, Markdown, and CSV exports
π€ MCP Integration: Seamless integration with AI assistants
β‘ Async Ready: Built for concurrent operations
π§ Configurable: Adjustable settings for any use case
Related MCP server: wigolo
π οΈ Installation
Prerequisites
Python 3.8+
MCP-compatible client (Claude Desktop, etc.)
Quick Install
# Clone the repository
git clone https://github.com/yourusername/mcp-web-research-agent.git
cd mcp-web-research-agent
# Install dependencies
pip install -e .MCP Client Configuration
Add to your MCP client configuration:
{
"mcpServers": {
"web-research-agent": {
"command": "python",
"args": ["/path/to/mcp-web-research-agent/server.py"]
}
}
}π Usage
Available Tools
scrape_url
Scrape a single URL for specific keywords
result = await scrape_url(
url="https://example.com",
keywords=["python", "automation", "scraping"],
extract_links=False,
max_depth=1
)search_and_scrape
Search the web and automatically scrape results
result = await search_and_scrape(
query="web scraping best practices",
keywords=["python", "beautifulsoup", "requests"],
search_engine_url="https://searx.gophernuttz.us/search/",
max_results=10
)get_scraping_results
Query the database for previous scraping results
result = await get_scraping_results(
keyword_filter="python",
limit=50
)export_results
Export results to various formats
result = await export_results(
format="markdown",
keyword_filter="python",
output_path="/path/to/output.md"
)get_scraping_stats
Get current statistics and status
result = await get_scraping_stats()ποΈ Database Schema
The agent uses SQLite with the following structure:
-- URLs table
CREATE TABLE urls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT UNIQUE NOT NULL,
title TEXT,
content TEXT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
-- Keywords table
CREATE TABLE keywords (
id INTEGER PRIMARY KEY AUTOINCREMENT,
keyword TEXT UNIQUE NOT NULL
);
-- URL-Keyword relationships
CREATE TABLE url_keywords (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url_id INTEGER,
keyword_id INTEGER,
matches INTEGER DEFAULT 1,
context TEXT,
FOREIGN KEY (url_id) REFERENCES urls (id),
FOREIGN KEY (keyword_id) REFERENCES keywords (id),
UNIQUE(url_id, keyword_id)
);π§ Configuration
Default Settings
Max Depth: 3 levels of recursive crawling
Request Delay: 1 second between requests
User Agent: Modern Chrome browser simulation
Database:
scraper_results.db(auto-created)
Customization
Modify settings in the MCPWebScraper constructor:
scraper = MCPWebScraper(
db_manager=db_manager,
max_depth=5, # Increase crawl depth
delay=0.5 # Faster requests
)π§ͺ Development
Running Tests
python test_mcp_scraper.pyExample Usage
python example_usage.pyProject Structure
mcp-web-research-agent/
βββ server.py # MCP server implementation
βββ scraper.py # Core scraping logic
βββ database.py # Database management
βββ requirements.txt # Python dependencies
βββ pyproject.toml # Package configuration
βββ test_mcp_scraper.py # Unit tests
βββ example_usage.py # Usage examples
βββ README.md # This fileπ€ Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
Fork the repository
Create your feature branch (
git checkout -b feature/amazing-feature)Commit your changes (
git commit -m 'Add some amazing feature')Push to the branch (
git push origin feature/amazing-feature)Open a Pull Request
π License
This project is licensed under the MIT License - see the LICENSE file for details.
π Acknowledgments
Built on the Model Context Protocol
Inspired by modern web scraping best practices
Thanks to the open-source community for amazing tools
Built with β€οΈ for the MCP ecosystem
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
Related MCP Connectors
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Scrape, map, crawl, search and monitor the web. Runs locally with no API key, cites every source.
Web search, scraping, crawling, research, a browser agent, and company and people data.
Web search, fetch, extract, and research for AI agents. Markdown output + AI-synthesized answers.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceEnables comprehensive web research through multi-engine search, intelligent website crawling, content analysis, trends data exploration, and automated research workflows with anti-detection measures and rich snippet detection.140MIT
- AlicenseAqualityAmaintenanceProvides local-first web intelligence over MCP with tools for search, fetch, crawl, extract, cache, find-similar, research, and autonomous agent loops, requiring no API keys.10841 npm5,558AGPL 3.0
- AlicenseNot gradedqualityCmaintenanceAI-powered multi-source research and crawling platform with MCP integration, enabling web search, crawling, social source aggregation, and structured research workflows.2MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents to search the web across configurable providers and crawl single pages or whole sites into clean, deduplicated Markdown or JSON. It runs on your own hardware, requiring no API key with SearXNG, and supports authenticated crawls plus STDIO and HTTP transport.-