🚀 MCP Web Scraper Server A production-ready MCP (Model Context Protocol) server for advanced web scraping and search, easily deployable on Railway.

✨ Features 🔍 Advanced Web Search - Search anything on the web using DuckDuckGo 🤖 Smart Search - Intelligent search with quick/standard/comprehensive modes 📰 News Search - Dedicated news article search with dates and sources 🎯 Search & Scrape - Automatically search and extract full content from results 📄 Article Extraction - Clean article content extraction (removes ads/navigation) 🔗 Link Extraction - Extract all links with regex filtering 📊 Table Extraction - Extract table data from webpages 📝 Metadata Extraction - Get page metadata and Open Graph tags 🚀 Easy Railway Deployment 💪 Production-ready 🛠️ Tools Available 🔍 Search Tools web_search - Search the web for anything (just give a query!) smart_search - Intelligent search with modes (quick/standard/comprehensive) search_and_scrape - Search + automatically scrape full content news_search - Search specifically for news articles 📄 Scraping Tools scrape_html - Scrape HTML content with optional CSS selectors extract_links - Extract all links with optional filtering extract_metadata - Get page metadata and Open Graph tags scrape_table - Extract table data from webpages extract_article - Clean article extraction (removes ads/navigation) 🚀 Quick Deploy to Railway Step 1: Create GitHub Repository bash

Clone or download this repository

git clone https://github.com/yourusername/mcp-web-scraper.git cd mcp-web-scraper

Or create new repository

mkdir mcp-web-scraper cd mcp-web-scraper

Copy all files here

Initialize git

git init git add . git commit -m "Initial commit: MCP Web Scraper Server" git branch -M main git remote add origin https://github.com/YOUR_USERNAME/mcp-web-scraper.git git push -u origin main Step 2: Deploy to Railway Go to railway.app Click "New Project" Select "Deploy from GitHub repo" Choose your repository Railway automatically detects Dockerfile and deploys! 🎉 Step 3: Get Your URL Click on your deployment in Railway Go to "Settings" → "Domains" Click "Generate Domain" Copy your URL (e.g., https://mcp-web-scraper-production.up.railway.app) Step 4: Test Your Server bash

Health check

curl https://your-app.up.railway.app/health

List available tools

curl https://your-app.up.railway.app/tools

Test web search

curl -X POST https://your-app.up.railway.app/call-tool
-H "Content-Type: application/json"
-d '{"name": "web_search", "arguments": {"query": "latest AI news"}}' 💻 Local Development bash

Clone repository

git clone https://github.com/yourusername/mcp-web-scraper.git cd mcp-web-scraper

Create virtual environment

python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

Install dependencies

pip install -r requirements.txt

Run server

uvicorn src.server:app --reload --port 8000 Visit http://localhost:8000 to see the server running!

🔌 Connect to Claude Desktop Add to your Claude Desktop config (claude_desktop_config.json):

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%\Claude\claude_desktop_config.json

json { "mcpServers": { "web-scraper": { "command": "npx", "args": [ "-y", "mcp-remote", "https://your-app.up.railway.app/sse" ] } } } Then restart Claude Desktop!

📋 Example Usage Search the Web bash curl -X POST http://localhost:8000/call-tool
-H "Content-Type: application/json"
-d '{ "name": "web_search", "arguments": { "query": "best pizza recipe", "max_results": 5 } }' Smart Search (Comprehensive) bash curl -X POST http://localhost:8000/call-tool
-H "Content-Type: application/json"
-d '{ "name": "smart_search", "arguments": { "query": "climate change solutions", "mode": "comprehensive" } }' Search and Scrape bash curl -X POST http://localhost:8000/call-tool
-H "Content-Type: application/json"
-d '{ "name": "search_and_scrape", "arguments": { "query": "machine learning tutorials", "num_results": 3 } }' News Search bash curl -X POST http://localhost:8000/call-tool
-H "Content-Type: application/json"
-d '{ "name": "news_search", "arguments": { "query": "technology", "max_results": 10 } }' Extract Article bash curl -X POST http://localhost:8000/call-tool
-H "Content-Type: application/json"
-d '{ "name": "extract_article", "arguments": { "url": "https://example.com/article" } }' 🎯 Use Cases in Claude Once connected, you can ask Claude:

"Search for the best Italian restaurants in Rome" "Find me recent articles about quantum computing" "What's the latest news on AI developments?" "Research blockchain technology and give me detailed info" "Scrape the table from this webpage: [URL]" "Extract all links from example.com" 📁 Project Structure mcp-web-scraper/ ├── src/ │ ├── init.py # Package initialization │ ├── server.py # FastAPI server and MCP integration │ └── tools.py # Web scraping and search tools ├── requirements.txt # Python dependencies ├── Dockerfile # Docker configuration ├── railway.json # Railway deployment config ├── .gitignore # Git ignore file └── README.md # This file 🔧 Configuration Environment Variables (Optional) You can set these in Railway dashboard under "Variables":

LOG_LEVEL - Logging level (default: INFO) PORT - Server port (default: 8000) HOST - Server host (default: 0.0.0.0) 📊 Monitoring Railway provides built-in monitoring:

Metrics - CPU, Memory, Network usage Logs - Real-time application logs Deployments - Deployment history and rollbacks Access these in your Railway dashboard.

💰 Cost Railway Free Tier:

$5 free credit per month 500 hours of usage Perfect for personal use and testing For production use, consider upgrading to Railway Pro.

🔒 Security Notes ⚠️ This server is deployed without authentication for easy use. For production:

Consider adding API key authentication Implement rate limiting Restrict allowed domains Use environment variables for sensitive data 🐛 Troubleshooting Server not starting? Check Railway logs in dashboard Verify all files are committed to Git Ensure Dockerfile is in root directory Tools not working? Check tool names match exactly Verify JSON format in requests Check server logs for errors Can't connect to Claude? Verify Railway URL is correct Ensure /sse endpoint is accessible Restart Claude Desktop after config change 🤝 Contributing Contributions are welcome! Feel free to:

Report bugs Suggest new features Submit pull requests 📄 License MIT License - feel free to use and modify!

🙏 Acknowledgments Built with:

FastAPI - Web framework MCP - Model Context Protocol DuckDuckGo Search - Web search Trafilatura - Content extraction BeautifulSoup - HTML parsing Railway - Deployment platform 📞 Support GitHub Issues: Report a bug Railway Docs: docs.railway.app MCP Docs: modelcontextprotocol.io Made with ❤️ for the MCP community