Which integrations are available for this server?

Enables authenticated web crawling and browser automation using Firefox session tokens and profiles. Provides web search results via Google, returning titles, URLs, and snippets.

How do I use WebSearchAndCrawl?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@WebSearchAndCrawl Crawl the documentation site and extract all API endpoints." That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

de en es ja ko ru zh

WebSearchAndCrawl

by bpweatherill

Overview Schema Source Related Servers Discussions Admin

Python

Local

WebSearchAndCrawl

An MCP Server for Authenticated Web Crawling, Searching, and Document Processing

---.

🚀 Purpose

WebSearchAndCrawl is an MCP (Model Context Protocol) server designed to:

Crawl websites (including authenticated ones) using Firefox session tokens or browser automation.
Search crawled content for regex matches and store results in a structured index.
Download and parse documents (PDF, DOCX, XLSX, etc.) from crawled sites.
Stream results in real-time via HTTP for integration with MCP clients.
Respect robots.txt and enforce rate limits (5 requests/sec, 5 threads max).

This tool is ideal for:

Researchers who need to scrape authenticated or dynamic websites.
Developers building AI agents that require web data.
Automation of repetitive web tasks (e.g., monitoring, data extraction).

Related MCP server: Scout MCP Server

🔧 Features

Feature	Description
Authenticated Crawling	Uses Firefox session tokens to access logged-in pages.
Browser Automation	Falls back to Playwright for dynamic content or login forms.
Domain Whitelisting	Only crawls URLs matching a comma-delimited list of domains.
Depth-limited Crawling	Configurable crawl depth (1-9 layers).
Regex Search	Search crawled content or index for regex patterns.
Document Parsing	Extracts text from PDF, DOCX, XLSX, and TXT files.
Real-time Streaming	Results are streamed as JSONL (chunked by page).
Indexing	Stores results in JSON files per domain for later search.
Rate Limiting	Enforces 5 requests/sec and 5 threads max.
Resume Support	Can resume interrupted crawls from checkpoints.
Session Validation	Validates token scopes to prevent misuse.

📦 Installation

Prerequisites

Python 3.9+ (recommended: 3.11+).
Firefox (required for browser automation).
System Libraries (for document parsing):
- PDF: poppler-utils (Linux) or pdfminer.six (cross-platform).
- DOCX/XLSX: python-docx, openpyxl.

Steps

1. Clone the Repository

git clone https://github.com/bpweatherill/WebSearchAndCrawl.git
cd WebSearchAndCrawl

2. Set Up a Virtual Environment

python -m venv venv
source venv/bin/activate  # Linux/Mac
# OR
venv\Scripts\activate   # Windows

3. Install Dependencies

pip install -r requirements.txt

4. Install Playwright Browsers

playwright install firefox

5. (Optional) Configure Environment Variables

Create a .env file in the project root:

# Server
MCP_PORT=8808
MCP_HOST=0.0.0.0

# Crawler
MAX_DEPTH=9
MAX_THREADS=5
RATE_LIMIT=5
REQUEST_TIMEOUT=10
MAX_MEMORY_MB=1024

# Firefox
FIREFOX_PROFILE=my_profile  # Optional: Specific Firefox profile
DEFAULT_SEARCH_ENGINE=google

# Directories
INDEX_DIR=./index
DOWNLOADS_DIR=./downloads
CHECKPOINTS_DIR=./checkpoints

🏃 Usage

1. Start the MCP Server

python -m server.main

The server will start on http://localhost:8808 (or the port specified in .env).

2. MCP Tools (HTTP Endpoints)

All tools return JSON responses and support streaming for real-time results.

Endpoint	Method	Description	Request Body
`/crawl_website`	POST	Crawl a website and stream results.	CrawlRequest
`/search_index`	POST	Search the local index for regex matches.	SearchIndexRequest
`/download_documents`	POST	Download documents matching a regex.	DownloadRequest
`/get_search_results`	POST	Use Firefox's search engine to fetch results.	WebSearchRequest
`/list_indexed_domains`	GET	List all domains with indexed content.	-
`/health`	GET	Health check.	-

Request/Response Schemas

CrawlRequest

{
  "url": "https://www.nasa.gov",
  "whitelist_domains": "nasa.gov",
  "max_depth": 3,
  "use_token": false,
  "firefox_profile": "my_profile"
}

url: Starting URL for the crawl.
whitelist_domains: Comma-delimited list of allowed domains (e.g., "nasa.gov,spacex.com").
max_depth: Maximum crawl depth (1-9).
use_token: Use Firefox session token if available.
firefox_profile: Firefox profile name (optional).

Streamed Response (JSONL):

{
  "excerpt": "NASA's Perseverance Rover lands on Mars...",
  "full_text": "Full article text here...",
  "url": "https://www.nasa.gov/mars2020",
  "timestamp": "2024-05-20T12:00:00Z",
  "domain": "nasa.gov"
}

---.

SearchIndexRequest

{
  "domain": "nasa.gov",
  "regex": ".*Mars.*",
  "max_results": 10
}

domain: Domain to search (e.g., "nasa.gov").
regex: Regex pattern to match.
max_results: Maximum number of results to return.

Response:

[
  {
    "url": "https://www.nasa.gov/mars2020",
    "excerpt": "NASA's Perseverance Rover lands on Mars...",
    "timestamp": "2024-05-20T12:00:00Z"
  }
]

---.

DownloadRequest

{
  "domain": "nasa.gov",
  "regex": ".*\\.pdf$",
  "output_dir": "./downloads/nasa.gov"
}

domain: Domain to download from.
regex: Regex pattern for files to download (e.g., "*.pdf").
output_dir: Custom output directory (optional).

Streamed Response (JSONL):

{
  "filename": "./downloads/nasa.gov/mars_rover.pdf",
  "url": "https://www.nasa.gov/pdf/mars_rover.pdf",
  "parsed_text": "Extracted text from PDF..."
}

---.

WebSearchRequest

{
  "query": "NASA Mars missions",
  "search_engine": "google",
  "use_token": false,
  "firefox_profile": "my_profile"
}

query: Search query.
search_engine: Search engine (default: Firefox default).
use_token: Use Firefox session token if available.
firefox_profile: Firefox profile name (optional).

Streamed Response (JSONL):

{
  "title": "Mars 2020 Mission - NASA",
  "url": "https://www.nasa.gov/mars2020",
  "snippet": "Learn about the Perseverance Rover..."
}

🔍 Examples

1. Crawl NASA.gov and Index Results

curl -X POST http://localhost:8808/crawl_website \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.nasa.gov",
    "whitelist_domains": "nasa.gov",
    "max_depth": 2,
    "use_token": false
  }'

2. Search Indexed Content for "Mars"

curl -X POST http://localhost:8808/search_index \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "nasa.gov",
    "regex": ".*Mars.*",
    "max_results": 5
  }'

3. Download PDFs from NASA.gov

curl -X POST http://localhost:8808/download_documents \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "nasa.gov",
    "regex": ".*\\.pdf$"
  }'

4. Use Firefox to Search Google

curl -X POST http://localhost:8808/get_search_results \
  -H "Content-Type: application/json" \
  -d '{
    "query": "NASA Mars missions",
    "search_engine": "google"
  }'

📁 Project Structure

WebSearchAndCrawl/
│
├── server/                          # Core server logic
│   ├── __init__.py
│   ├── main.py                       # FastAPI app + MCP tools
│   ├── config.py                     # Configuration settings
│   ├── schemas.py                    # Pydantic request/response models
│   │
│   ├── firefox/                      # Firefox browser automation
│   │   ├── __init__.py
│   │   ├── controller.py              # Playwright Firefox management
│   │   └── token_manager.py           # Session token handling
│   │
│   ├── crawler/                     # Web crawling logic
│   │   ├── __init__.py
│   │   ├── crawler.py                # Main crawling logic
│   │   └── rate_limiter.py            # Thread/rate limiting
│   │
│   ├── indexer/                     # Indexing and search
│   │   ├── __init__.py
│   │   ├── indexer.py                 # JSON index management
│   │   └── search_engine.py           # Regex search
│   │
│   ├── downloader/                  # Document downloading and parsing
│   │   ├── __init__.py
│   │   ├── downloader.py              # Download logic
│   │   └── parsers/                  # File type parsers
│   │       ├── __init__.py
│   │       ├── pdf_parser.py
│   │       ├── docx_parser.py
│   │       └── xlsx_parser.py
│   │
│   └── streamer.py                   # Chunked JSON streaming
│
├── tests/                           # Unit and integration tests
│   ├── __init__.py
│   ├── test_firefox.py
│   └── test_crawler.py
│
├── index/                           # Index files (auto-generated)
│   ├── nasa.gov.json
│   └── ...
│
├── downloads/                       # Downloaded documents (auto-generated)
│   ├── nasa.gov/
│   │   ├── document1.pdf
│   │   └── ...
│   └── ...
│
├── checkpoints/                     # Crawl checkpoints (auto-generated)
│   └── ...
│
├── requirements.txt                 # Python dependencies
├── .env.example                     # Example environment variables
└── README.md                        # This file

⚙️ Configuration

Environment Variables

Variable	Default	Description
`MCP_PORT`	`8808`	HTTP server port.
`MCP_HOST`	`0.0.0.0`	HTTP server host.
`MAX_DEPTH`	`9`	Maximum crawl depth (1-9).
`MAX_THREADS`	`5`	Maximum concurrent threads.
`RATE_LIMIT`	`5`	Maximum requests per second.
`REQUEST_TIMEOUT`	`10`	Timeout for requests (seconds).
`MAX_MEMORY_MB`	`1024`	Maximum memory usage (MB).
`FIREFOX_PROFILE`	`None`	Firefox profile name (optional).
`DEFAULT_SEARCH_ENGINE`	`google`	Default search engine.
`INDEX_DIR`	`./index`	Directory for index files.
`DOWNLOADS_DIR`	`./downloads`	Directory for downloaded files.
`CHECKPOINTS_DIR`	`./checkpoints`	Directory for crawl checkpoints.

🛡️ Security Considerations

Session Tokens:
- Tokens are only stored in memory (not persisted to disk).
- Token scopes are validated to prevent misuse (e.g., a token for nasa.gov cannot be used for evil.com).
Input Sanitization:
- All inputs (URLs, regex, etc.) are sanitized to prevent injection attacks.
Rate Limiting:
- Enforces 5 requests/sec and 5 threads max to avoid overwhelming servers.
robots.txt Compliance:
- The crawler respects robots.txt and skips disallowed URLs.
Whitelisting:
- Only URLs matching the whitelisted domains are crawled.

🚀 Enhancements (Roadmap)

Enhancement	Description	Priority
Persistent Tokens	Store tokens in an encrypted file for persistence across restarts.	Medium
Full `robots.txt` Parsing	Properly parse `robots.txt` rules instead of simple checks.	Medium
Advanced Pagination Handling	Detect and follow pagination links (e.g., "Next" buttons).	High
Lazy-Loading Support	Detect and trigger lazy-loaded content (e.g., infinite scroll).	High
Checkpointing	Save crawl state to resume interrupted crawls.	High
Full-Text Search	Support full-text search in addition to regex.	Low
Database Backend	Replace JSON files with SQLite/PostgreSQL for scalability.	Low
Distributed Crawling	Support horizontal scaling with multiple workers.	Low
Docker Support	Add a `Dockerfile` for containerized deployment.	Medium
Authentication Helpers	Built-in support for common auth methods (OAuth, SAML).	Medium
Proxy Support	Add proxy support for crawling behind firewalls.	Low
Custom Headers	Allow users to specify custom headers for requests.	Medium
Webhook Notifications	Notify a webhook URL when new results are found.	Low

🤝 Contributing

Fork the repository.
Create a feature branch (git checkout -b feature/your-feature).
Commit your changes (git commit -m "Add your feature").
Push to the branch (git push origin feature/your-feature).
Open a Pull Request.

📜 License

This project is licensed under the MIT License. See LICENSE for details.

📞 Support

Issues: Report bugs or request features in the GitHub Issues tab.
Discussions: Join the GitHub Discussions for Q&A.

🏆 Acknowledgments

Playwright: For browser automation.
FastAPI: For the HTTP server.
pdfminer.six: For PDF parsing.
python-docx/openpyxl: For Office file parsing.

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityMaintained

ResponsivenessSyncing

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

Aginx Browser A
net.aginx
Stealth web browser for agents: search, fetch, click, download and type in persistent MCP sessions.
Browser Automation Agent Orchestration
37128
luminati-io-brightdata-mcp
ai.smithery
One MCP for the Web. Easily search, crawl, navigate, and extract websites without getting blocked.…
Browser Automation Web Scraping Search
2
MESSORA A
dev.messora
Scrape, crawl and search the web for AI agents via MCP.
Web Scraping Browser Automation Search
51
ReplyNodes B
com.replynodes
Web MCP: scrape/crawl sites, web search, brand assets, app stores, YouTube, Reddit, Hacker News.
Browser Automation Web Scraping Search
5113

Related MCP Servers

web-scraper-server
Browser Automation Web Scraping
ImYourBoyRoy
A
license
C
quality
C
maintenance
Provides browser automation and web scraping as MCP tools, enabling autonomous URL ingestion, crawling, extraction, and anti-bot handling with interactive browser control.
Updated a month ago (2026-08-03 15:17 UTC)
62
5
MIT
Scout MCP Serverofficial
Browser Automation Web Scraping Search
Scout-AI-Labs
A
license
Not graded
quality
D
maintenance
Enables web search, scraping, extraction, and crawling through an MCP interface, allowing coding agents to access real-time web data.
Updated 3 months ago (2026-06-21 09:51 UTC)
1
MIT
OpenCode Web Audit MCP
Web Scraping Browser Automation Search
humbertolvarona
A
license
Not graded
quality
C
maintenance
A read-only Python MCP server for authorized website research, structured data extraction, downloadable-document analysis, and content auditing inside OpenCode. It crawls authorized public domains with safety constraints including robots.txt respect, SSRF defenses, bounded concurrency, and content-type allowlists.
Updated a month ago (2026-08-03 22:22 UTC)
MIT
webharvest
Browser Automation Web Scraping Search
dan-rozhkov
A
license
Not graded
quality
B
maintenance
Enables web search and scraping through MCP, running locally with courtesy rate limiting and caching.
Updated 2 days ago (2026-09-15 23:04 UTC)
1 npm
ISC

WebSearchAndCrawl

🚀 Purpose

🔧 Features

📦 Installation

Prerequisites

Steps

1. Clone the Repository

2. Set Up a Virtual Environment

3. Install Dependencies

4. Install Playwright Browsers

5. (Optional) Configure Environment Variables

🏃 Usage

1. Start the MCP Server

2. MCP Tools (HTTP Endpoints)

Request/Response Schemas

CrawlRequest

SearchIndexRequest

DownloadRequest

WebSearchRequest

🔍 Examples

1. Crawl NASA.gov and Index Results

2. Search Indexed Content for "Mars"

3. Download PDFs from NASA.gov

4. Use Firefox to Search Google

📁 Project Structure

⚙️ Configuration

Environment Variables

🛡️ Security Considerations

🚀 Enhancements (Roadmap)

🤝 Contributing

📜 License

📞 Support

🏆 Acknowledgments

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

web-scraper-server

Scout MCP Serverofficial

OpenCode Web Audit MCP

webharvest