# crawl-mcp-server
A comprehensive MCP (Model Context Protocol) server providing **11 powerful tools** for web crawling and search. Transform web content into clean, LLM-optimized Markdown or search the web with SearXNG integration.
[](https://github.com/Git-Fg/searchcrawl-mcp-server/actions)
[](https://codecov.io/gh/Git-Fg/searchcrawl-mcp-server)
## ✨ Features
- 🔍 **SearXNG Web Search** - Search the web with automatic browser management
- 📄 **4 Crawling Tools** - Extract and convert web content to Markdown
- 🚀 **Auto-Browser-Launch** - Search tools automatically manage browser lifecycle
- 📦 **11 Total Tools** - Complete toolkit for web interaction
- 💾 **Built-in Caching** - SHA-256 based caching with graceful fallbacks
- ⚡ **Concurrent Processing** - Handle multiple URLs simultaneously (up to 50)
- 🎯 **LLM-Optimized Output** - Clean Markdown perfect for AI consumption
- 🛡️ **Robust Error Handling** - Graceful failure with detailed error messages
- 🧪 **Comprehensive Testing** - Full CI/CD with performance benchmarks
## 📦 Installation
### Method 1: npm (Recommended)
```bash
npm install crawl-mcp-server
```
### Method 2: Direct from Git
```bash
# Install latest from GitHub
npm install git+https://github.com/Git-Fg/searchcrawl-mcp-server.git
# Or specific branch
npm install git+https://github.com/Git-Fg/searchcrawl-mcp-server.git#main
# Or from a fork
npm install git+https://github.com/YOUR_FORK/searchcrawl-mcp-server.git
```
### Method 3: Clone and Build
```bash
git clone https://github.com/Git-Fg/searchcrawl-mcp-server.git
cd crawl-mcp-server
npm install
npm run build
```
### Method 4: npx (No Installation)
```bash
# Run directly without installing
npx git+https://github.com/Git-Fg/searchcrawl-mcp-server.git
```
## 🔧 Setup for Claude Code
### Option 1: MCP Desktop (Recommended)
Add to your Claude Desktop configuration file:
** macOS/Linux: `~/.config/claude/claude_desktop_config.json`**
```json
{
"mcpServers": {
"crawl-server": {
"command": "npx",
"args": [
"git+https://github.com/Git-Fg/searchcrawl-mcp-server.git"
],
"env": {
"NODE_ENV": "production"
}
}
}
}
```
** Windows: `%APPDATA%\Claude\claude_desktop_config.json`**
```json
{
"mcpServers": {
"crawl-server": {
"command": "npx",
"args": [
"git+https://github.com/Git-Fg/searchcrawl-mcp-server.git"
],
"env": {
"NODE_ENV": "production"
}
}
}
}
```
### Option 2: Local Installation
If you've installed locally:
```json
{
"mcpServers": {
"crawl-server": {
"command": "node",
"args": [
"/path/to/crawl-mcp-server/dist/index.js"
],
"env": {}
}
}
}
```
### Option 3: Custom Path
For a specific installation:
```json
{
"mcpServers": {
"crawl-server": {
"command": "node",
"args": [
"/usr/local/lib/node_modules/crawl-mcp-server/dist/index.js"
],
"env": {}
}
}
}
```
After configuration, restart Claude Desktop.
## 🔧 Setup for Other MCP Clients
### Claude CLI
```bash
# Using npx
claude mcp add crawl-server npx git+https://github.com/Git-Fg/searchcrawl-mcp-server.git
# Using local installation
claude mcp add crawl-server node /path/to/crawl-mcp-server/dist/index.js
```
### Zed Editor
Add to `~/.config/zed/settings.json`:
```json
{
"assistant": {
"mcp": {
"servers": {
"crawl-server": {
"command": "node",
"args": ["/path/to/crawl-mcp-server/dist/index.js"]
}
}
}
}
}
```
### VSCode with Copilot Chat
```json
{
"mcpServers": {
"crawl-server": {
"command": "node",
"args": ["/path/to/crawl-mcp-server/dist/index.js"]
}
}
}
```
## 🚀 Quick Start
### Using MCP Inspector (Testing)
```bash
# Install MCP Inspector globally
npm install -g @modelcontextprotocol/inspector
# Run the server
node dist/index.js
# In another terminal, test tools
npx @modelcontextprotocol/inspector --cli node dist/index.js --method tools/list
```
### Development Mode
```bash
# Watch mode (auto-rebuild on changes)
npm run dev
# Build TypeScript
npm run build
# Run tests
npm run test:run
```
## 📚 Available Tools
### Search Tools (7 tools)
#### 1. search_searx
Search the web using SearXNG with automatic browser management.
```typescript
// Example call
{
"query": "TypeScript MCP server",
"maxResults": 10,
"category": "general",
"timeRange": "week",
"language": "en"
}
```
**Parameters:**
- `query` (string, required): Search query
- `maxResults` (number, default: 20): Results to return (1-50)
- `category` (enum, default: general): one of general, images, videos, news, map, music, it, science
- `timeRange` (enum, optional): one of day, week, month, year
- `language` (string, default: en): Language code
**Returns:** JSON with search results array, URLs, and metadata
---
#### 2. launch_chrome_cdp
Launch system Chrome with remote debugging for advanced SearX usage.
```typescript
{
"headless": true,
"port": 9222,
"userDataDir": "/path/to/profile"
}
```
**Parameters:**
- `headless` (boolean, default: true): Run Chrome headless
- `port` (number, default: 9222): Remote debugging port
- `userDataDir` (string, optional): Custom Chrome profile
---
#### 3. connect_cdp
Connect to remote CDP browser (Browserbase, etc.).
```typescript
{
"cdpWsUrl": "http://localhost:9222"
}
```
**Parameters:**
- `cdpWsUrl` (string, required): CDP WebSocket URL or HTTP endpoint
---
#### 4. launch_local
Launch bundled Chromium for SearX search.
```typescript
{
"headless": true,
"userAgent": "custom user agent string"
}
```
**Parameters:**
- `headless` (boolean, default: true): Run headless
- `userAgent` (string, optional): Custom user agent
---
#### 5. chrome_status
Check Chrome CDP status and health.
```typescript
{}
```
**Returns:** Running status, health, endpoint URL, and PID
---
#### 6. close
Close browser session (keeps Chrome CDP running).
```typescript
{}
```
---
#### 7. shutdown_chrome_cdp
Shutdown Chrome CDP and cleanup resources.
```typescript
{}
```
### Crawling Tools (4 tools)
#### 1. crawl_read ⭐ (Simple & Fast)
Quick single-page extraction to Markdown.
```typescript
{
"url": "https://example.com/article",
"options": {
"timeout": 30000
}
}
```
**Best for:**
- ✅ News articles
- ✅ Blog posts
- ✅ Documentation pages
- ✅ Simple content extraction
**Returns:** Clean Markdown content
---
#### 2. crawl_read_batch ⭐ (Multiple URLs)
Process 1-50 URLs concurrently.
```typescript
{
"urls": [
"https://example.com/article1",
"https://example.com/article2",
"https://example.com/article3"
],
"options": {
"maxConcurrency": 5,
"timeout": 30000,
"maxResults": 10
}
}
```
**Best for:**
- ✅ Processing multiple articles
- ✅ Building content aggregates
- ✅ Bulk content extraction
**Returns:** Array of Markdown results with summary statistics
---
#### 3. crawl_fetch_markdown
Controlled single-page extraction with full option control.
```typescript
{
"url": "https://example.com/article",
"options": {
"timeout": 30000
}
}
```
**Best for:**
- ✅ Advanced crawling options
- ✅ Custom timeout control
- ✅ Detailed extraction
---
#### 4. crawl_fetch
Multi-page crawling with intelligent link extraction.
```typescript
{
"url": "https://example.com",
"options": {
"pages": 5,
"maxConcurrency": 3,
"sameOriginOnly": true,
"timeout": 30000,
"maxResults": 20
}
}
```
**Best for:**
- ✅ Crawling entire sites
- ✅ Link-based discovery
- ✅ Multi-page scraping
**Features:**
- Extracts links from starting page
- Crawls discovered pages
- Concurrent processing
- Same-origin filtering (configurable)
## 💡 Usage Examples
### Example 1: Search + Crawl Workflow
```typescript
// Step 1: Search for topics
{
"tool": "search_searx",
"arguments": {
"query": "TypeScript best practices 2024",
"maxResults": 5
}
}
// Step 2: Extract URLs from results
// (Parse the search results to get URLs)
// Step 3: Crawl selected articles
{
"tool": "crawl_read_batch",
"arguments": {
"urls": [
"https://example.com/article1",
"https://example.com/article2",
"https://example.com/article3"
]
}
}
```
### Example 2: Batch Content Extraction
```typescript
{
"tool": "crawl_read_batch",
"arguments": {
"urls": [
"https://news.site/article1",
"https://news.site/article2",
"https://news.site/article3"
],
"options": {
"maxConcurrency": 10,
"timeout": 30000,
"maxResults": 3
}
}
}
```
### Example 3: Site Crawling
```typescript
{
"tool": "crawl_fetch",
"arguments": {
"url": "https://docs.example.com",
"options": {
"pages": 10,
"maxConcurrency": 5,
"sameOriginOnly": true,
"timeout": 30000,
"maxResults": 10
}
}
}
```
## 🎯 Tool Selection Guide
| Use Case | Recommended Tool | Complexity |
|----------|----------------|------------|
| Single article | `crawl_read` | Simple |
| Multiple articles | `crawl_read_batch` | Simple |
| Advanced options | `crawl_fetch_markdown` | Medium |
| Site crawling | `crawl_fetch` | Complex |
| Web search | `search_searx` | Simple |
| Research workflow | `search_searx` → `crawl_read` | Medium |
## 🏗️ Architecture
### Core Components
```
┌─────────────────────────────────────────┐
│ crawl-mcp-server │
├─────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────┐ │
│ │ MCP Server Core │ │
│ │ - 11 registered tools │ │
│ │ - STDIO/HTTP transport │ │
│ └──────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────────┐ │
│ │ @just-every/crawl │ │
│ │ - HTML → Markdown │ │
│ │ - Mozilla Readability │ │
│ │ - Concurrent crawling │ │
│ └──────────────────────────────┘ │
│ │ │
│ ┌──────────────────────────────┐ │
│ │ Playwright (Browser) │ │
│ │ - SearXNG integration │ │
│ │ - Auto browser management │ │
│ │ - Anti-detection │ │
│ └──────────────────────────────┘ │
│ │
└─────────────────────────────────────────┘
```
### Technology Stack
- **Runtime**: Node.js 18+
- **Language**: TypeScript 5.7
- **Framework**: MCP SDK (@modelcontextprotocol/sdk)
- **Crawling**: @just-every/crawl
- **Browser**: Playwright Core
- **Validation**: Zod
- **Transport**: STDIO (local) + HTTP (remote)
### Data Flow
```
Client Request
↓
MCP Protocol
↓
Tool Handler
↓
┌─────────────────────┐
│ Crawl/Search │
│ @just-every/crawl │ → HTML content
│ or SearXNG │ → Search results
└─────────────────────┘
↓
HTML → Markdown
↓
Result Formatting
↓
MCP Response
↓
Client
```
## 🧪 Testing
### Run Test Suite
```bash
# All unit tests
npm run test:run
# Performance benchmarks
npm run test:performance
# Full CI suite
npm run test:ci
# Individual tool test
npx @modelcontextprotocol/inspector --cli node dist/index.js \
--method tools/call \
--tool-name crawl_read \
--tool-arg url="https://example.com"
```
### Test Coverage
- ✅ All 11 tools tested
- ✅ Error handling validated
- ✅ Performance benchmarks
- ✅ Integration workflows
- ✅ Multi-Node support (Node 18, 20, 22)
### CI/CD Pipeline
```
┌────────────────────────────────────┐
│ GitHub Actions │
├────────────────────────────────────┤
│ 1. Test (Matrix: Node 18,20,22) │
│ 2. Integration Tests (PR only) │
│ 3. Performance Tests (main) │
│ 4. Security Scan │
│ 5. Coverage Report │
└────────────────────────────────────┘
```
## 🔧 Development
### Prerequisites
- Node.js 18 or higher
- npm or yarn
### Setup
```bash
# Clone the repository
git clone https://github.com/Git-Fg/searchcrawl-mcp-server.git
cd crawl-mcp-server
# Install dependencies
npm install
# Build TypeScript
npm run build
# Run in development mode (watch)
npm run dev
```
### Development Commands
```bash
# Build project
npm run build
# Watch mode (auto-rebuild)
npm run dev
# Run tests
npm run test:run
# Lint code
npm run lint
# Type check
npm run typecheck
# Clean build artifacts
npm run clean
```
### Project Structure
```
crawl-mcp-server/
├── src/
│ ├── index.ts # Main server (11 tools)
│ ├── types.ts # TypeScript interfaces
│ └── cdp.ts # Chrome CDP manager
├── test/
│ ├── run-tests.ts # Unit test suite
│ ├── performance.ts # Performance tests
│ └── config.ts # Test configuration
├── dist/ # Compiled JavaScript
├── .github/workflows/ # CI/CD pipeline
└── package.json
```
## 📊 Performance
### Benchmarks
| Operation | Avg Duration | Max Memory |
|-----------|-------------|------------|
| crawl_read | ~1500ms | 32MB |
| crawl_read_batch (2 URLs) | ~2500ms | 64MB |
| search_searx | ~4000ms | 128MB |
| crawl_fetch | ~2000ms | 48MB |
| tools/list | ~100ms | 8MB |
### Performance Features
- ✅ Concurrent request processing (up to 20)
- ✅ Built-in caching (SHA-256)
- ✅ Automatic timeout management
- ✅ Memory optimization
- ✅ Resource cleanup
## 🛡️ Error Handling
All tools include comprehensive error handling:
- **Network errors**: Graceful degradation with error messages
- **Timeout handling**: Configurable timeouts
- **Partial failures**: Batch operations continue on individual failures
- **Structured errors**: Clear error codes and messages
- **Recovery**: Automatic retries where appropriate
Example error response:
```json
{
"content": [
{
"type": "text",
"text": "Error: Failed to fetch https://example.com: Timeout after 30000ms"
}
],
"structuredContent": {
"error": "Network timeout",
"url": "https://example.com",
"code": "TIMEOUT"
}
}
```
## 🔐 Security
- **No API keys required** for basic crawling
- **Respect robots.txt** (configurable)
- **User agent rotation**
- **Rate limiting** (built-in via concurrency limits)
- **Input validation** (Zod schemas)
- **Dependency scanning** (npm audit, Snyk)
## 🌐 Transport Modes
### STDIO (Default)
For local MCP clients:
```bash
node dist/index.js
```
### HTTP
For remote access:
```bash
TRANSPORT=http PORT=3000 node dist/index.js
```
Server runs on: `http://localhost:3000/mcp`
## 📝 Configuration
### Environment Variables
```bash
# Transport mode (stdio or http)
TRANSPORT=stdio
# HTTP port (when TRANSPORT=http)
PORT=3000
# Node environment
NODE_ENV=production
```
### Tool Configuration
Each tool accepts an `options` object:
```typescript
{
"timeout": 30000, // Request timeout (ms)
"maxConcurrency": 5, // Concurrent requests (1-20)
"maxResults": 10, // Limit results (1-50)
"respectRobots": false, // Respect robots.txt
"sameOriginOnly": true // Only same-origin URLs
}
```
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch: `git checkout -b feature/amazing-feature`
3. Make changes and add tests
4. Run tests: `npm run test:ci`
5. Commit: `git commit -m 'Add amazing feature'`
6. Push: `git push origin feature/amazing-feature`
7. Open a Pull Request
### Development Guidelines
- Follow TypeScript strict mode
- Add tests for new features
- Update documentation
- Run linting: `npm run lint`
- Ensure CI passes
## 📄 License
MIT License - see [LICENSE](LICENSE) file
## 🙏 Acknowledgments
- [@just-every/crawl](https://github.com/just-every/crawl) - Web crawling
- [Model Context Protocol](https://modelcontextprotocol.io) - MCP specification
- [SearXNG](https://github.com/searxng/searxng) - Search aggregator
- [Playwright](https://playwright.dev) - Browser automation
## 📞 Support
- **Issues**: [GitHub Issues](https://github.com/Git-Fg/searchcrawl-mcp-server/issues)
- **Discussions**: [GitHub Discussions](https://github.com/Git-Fg/searchcrawl-mcp-server/discussions)
- **Email**: your-email@example.com
## 🚀 What's Next?
- [ ] Add DuckDuckGo search support
- [ ] Implement content filtering
- [ ] Add screenshot capabilities
- [ ] Support for authenticated content
- [ ] PDF extraction
- [ ] Real-time monitoring
---
**Built with ❤️ using TypeScript, MCP, and modern web technologies.**
