# Spider MCP - Web Search Crawler Service

A web search MCP service based on pure crawler technology, built with Node.js.

## Features

- ❌ **No Official API Required**: Completely based on crawler technology, no dependency on third-party official APIs
- 🔍 **Intelligent Search**: Supports Bing web and news search
- 📰 **News Search**: Built-in news search with time filtering
- 🕷️ **Pure Crawler**: No official API dependency, uses Puppeteer for web scraping
- 🚀 **High Performance**: Supports batch web scraping
- 📊 **Health Monitoring**: Complete health check and metrics monitoring
- 📝 **Structured Logging**: Uses Winston for structured logs
- 🔒 **Anti-Detection**: Supports User-Agent rotation and other anti-bot measures
- 🔗 **Smart URL Cleaning**: Automatically cleans promotional parameters while preserving essential information

## Tech Stack

- **Node.js** (>= 18.0.0)
- **Express.js** - Web framework
- **Puppeteer** - Browser automation
- **Cheerio** - HTML parsing
- **Axios** - HTTP client
- **Winston** - Logging
- **@modelcontextprotocol/sdk** - MCP protocol support

## Quick Start

### 1. Install dependencies

```bash
npm install
```

or use `pnpm`

```bash
pnpm install
```

### 2. Download Puppeteer browser

```bash
npx puppeteer browsers install chrome
```

### 3. Environment configuration

Copy and configure the environment variables file:

```bash
cp .env.example .env
```

Edit the `.env` file according to your needs.

### 4. Start the service

Development mode:
```bash
npm run dev
```

Production mode:
```bash
npm start
```

The service will start at `http://localhost:3000`.

## MCP Tools

### web_search

Unified search tool supporting both web and news search:

- **Web Search**: `searchType: "web"`
- **News Search**: `searchType: "news"` with time filtering
- **Note**: `searchType` is a required parameter and must be explicitly specified

#### Usage Examples:

```
# Web search
Use web_search tool to search "Node.js tutorial" with searchType set to web, return 10 results

# News search
Use web_search tool to search "tech news" with searchType set to news, return 5 results from past 24 hours
```

### Other Tools

- `get_webpage_content`: Get webpage content and convert to specified format
- `get_webpage_source`: Get raw HTML source code of webpage
- `batch_webpage_scrape`: Batch scrape multiple webpages

## MCP Configuration

### Chatbox Configuration

Create `mcp-config.json` file in Chatbox:

```json
{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["src/mcp/server.js"],
      "env": {
        "NODE_ENV": "production"
      },
      "description": "Spider MCP - Web search and webpage scraping tools",
      "capabilities": {
        "tools": {}
      }
    }
  }
}
```

### Other MCP Clients

```json
{
  "mcpServers": {
    "spider-mcp": {
      "command": "node",
      "args": ["path/to/spider-mcp/src/mcp/server.js"]
    }
  }
}
```


## Important Notes

1. **Anti-bot Measures**: This service uses various techniques to avoid detection, but still needs to comply with robots.txt and terms of use
2. **Rate Limiting**: It's recommended to control request frequency reasonably to avoid putting pressure on target websites
3. **Legal Compliance**: Please ensure compliance with local laws and website terms of use when using this service
4. **Resource Consumption**: Puppeteer will start Chrome browser, please pay attention to memory and CPU usage
5. **URL Cleaning**: Automatically cleans promotional parameters but may affect some special link functionality

## Development

### Project Structure

```tree
spider-mcp/
├── src/
│   ├── index.js          # Main entry file
│   ├── mcp/
│   │   └── server.js     # MCP server
│   ├── routes/           # Route definitions
│   │   ├── search.js     # Search routes
│   │   └── health.js     # Health check routes
│   ├── services/         # Business logic
│   │   └── searchService.js # Search service
│   └── utils/            # Utility functions
│       └── logger.js     # Logging utility
├── logs/                 # Log files directory
├── tests/                # Test files
├── package.json          # Project configuration
├── .env.example          # Environment variables example
├── mcp-config.json       # MCP configuration example
└── README.md             # Project documentation
```

## License

MIT License

## Contributing

Issues and Pull Requests are welcome!