# MCP WebScout

A Model Context Protocol (MCP) server providing web search (DuckDuckGo) and intelligent content extraction with LLM-powered analysis.

## Features

- **search**: Search the web using DuckDuckGo
- **fetch**: Advanced web fetching with Crawl4AI and LLM extraction

## System Requirements

| Requirement      | Version | Notes                                       |
| ---------------- | ------- | ------------------------------------------- |
| Python           | >= 3.10 | Required runtime environment                |
| pip              | latest  | Package manager (included with Python)      |
| Playwright       | latest  | Required by Crawl4AI for browser automation |
| DeepSeek API Key | -       | Required for LLM extraction mode            |
| Proxy (optional) | -       | Required for users in mainland China        |

### Python Dependencies (14 packages)

| Package           | Version  | Purpose                        |
| ----------------- | -------- | ------------------------------ |
| mcp               | >=1.0.0  | MCP protocol implementation    |
| duckduckgo-search | >=3.0.0  | DuckDuckGo search API          |
| requests          | >=2.32.0 | HTTP requests                  |
| beautifulsoup4    | >=4.12.0 | HTML parsing                   |
| openai            | >=1.30.0 | OpenAI API client for DeepSeek |
| crawl4ai          | >=0.5.0  | Advanced web scraping          |

## Quick Start

Get started in 5 steps:

### 1. Clone and Setup Environment

```bash
git clone <repository>
cd mcp-webscout
python -m venv .venv
```

On Windows:
```powershell
.venv\Scripts\activate
```

On macOS/Linux:
```bash
source .venv/bin/activate
```

### 2. Install Dependencies

```bash
pip install -e ".[dev]"
```

### 3. Install Playwright Browsers

```bash
playwright install chromium
```

### 4. Configure Environment Variables

```bash
cp .env.example .env
```

Edit `.env` and add your configuration:

```env
# Required for LLM extraction
DEEPSEEK_API_KEY=sk-your-actual-key-here

# Required for mainland China users
PROXY_URL=http://127.0.0.1:7890
USE_PROXY=true
```

### 5. Verify Installation

```bash
# Run tests
pytest tests/ -v

# Test the server
python -m mcp_webscout --help
```

## Detailed Configuration

For detailed environment setup instructions, see [ENV_SETUP.md](ENV_SETUP.md).

## Usage

### As a Command

```bash
mcp-webscout
```

### As a Python Module

```bash
python -m mcp_webscout
```

### With Claude Desktop

Add to your `claude_desktop_config.json`:

#### Basic Configuration

```json
{
  "mcpServers": {
    "webscout": {
      "command": "mcp-webscout"
    }
  }
}
```

#### With Environment Variables (Recommended)

```json
{
  "mcpServers": {
    "webscout": {
      "command": "mcp-webscout",
      "env": {
        "DEEPSEEK_API_KEY": "sk-your-key-here",
        "PROXY_URL": "http://127.0.0.1:7890",
        "USE_PROXY": "true",
        "DEFAULT_MAX_LENGTH": "5000",
        "PYTHONUTF8": "1"
      }
    }
  }
}
```

#### Windows Configuration

```json
{
  "mcpServers": {
    "webscout": {
      "command": "python",
      "args": ["-m", "mcp_webscout"],
      "env": {
        "DEEPSEEK_API_KEY": "sk-your-key-here",
        "PROXY_URL": "http://127.0.0.1:7890",
        "USE_PROXY": "true",
        "PYTHONUTF8": "1"
      }
    }
  }
}
```

## Tools

### search

Search the web using DuckDuckGo.

**Parameters:**

| Name        | Type    | Required | Description                        |
| ----------- | ------- | -------- | ---------------------------------- |
| query       | string  | Yes      | Search query                       |
| max_results | integer | No       | Maximum results (1-10, default: 5) |

**Returns:**

Formatted search results with titles, URLs, and snippets.

**Example:**

```json
{
  "query": "Python programming",
  "max_results": 3
}
```

### fetch

Advanced web fetching with Crawl4AI and LLM extraction.

**Parameters:**

| Name       | Type    | Required | Description                                    |
| ---------- | ------- | -------- | ---------------------------------------------- |
| url        | string  | Yes      | URL to fetch                                   |
| mode       | string  | No       | Extraction mode: simple, llm (default: simple) |
| prompt     | string  | No       | Custom extraction prompt for LLM mode          |
| max_length | integer | No       | Maximum characters (default: 5000)             |
| use_proxy  | boolean | No       | Use proxy (default: true)                      |

**Returns:**

Fetched and optionally extracted content.