# WebFetch MCP Server

An MCP (Model Context Protocol) server that fetches webpages by URL, renders JavaScript using Puppeteer with stealth plugins to avoid bot detection, converts the HTML content to clean Markdown, and returns it for LLM analysis.

## Features

- **JavaScript Rendering**: Uses Puppeteer with stealth plugin to render dynamic content
- **Bot Evasion**: Stealth plugins prevent detection as a bot
- **Markdown Conversion**: Converts HTML to clean, readable Markdown using Turndown
- **Content Filtering**: Removes ads, scripts, styles, and other non-essential elements
- **Link Preservation**: Maintains links with their URLs for context
- **Image References**: Preserves image alt text and sources as markdown references

## Quick Start

### Using Docker with Streamable Transport

Build the image (no server needed to be running):

```bash
make build
# or: docker build -t webfetch-mcp-server .
```

Configure your MCP client to use streamable transport with Docker. The container will be started on-demand by the MCP client.

### Using Make

A `Makefile` is included for easy management:

```bash
make build      # Build the Docker image
make clean      # Remove container and image
make logs       # Follow container logs (if running)
make debug      # Run in foreground to see output
```

### Running from GitHub (npx)

Run directly from your GitHub repository without cloning:

```bash
npx git+https://github.com/Sh1d0w/webfetch-mcp.git -- -i
```

Or with the package name:

```bash
npx webfetch-mcp-server@latest -- -i
```

### Running Locally

Install dependencies:

```bash
npm install
```

Run in development mode:

```bash
npm run dev
```

Or build and run:

```bash
npm run build
npm start
```

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD` | Skip Chromium download (use system) | `true` |
| `PUPPETEER_EXECUTABLE_PATH` | Path to Chromium executable | `/usr/bin/chromium-browser` |

## MCP Client Configuration

### Docker Streamable Transport

Configure your MCP client to use streamable transport. The container starts on-demand when the MCP client connects.

**Docker run command (for reference):**

```bash
docker run --rm -i webfetch-mcp-server node dist/index.js
```

### Cursor

Add to your `.cursorrules` or Cursor MCP settings:

```json
{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}
```

### Claude Desktop (macOS)

Edit `~/Library/Application Support/Claude/claude_config.json`:

```json
{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}
```

### Claude Desktop (Linux)

Edit `~/.config/claude/claude_config.json`:

```json
{
  "mcpServers": {
    "webfetch": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "webfetch-mcp-server", "node", "dist/index.js"]
    }
  }
}
```

### LM Studio

In LM Studio MCP settings:

- **Name**: webfetch
- **Type**: stdio (streamable transport)
- **Command**: `docker`
- **Arguments**: `run --rm -i webfetch-mcp-server node dist/index.js`

**Why this works:**
- `--rm`: Automatically removes container after exit (clean state each time)
- `-i`: Keeps stdin open for MCP communication
- Container starts on-demand when LM Studio connects

### opencode

Add to your opencode configuration:

```typescript
import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js';

// In your config file
{
  mcpServers: {
    webfetch: {
      command: 'docker',
      args: ['run', '--rm', '-i', 'webfetch-mcp-server', 'node', 'dist/index.js']
    }
  }
}
```

## Why Xvfb is Required

Even in "headless" mode, Chromium/Puppeteer requires an X server for rendering. The Dockerfile includes:

- **Xvfb** (X Virtual Framebuffer): Provides a virtual display
- **DISPLAY=:99**: Tells Chromium to use the virtual display
- **Startup script**: Starts Xvfb before running Node.js

Without Xvfb, Puppeteer will crash immediately after starting, causing the container to restart in a loop.

## Available Tools

### `fetch_url`

Fetches a webpage and converts it to Markdown.

**Parameters:**
- `url` (string, required): The URL of the webpage to fetch

**Returns:**
- Clean Markdown content with title and URL header
- Error message if fetching fails

**Example Usage:**

```markdown
User: Please analyze this article about MCP protocol
Assistant: [calls tool] fetch_url with url="https://modelcontextprotocol.io"
```

## Output Format

The server returns content in the following format:

```markdown
# Page Title

https://example.com

---

[Clean markdown content here...]
```

### Content Processing

- Removes `<script>`, `<style>`, `nav`, `footer`, `header`, `aside` elements
- Strips advertisement containers (`.ad`, `.advertisement`, `#ads`)
- Removes cookie consent banners
- Converts links to include their URLs in parentheses
- Preserves images as markdown references with alt text
- Maintains code blocks formatting
- Normalizes whitespace and removes excessive line breaks

## Building for Production

Build with Make (recommended):

```bash
make build
```

Or directly with Docker:

```bash
docker build -t webfetch-mcp-server .
```

Configure your MCP client to use the built image with streamable transport. No persistent server needed.

## Troubleshooting

### Container keeps restarting / "Container is not running"

**Cause:** Puppeteer needs a virtual display (Xvfb) even in headless mode. Chromium's headless mode still requires an X server for rendering.

**Solution:** The Dockerfile includes Xvfb automatically. If you're using a custom setup, ensure:
1. Install Xvfb: `apk add xvfb` (Alpine) or `apt-get install xvfb` (Debian/Ubuntu)
2. Start Xvfb before running Chromium: `Xvfb :99 -screen 0 1024x768x24 &`
3. Set DISPLAY env var: `export DISPLAY=:99`

### "Connection closed" / MCP error -32000

**Cause:** Container crashed when the MCP client tried to connect.

**Solution:** Ensure you're using `docker run --rm` in your MCP configuration. This starts a fresh container for each connection, avoiding issues with stale containers.

### Check container logs

```bash
make logs
# or: docker logs -f webfetch-mcp-server
```

### Connection Timeout

The default timeout is 30 seconds. For slower websites, you may need to increase this in the source code.

### Stealth Detection Bypassed

Some websites have advanced bot detection. The stealth plugin helps but cannot guarantee bypass for all sites.

## Development

Run tests:

```bash
npm test
```

Lint and type-check:

```bash
npm run lint
npm run typecheck
```

## License

MIT License - See LICENSE file for details.