# MCP Fetch Page

Browser-based web page fetching with automatic cookie support and CSS selector extraction.

## Features

- 🤖 **Browser Automation**: Full JavaScript rendering with Puppeteer
- 🍪 **Automatic Cookie Management**: Loads all saved cookies automatically
- 🎯 **CSS Selector Support**: Extract specific content with selectors
- 🌐 **Domain Presets**: Built-in selectors for common websites
- 📱 **SPA Support**: Fully supports dynamic content and AJAX

## Quick Start

### 1. Configure MCP Server

Add to your Claude Desktop config (`~/Library/Application Support/Claude/claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "mcp-fetch-page": {
      "command": "npx",
      "args": ["-y", "mcp-fetch-page@latest"]
    }
  }
}
```

To customize runtime data directory (recommended on VPS), set `MCP_FETCH_PAGE_DATA_DIR` in MCP `env`:

```json
{
  "mcpServers": {
    "mcp-fetch-page": {
      "command": "npx",
      "args": ["-y", "mcp-fetch-page@latest"],
      "env": {
        "MCP_FETCH_PAGE_DATA_DIR": "/data/mcp-fetch-page"
      }
    }
  }
}
```

Restart Claude Desktop.

### 2. Install Chrome Extension (Optional - for authenticated pages)

Download and install the Chrome extension to save cookies from authenticated sessions:

**[📥 Download Extension from Releases](https://github.com/kaiye/mcp-fetch-page/releases/latest)**

Installation steps:
1. Download `mcp-fetch-page-extension-vX.X.X.zip` from the latest release
2. Unzip the file
3. Open Chrome and go to `chrome://extensions/`
4. Enable "Developer mode" (top right)
5. Click "Load unpacked" and select the unzipped folder

## Usage

### Basic Usage
1. **Login** to a website in Chrome
2. **Click** the "Fetch Page MCP Tools" extension icon  
3. **Click** "Save Cookies" button
4. **Use** in Claude/Cursor: `fetchpage(url="https://example.com")`

### Advanced Usage

```javascript
// Basic fetching with automatic cookie loading
fetchpage(url="https://example.com")

// Extract specific content with CSS selector
fetchpage(url="https://example.com", waitFor="#main-content")

// WeChat articles (automatic selector)
fetchpage(url="https://mp.weixin.qq.com/s/xxxxx")

// Run in non-headless mode for debugging
fetchpage(url="https://example.com", headless=false)
```

### Domain Presets

The system automatically uses optimized selectors for:
- **mp.weixin.qq.com** → `.rich_media_wrp` (WeChat articles)
- **wx.zsxq.com** → `.content` (Knowledge Planet)
- **cnblogs.com** → `.post` (Blog Garden)
- Add more in `mcp-server/domain-rules.json` (`domain-selectors.json` remains supported for compatibility)

### Debug Tools

```bash
# Standalone debug script (recommended for development)
cd mcp-server
node debug.js test-page "https://example.com"
node debug.js test-spa "https://example.com" "#content"

# MCP Inspector (for integration testing)
npx @modelcontextprotocol/inspector
# Then visit http://localhost:6274
```

### Data Directory (Optional)

By default, runtime data is stored under `~/Downloads/mcp-fetch-page/`:
- Cookies: `~/Downloads/mcp-fetch-page/cookies`
- Pages: `~/Downloads/mcp-fetch-page/pages`

For MCP usage, configure `MCP_FETCH_PAGE_DATA_DIR` in your MCP client config `env` field.
The server will always use:
- `<MCP_FETCH_PAGE_DATA_DIR>/cookies`
- `<MCP_FETCH_PAGE_DATA_DIR>/pages`
- `<MCP_FETCH_PAGE_DATA_DIR>/domain-rules.json` (optional user overrides merged with built-in rules)

`node mcp-server/server.js` is only for local development/debugging.

## Parameters

- `url` (required): The URL to fetch
- `waitFor` (optional): CSS selector to extract specific content
- `headless` (optional): Run browser in headless mode (default: true)
- `timeout` (optional): Timeout in milliseconds (default: 30000)

## File Structure

```
mcp-fetch-page/
├── package.json              # npm package config
├── package-lock.json         # npm lockfile
├── node_modules/             # npm dependencies
├── README.md                 # This file
├── README-zh.md              # Chinese version
├── CLAUDE.md                 # Claude Code usage guide
├── chrome-extension/         # Chrome extension
│   ├── manifest.json
│   ├── popup.js
│   ├── popup.html
│   └── background.js
└── mcp-server/              # MCP server
    ├── server.js            # Main server
    ├── debug.js             # Debug tools
    ├── domain-rules.json     # Domain rules config (selector + blocked markers)
    └── domain-selectors.json # Legacy selector config (compatibility fallback)
```

## Troubleshooting

- **Extension not working**: Make sure you're on a normal website (not chrome:// pages)
- **No cookies found**: Try logging in again and saving cookies
- **MCP not connecting**: Check Node.js installation and restart your editor
- **Path error**: Set `MCP_FETCH_PAGE_DATA_DIR` in MCP config `env` to a writable absolute path on your machine/VPS
- **CSS selector not working**: Verify the selector exists on the page

That's it! 🍪