# Web Fetch MCP Server

A Model Context Protocol (MCP) server that enables LLMs to fetch and process web page contents. This server provides tools for retrieving raw HTML, extracting clean text, and converting web pages to markdown format.

## Features

- **fetch_webpage**: Retrieves raw HTML content from any URL
- **fetch_webpage_text**: Extracts clean text content without HTML tags
- **fetch_webpage_markdown**: Converts web pages to markdown format

## Installation

1. Install dependencies:
```bash
npm install
```

2. Build the project:
```bash
npm run build
```

## Usage with LM Studio

To use this MCP server with LM Studio, you need to configure it in your MCP settings.

### Configuration

Add the following to your LM Studio MCP configuration file (typically located at `~/.lmstudio/mcp-settings.json` or similar):

```json
{
  "mcpServers": {
    "web-fetch": {
      "command": "node",
      "args": ["D:\\Projects\\web search mcp\\dist\\index.js"]
    }
  }
}
```

Adjust the path to match your actual installation directory.

### Alternative: Global Installation

You can also install the server globally or use npx:

```bash
npm install -g .
```

Then configure LM Studio to use:
```json
{
  "mcpServers": {
    "web-fetch": {
      "command": "web-fetch-mcp"
    }
  }
}
```

## Available Tools

### 1. fetch_webpage

Fetches the raw HTML content of a webpage.

**Input:**
- `url` (string, required): The URL of the webpage to fetch

**Example:**
```json
{
  "url": "https://example.com"
}
```

### 2. fetch_webpage_text

Fetches a webpage and extracts clean text content without HTML tags, scripts, or styles.

**Input:**
- `url` (string, required): The URL of the webpage to fetch

**Example:**
```json
{
  "url": "https://example.com"
}
```

### 3. fetch_webpage_markdown

Fetches a webpage and converts it to markdown format, preserving structure like headings, links, and lists.

**Input:**
- `url` (string, required): The URL of the webpage to fetch

**Example:**
```json
{
  "url": "https://example.com"
}
```

## Development

### Build
```bash
npm run build
```

### Watch mode (auto-rebuild on changes)
```bash
npm run dev
```

### Run directly
```bash
npm start
```

## Technical Details

- **Protocol**: Model Context Protocol (MCP) via stdio transport
- **HTTP Client**: Axios with 30-second timeout and automatic redirect handling
- **HTML Parsing**: Cheerio for efficient HTML parsing
- **Markdown Conversion**: Turndown for high-quality HTML to Markdown conversion
- **User Agent**: Mimics modern browser to avoid bot detection

## Error Handling

The server includes comprehensive error handling for:
- Network failures
- Invalid URLs
- Timeout errors
- HTTP errors (404, 500, etc.)

Errors are returned in a structured format with descriptive messages.

## Security Notes

- The server fetches content from any URL provided
- Be cautious when fetching content from untrusted sources
- Consider implementing URL allowlists for production use
- The server respects redirects (max 5)

## License

MIT