mult-fetch-mcp-server
by lmcc-dev
README.md
[](https://mseep.ai/app/lmcc-dev-mult-fetch-mcp-server)
# mult-fetch-mcp-server
[](https://www.npmjs.com/package/@lmcc-dev/mult-fetch-mcp-server)
[](https://opensource.org/licenses/MIT)
[](https://nodejs.org/)
[](https://www.typescriptlang.org/)
[](https://github.com/modelcontextprotocol/typescript-sdk)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/stargazers)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/network/members)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/issues)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/pulls)
[](https://www.npmjs.com/package/@lmcc-dev/mult-fetch-mcp-server)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/commits/main)
[](https://github.com/lmcc-dev/mult-fetch-mcp-server/graphs/contributors)
[](https://lightnow.ai/servers/io.github.lmcc-dev/mult-fetch-mcp-server)
[](https://codecov.io/gh/lmcc-dev/mult-fetch-mcp-server)
[](https://www.codefactor.io/repository/github/lmcc-dev/mult-fetch-mcp-server)
<!-- Future badges to consider:
[](https://www.codefactor.io/repository/github/lmcc-dev/mult-fetch-mcp-server)
[](https://codeclimate.com/github/lmcc-dev/mult-fetch-mcp-server/maintainability)
-->
This project implements an MCP-compliant client and server for communication between AI assistants and external tools.
[English](./README.md) | [中文文档](./README.zh.md)
## Version 2.0
Call the `fetch` tool with a URL. The default format is Markdown, using an HTTP fetch plus readable-article extraction. `startCursor` is not required.
| `format` | What you get |
| --- | --- |
| `markdown` | Readable article as Markdown (default) |
| `plaintext` | Readable article as plain text |
| `html` | HTML markup |
| `json` | JSON document |
| `txt` | Raw response body |
A normal page comes back in one response. The default budget is `maxChars` 100000 and `maxBytes` 1048576, and a cut prefers a newline or a space. When the body is longer, the response includes `nextCursor` and `chunkId`. Call `fetch` again with `cursor` set to `nextCursor`.
`fetch_html`, `fetch_json`, `fetch_txt`, `fetch_markdown`, and `fetch_plaintext` are thin aliases of `fetch` with a fixed format.
### Cursor
Add this to `~/.cursor/mcp.json`, or to `.cursor/mcp.json` in a project:
```json
{
"mcpServers": {
"mult-fetch": {
"command": "npx",
"args": ["-y", "@lmcc-dev/mult-fetch-mcp-server"],
"env": {
"MCP_LANG": "en"
}
}
}
}
```
### Claude Desktop
- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows**: `%APPDATA%/Claude/claude_desktop_config.json`
```json
{
"mcpServers": {
"mult-fetch": {
"command": "npx",
"args": ["-y", "@lmcc-dev/mult-fetch-mcp-server"],
"env": {
"MCP_LANG": "en"
}
}
}
}
```
Restart the host after saving. Ask it to fetch a URL. It should call `fetch` without `startCursor`.
### Browser mode
The default HTTP fetcher does not need Chrome or Puppeteer. A normal install works without them (`npm install --omit=optional` skips the optional packages). To render pages in a headless browser, install the optional packages and set `useBrowser` to true:
```bash
npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
```
`autoDetectMode` defaults to false, so a 403 does not launch a browser unless you set that flag.
### Security
Every request, including redirects, is checked before it is sent:
- Loopback (`127.0.0.0/8`, `::1`, `localhost`)
- Private networks (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`)
- Link-local and cloud metadata (`169.254.0.0/16`, including `169.254.169.254`, and IPv6 `fe80::/10`)
- Carrier-grade NAT `100.64.0.0/10`, documentation ranges, and private addresses embedded in 6to4, NAT64, or IPv4-mapped IPv6
Direct connections are pinned to the addresses that passed that check. The default timeout is 30 seconds, and each response is limited by `maxChars` and `maxBytes`. Browser mode keeps the same URL checks. Set `ALLOW_PRIVATE_NETWORK_REQUESTS=true` only when you intentionally want the browser to open a private-network target.
## Project Structure
```
fetch-mcp/
├── src/ # Source code directory
│ ├── lib/ # Library files
│ │ ├── fetchers/ # Web fetching implementation
│ │ │ ├── browser/ # Browser-based fetching
│ │ │ │ ├── BrowserFetcher.ts # Browser fetcher implementation
│ │ │ │ ├── BrowserInstance.ts # Browser instance management
│ │ │ │ └── PageOperations.ts # Page interaction operations
│ │ │ ├── node/ # Node.js-based fetching
│ │ │ └── common/ # Shared fetching utilities
│ │ ├── utils/ # Utility modules
│ │ │ ├── ChunkManager.ts # Content chunking
│ │ │ ├── ContentProcessor.ts # HTML to text conversion
│ │ │ ├── ContentExtractor.ts # Intelligent content extraction
│ │ │ ├── ContentSizeManager.ts # Content size limiting
│ │ │ └── ErrorHandler.ts # Error handling
│ │ ├── server/ # Server-related modules
│ │ │ ├── index.ts # Server entry
│ │ │ ├── browser.ts # Browser management
│ │ │ ├── fetcher.ts # Web fetching logic
│ │ │ ├── tools.ts # Tool registration and handling
│ │ │ ├── resources.ts # Resource handling
│ │ │ ├── prompts.ts # Prompt templates
│ │ │ └── types.ts # Server type definitions
│ │ ├── i18n/ # Internationalization support
│ │ └── types.ts # Common type definitions
│ ├── client.ts # MCP client implementation
│ └── mcp-server.ts # MCP server main entry
├── index.ts # Server entry point
├── tests/ # Test files
└── dist/ # Compiled files
```
## MCP Specification
The Model Context Protocol (MCP) defines two main transport methods:
1. **Standard Input/Output (Stdio)**: The client starts the MCP server as a child process, and they communicate through standard input (stdin) and standard output (stdout).
2. **Server-Sent Events (SSE)**: Used to pass messages between client and server.
This project implements the Standard Input/Output (Stdio) transport method.
## Features
- Implementation based on the official MCP SDK
- Support for Standard Input/Output (Stdio) transport
- One `fetch` tool with `format` set to HTML, JSON, text, Markdown, or plain text
- HTTP fetch by default, with optional headless-browser rendering
- Single-response content window (`maxChars` / `maxBytes`) and optional `cursor` continuation when the body is truncated
- Detailed debug logging to stderr
- Bilingual internationalization (English and Chinese)
- Modular design for easy maintenance and extension
- **Intelligent Content Extraction**: Based on Mozilla's Readability library, capable of extracting meaningful content from web pages while filtering out advertisements and navigation elements
- **Metadata Support**: Ability to extract webpage metadata such as title, author, publication date, and site information
- **Smart Content Detection**: Automatically detects if a page contains meaningful content, filtering out login pages, error pages, and other pages without substantial content
- **Browser Automation Enhancements**: Support for page scrolling, cookie management, selector waiting, and other advanced browser interactions
## Installation
### Installing via Smithery
To install Mult Fetch MCP Server for Claude Desktop automatically via [Smithery](https://smithery.ai/server/@lmcc-dev/mult-fetch-mcp-server):
```bash
npx -y @smithery/cli install @lmcc-dev/mult-fetch-mcp-server --client claude
```
### Local Installation
```bash
pnpm install
```
### Global Installation
```bash
pnpm add -g @lmcc-dev/mult-fetch-mcp-server
```
Or run directly with npx (no installation required):
```bash
npx @lmcc-dev/mult-fetch-mcp-server
```
## Integration with Claude
To integrate this tool with Claude desktop, you need to add server configuration:
### Configuration File Location
- **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
- **Windows**: `%APPDATA%/Claude/claude_desktop_config.json`
### Configuration Examples
#### Method 1: Using npx (Recommended)
This method is the simplest, doesn't require specifying the full path, and is suitable for global installation or direct use with npx:
```json
{
"mcpServers": {
"mult-fetch-mcp-server": {
"command": "npx",
"args": ["@lmcc-dev/mult-fetch-mcp-server"],
"env": {
"MCP_LANG": "en" // Set language to English, options: "zh" or "en"
}
}
}
}
```
#### Method 2: Specifying Full Path
If you need to use a specific installation location, you can specify the full path:
```json
{
"mcpServers": {
"mult-fetch-mcp-server": {
"command": "path-to/bin/node",
"args": ["path-to/@lmcc-dev/mult-fetch-mcp-server/dist/index.js"],
"env": {
"MCP_LANG": "en" // Set language to English, options: "zh" or "en"
}
}
}
}
```
Please replace `path-to/bin/node` with the path to the Node.js executable on your system, and replace `path-to/@lmcc-dev/mult-fetch-mcp-server` with the actual path to this project.
### Usage Examples
Below is an example of using this tool in Claude desktop client:
The image shows how Claude can use the fetch tools to retrieve web content and process it according to your instructions.
### Usage
After configuration, restart Claude desktop, and you can use the following tools in your conversation:
- `fetch`: Fetch a URL. Set `format` to `html`, `markdown` (default), `plaintext`, `json`, or `txt`
- `fetch_html`, `fetch_json`, `fetch_txt`, `fetch_markdown`, `fetch_plaintext`: aliases of `fetch` kept from 1.x
## Build
```bash
pnpm run build
```
## Run Server
```bash
pnpm run server
# or
node dist/index.js
# if globally installed, you can run directly
@lmcc-dev/mult-fetch-mcp-server
# or use npx
npx @lmcc-dev/mult-fetch-mcp-server
```
## Client Demo Tools
> **Note**: The following client.js functionality is provided for demonstration and testing purposes only. When used with Claude or other AI assistants, the MCP server is driven by the AI, which manages the chunking process automatically.
### Command Line Client
The project includes a command-line client for testing and development purposes:
```bash
pnpm run client <method> <params_json>
# example
pnpm run client fetch_html '{"url": "https://example.com", "debug": true}'
```
### Demo Client Chunk Control Parameters
When testing with the command-line client, you can use these parameters to demonstrate content chunking capabilities:
- `--all-chunks`: Command line flag to automatically fetch all chunks in sequence (demonstration purpose only)
- `--max-chunks`: Command line flag to limit the maximum number of chunks to fetch (optional, default is 10)
#### Real-time Output Demo
The client.js demo tool provides real-time output capabilities:
```bash
node dist/src/client.js fetch_html '{"url":"https://example.com", "startCursor": 0, "contentSizeLimit": 500}' --all-chunks --debug
```
The demo client will automatically fetch all chunks in sequence and display them immediately, showcasing how large content can be processed in real-time.
## Run Tests
```bash
# Run MCP functionality tests
npm run test:mcp
# Run mini4k.com website tests
npm run test:mini4k
# Run direct client call tests
npm run test:direct
```
## Language Settings
This project supports Chinese and English bilingual internationalization. You can set the language using environment variables:
### Using Environment Variables
Set the `MCP_LANG` environment variable to control the language:
```bash
# Set to English
export MCP_LANG=en
npm run server
# Set to Chinese
export MCP_LANG=zh
npm run server
# Windows system
set MCP_LANG=zh
npm run server
```
Using environment variables ensures that all related processes (including the MCP server) use the same language settings.
### Default Language
By default, the system will choose a language according to the following priority:
1. `MCP_LANG` environment variable
2. Operating system language (if it starts with "zh", use Chinese)
3. English (as the final fallback option)
## Debugging
This project follows the MCP protocol specification and does not output any logs by default to avoid interfering with JSON-RPC communication. Debug information is controlled through call parameters:
### Using the debug Parameter
Set the `debug: true` parameter when calling a tool:
```json
{
"url": "https://example.com",
"debug": true
}
```
Debug messages are sent to the standard error stream (stderr) using the following format:
```
[MCP-SERVER] MCP server starting...
[CLIENT] Fetching URL: https://example.com
```
### Debug Log File
When debug mode is enabled, all debug messages are also written to a log file located at:
```
~/.mult-fetch-mcp-server/debug.log
```
This log file can be accessed through the MCP resources API:
```typescript
// Access the debug log file
const result = await client.readResource({ uri: "file:///logs/debug" });
console.log(result.contents[0].text);
// Clear the debug log file
const clearResult = await client.readResource({ uri: "file:///logs/clear" });
console.log(clearResult.contents[0].text);
```
## Proxy Settings
This tool supports various methods to configure proxy settings:
### 1. Using the `proxy` Parameter
The most direct way is to specify the proxy in the request parameters:
```json
{
"url": "https://example.com",
"proxy": "http://your-proxy-server:port",
"debug": true
}
```
### 2. Using Environment Variables
The tool will automatically detect and use proxy settings from standard environment variables:
```bash
# Set proxy environment variables
export HTTP_PROXY=http://your-proxy-server:port
export HTTPS_PROXY=http://your-proxy-server:port
# Run the server
npm run server
```
### 3. System Proxy Detection
The tool attempts to detect system proxy settings based on your operating system:
- **Windows**: Reads proxy settings from environment variables using the `set` command
- **macOS/Linux**: Reads proxy settings from environment variables using the `env` command
### 4. Proxy Troubleshooting
If you're having issues with proxy detection:
1. Use the `debug: true` parameter to see detailed logs about proxy detection
2. Explicitly specify the proxy using the `proxy` parameter
3. Ensure your proxy URL is in the correct format: `http://host:port` or `https://host:port`
4. For websites that require browser capabilities, set `useBrowser: true` to use browser mode
### 5. Browser Mode and Proxies
When using browser mode (`useBrowser: true`), the tool will:
1. First try to use the explicitly specified proxy (if provided)
2. Then try to use system proxy settings
3. Finally, proceed without a proxy if none is found
Browser mode is particularly useful for websites that implement anti-scraping measures or require JavaScript execution.
## Parameter Handling
This project handles parameters in the following ways:
- **debug**: Passed through call parameters, each request can individually control whether to enable debug output
- **MCP_LANG**: Retrieved from environment variables, controls the language settings of the entire server
## Usage
### Creating a Client
```typescript
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
import path from 'path';
import { fileURLToPath } from 'url';
// Get the directory path of the current file
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
// Create client transport layer
const transport = new StdioClientTransport({
command: 'node',
args: [path.resolve(__dirname, 'dist/index.js')],
stderr: 'inherit',
env: {
...process.env // Pass all environment variables, including MCP_LANG
}
});
// Create client
const client = new Client({
name: "example-client",
version: "1.0.0"
});
// Connect to transport layer
await client.connect(transport);
// Use client
const result = await client.callTool({
name: 'fetch_html',
arguments: {
url: 'https://example.com',
debug: true // Control debug output through parameters
}
});
if (result.isError) {
console.error('Fetch failed:', result.content[0].text);
} else {
console.log('Fetch successful!');
console.log('Content preview:', result.content[0].text.substring(0, 500));
}
```
### Supported Tools
- `fetch`: Fetch a URL. `format` is `html`, `markdown` (default), `plaintext`, `json`, or `txt`
- `fetch_html`, `fetch_json`, `fetch_txt`, `fetch_markdown`, `fetch_plaintext`: 1.x aliases of `fetch`
### Resources Support
The server includes support for the resources/list and resources/read methods, but currently no resources are defined in the implementation. The resource system is designed to provide access to project files and documentation, but this feature is not fully implemented yet.
#### Resource Usage Example
```typescript
// Example: List available resources
const resourcesResult = await client.listResources({});
console.log('Available resources:', resourcesResult);
// Note: Currently this will return empty lists for resources and resourceTemplates
```
### Supported Prompt Templates
The server provides the following prompt templates:
- `fetch-website`: Get website content, supporting different formats and browser mode
- `extract-content`: Extract specific content from a website, supporting CSS selectors and data type specification
- `debug-fetch`: Debug website fetching issues, analyze possible causes and provide solutions
#### Prompt Template Usage
1. Use `prompts/list` to get a list of available prompt templates
2. Use `prompts/get` to get specific prompt template content
```typescript
// Example: List available prompt templates
const promptsResult = await client.listPrompts({});
console.log('Available prompts:', promptsResult);
// Example: Get website content prompt
const fetchPrompt = await client.getPrompt({
name: "fetch-website",
arguments: {
url: "https://example.com",
format: "html",
useBrowser: "false"
}
});
console.log('Fetch website prompt:', fetchPrompt);
// Example: Debug website fetching issues
const debugPrompt = await client.getPrompt({
name: "debug-fetch",
arguments: {
url: "https://example.com",
error: "Connection timeout"
}
});
console.log('Debug fetch prompt:', debugPrompt);
```
### Parameter Options
Each tool supports the following parameters:
#### Basic Parameters
- `url`: URL to fetch (required)
- `headers`: Custom request headers (optional, default {})
- `proxy`: Proxy server URL in the format http://host:port or https://host:port (optional)
#### Network Control Parameters
- `timeout`: Timeout in milliseconds (optional, default is 30000)
- `maxRedirects`: Maximum number of redirects to follow (optional, default is 10)
- `noDelay`: Whether to disable random delay between requests (optional, default is false)
- `useSystemProxy`: Whether to use system proxy (optional, default is true)
#### Content Window
- `maxChars`: Maximum characters in this response (optional, default 100000). The cut prefers a newline or space.
- `maxBytes`: Maximum UTF-8 bytes in this response (optional, default 1048576).
- `cursor`: Continuation token from `nextCursor`. Omit it on the first call. When it is set, the server does not refetch the URL.
- `enableContentSplitting`: When false, oversized content is truncated with no cursor (optional, default true).
- `contentSizeLimit`: Legacy byte-chunk size. When set without `maxChars` or `maxBytes`, the 1.x byte splitter is used.
- `startCursor`: Legacy byte offset. Not required. Use it only with a 1.x `chunkId`.
#### Chunk Management
- `chunkId`: Document id returned with `nextCursor` when a response was truncated.
Pass `cursor` back to `fetch` to read the next slice. `startCursor` is not part of that flow.
#### Mode Control Parameters
- `useBrowser`: Whether to use browser mode (optional, default is false)
- `useNodeFetch`: Whether to force using Node.js mode (optional, default is false, mutually exclusive with `useBrowser`)
- `autoDetectMode`: Retry a 403 or challenge page in the browser (optional, default false). Requires the optional Puppeteer packages.
#### Browser Mode Specific Parameters
- `waitForSelector`: Selector to wait for in browser mode (optional, default is 'body')
- `waitForTimeout`: Timeout to wait in browser mode in milliseconds (optional, default is 5000)
- `scrollToBottom`: Whether to scroll to the bottom of the page in browser mode (optional, default is false)
- `saveCookies`: Whether to save cookies in browser mode (optional, default is true)
- `closeBrowser`: Whether to close the browser instance (optional, default is false)
#### Content Extraction Parameters
- `extractContent`: Use Readability before conversion (optional). Default true for `markdown` and `plaintext`, false for `html`, `json`, and `txt`.
- `includeMetadata`: Whether to include metadata in the extracted content (optional, default false, only works when `extractContent` is true)
- `fallbackToOriginal`: Whether to fall back to the original content when extraction fails (optional, default true, only works when `extractContent` is true)
#### Debug Parameters
- `debug`: Whether to enable debug output (optional, default false)
### Content Extraction Feature
Use the content extraction feature to get the core content of a webpage, filtering out navigation bars, advertisements, sidebars, and other distracting elements:
```json
{
"url": "https://example.com/article",
"extractContent": true,
"includeMetadata": true
}
```
The extracted content will include the following metadata (if available):
- Title
- Byline (author)
- Site name
- Excerpt
- Content length
- Readability flag (isReaderable)
### Special Usage
#### Content Extraction Examples
To extract only the meaningful content from an article webpage:
```json
{
"url": "https://example.com/news/article",
"extractContent": true,
"includeMetadata": true
}
```
For websites where content extraction might fail, you can use `fallbackToOriginal` to ensure you get some content:
```json
{
"url": "https://example.com/complex-layout",
"extractContent": true,
"fallbackToOriginal": true
}
```
#### Closing Browser Without Fetching
To close the browser instance without performing any fetch operation:
```json
{
"url": "about:blank",
"closeBrowser": true
}
```
#### Proxy Priority
The proxy is determined in the following order:
1. Command line specified proxy
2. `proxy` parameter in the request
3. Environment variables (if `useSystemProxy` is true)
4. Git configuration (if `useSystemProxy` is true)
If `proxy` is set, `useSystemProxy` will be automatically set to false.
### Debug Output
When `debug: true` is set, logs will be output to stderr with the following prefixes:
- `[MCP-SERVER]`: Logs from the MCP server
- `[NODE-FETCH]`: Logs from the Node.js fetcher
- `[BROWSER-FETCH]`: Logs from the browser fetcher
- `[CLIENT]`: Logs from the client
- `[TOOLS]`: Logs from the tool implementation
- `[FETCHER]`: Logs from the main fetcher interface
- `[CONTENT]`: Logs related to content handling
- `[CONTENT-PROCESSOR]`: Logs from the HTML content processor
- `[CONTENT-SIZE]`: Logs related to content size management
- `[CHUNK-MANAGER]`: Logs related to content chunking operations
- `[ERROR-HANDLER]`: Logs related to error handling
- `[BROWSER-MANAGER]`: Logs from the browser instance manager
- `[CONTENT-EXTRACTOR]`: Logs from the content extractor
## License
MIT
---
Updated by lmcc-dev
TDQS
A3.6/5.0
Scored across 6 tools
Disambiguation2/5
The five alias tools are explicitly deprecated duplicates of fetch with a preset format, so an agent can easily pick fetch_html instead of fetch with format=html. The descriptions do warn against this, but the tool set still presents six overlapping entry points for essentially one operation.
Naming Consistency4/5
All names use a consistent snake_case fetch_* pattern, with the base tool simply named fetch. The convention is predictable, though the aliases create redundant variants rather than distinct operations.
Tool Count2/5
Six tools are listed, but five are deprecated aliases that add no new capability. The effective surface is one tool, making the count misleading and cluttered rather than well-scoped.
Completeness4/5
The core fetch tool covers multiple formats, truncation cursors, timeouts, size limits, and optional browser rendering, which is solid coverage for an HTTP fetching server. The aliases do not fill real gaps, but the primary tool is fairly complete.
Maintenance
ActivityMaintained
ResponsivenessNo issues