Supports containerization with Docker, including instructions for building, tagging, and pushing Docker images
Enables publishing the MCP server to GitHub, with specific instructions for repository setup and release creation
Allows publishing the package to PyPI for easier installation via package managers
MCP Fetch
A Model Context Protocol server that provides web content fetching capabilities with robots.txt checking removed. This server enables LLMs to retrieve and process content from web pages, converting HTML to markdown for easier consumption.
This is a modified version of the original mcp-server-fetch that removes all robots.txt checking, allowing unrestricted access to web content.
This server can access local/internal IP addresses and may represent a security risk. Exercise caution when using this MCP server to ensure this does not expose any sensitive data. Additionally, this version ignores robots.txt restrictions which may violate some websites' access policies.
The fetch tool will truncate the response, but by using the start_index
argument, you can specify where to start the content extraction. This lets models read a webpage in chunks, until they find the information they need.
Available Tools
fetch
- Fetches a URL from the internet and extracts its contents as markdown.url
(string, required): URL to fetchmax_length
(integer, optional): Maximum number of characters to return (default: 5000)start_index
(integer, optional): Start content from this character index (default: 0)raw
(boolean, optional): Get raw content without markdown conversion (default: false)
Available Prompts
fetch
Fetch a URL and extract its contents as markdown
Arguments:
url
(string, required): URL to fetch
Installation and Usage
Local Development Setup
Clone or download the source code:
git clone https://github.com/LangGPT/mcp-fetch.git cd mcp-fetchInstall dependencies using uv:
uv syncTest the server:
uv run python -m mcp_fetch --help
Using with Claude Desktop (Local Source)
Create Claude Desktop configuration:
{ "mcpServers": { "mcp-fetch": { "command": "uv", "args": [ "run", "--directory", "/path/to/your/mcp-fetch", "python", "-m", "mcp_fetch" ] } } }Add configuration to Claude Desktop:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.json
Windows:
%APPDATA%/Claude/claude_desktop_config.json
Linux:
~/.config/Claude/claude_desktop_config.json
Restart Claude Desktop to load the new server.
Using with VS Code (Local Source)
Add to your VS Code settings or .vscode/mcp.json
:
Installation via Package Manager
Using uv (recommended)
When using uv
no specific installation is needed. We will
use uvx
to directly run mcp-fetch:
Using pip
After installation, run it as:
Package Manager Configuration
Claude Desktop with uvx
VS Code with uvx
Development
Setting up Development Environment
Install development dependencies:
uv sync --devRun linting and type checking:
uv run ruff check uv run pyrightBuild the package:
uv build
Testing
Test the server locally:
Use the MCP inspector for debugging:
Making Changes
Edit the source code in
src/mcp_fetch/
Test your changes with
uv run python -m mcp_fetch
Update version in
pyproject.toml
if neededRun tests and linting
Publishing
Publishing to PyPI
Build the package:
uv buildPublish to PyPI:
uv publishOr using twine:
pip install twine twine upload dist/*
Publishing to GitHub
Initialize git repository (if not already done):
git init git branch -m mainAdd and commit files:
git add . git commit -m "Initial commit: MCP Web Fetch server without robots.txt checking"Create GitHub repository and push:
# Create repository on GitHub first, then: git remote add origin https://github.com/LangGPT/mcp-fetch.git git push -u origin mainCreate a release on GitHub:
Go to your repository on GitHub
Click "Releases" → "Create a new release"
Tag version:
v0.6.3
Release title:
v0.6.3 - MCP Fetch
Describe your changes
Publish release
Building Docker Image
Customization
robots.txt
This version has robots.txt checking completely removed. All web requests will proceed regardless of robots.txt restrictions.
User-agent
By default, depending on if the request came from the model (via a tool), or was user initiated (via a prompt), the server will use either the user-agent:
or:
This can be customized by adding the argument --user-agent=YourUserAgent
to the args
list in the configuration.
Proxy
The server can be configured to use a proxy by using the --proxy-url
argument.
Debugging
You can use the MCP inspector to debug the server:
For local development:
For uvx installations:
Contributing
We encourage contributions to help expand and improve mcp-fetch. Whether you want to add new tools, enhance existing functionality, or improve documentation, your input is valuable.
License
mcp-fetch is licensed under the MIT License. This means you are free to use, modify, and distribute the software, subject to the terms and conditions of the MIT License. For more details, please see the LICENSE file in the project repository.
Tools
A Model Context Protocol server that allows LLMs to retrieve and convert web content to markdown without robots.txt restrictions.
Related MCP Servers
- -securityFlicense-qualityA Model Context Protocol server that enables LLMs to extract and use content from unstructured documents across a wide variety of file formats.Last updated -6
- AsecurityFlicenseAqualityA Model Context Protocol server that enables LLMs to fetch and process web content in multiple formats (HTML, JSON, Markdown, text) with automatic format detection.Last updated -54
- AsecurityAlicenseAqualityA Model Context Protocol server that converts Markdown content to HTML format.Last updated -12,9253MIT License
- AsecurityAlicenseAqualityA Model Context Protocol server that converts various file formats (PDF, PowerPoint, Word, Excel, Images, etc.) to Markdown to make them accessible to LLMs.Last updated -1MIT License