Skip to main content
Glama
mcp-webfetch

mcp-webfetch

by Mahgal
README.md
# mcp-webfetch

**Give your AI agent the web — as clean Markdown, not raw HTML.** A tiny [Model Context Protocol](https://modelcontextprotocol.io) server that fetches a page and returns readable Markdown plus its link list, in ~600 lines of Python with zero required dependencies.

![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)
![License: MIT](https://img.shields.io/badge/license-MIT-green)
![MCP](https://img.shields.io/badge/MCP-stdio-orange)
![Dependencies](https://img.shields.io/badge/dependencies-0%20required-brightgreen)
![Tests](https://img.shields.io/badge/tests-20%20passing-brightgreen)

## Why

Agents fed raw HTML waste context on tags, scripts and navigation chrome. `mcp-webfetch` does one job well: turn a URL into the part a model actually wants to read — title, headings, paragraphs, emphasis, code and links — and hand back the page's links when the agent wants to crawl onward.

## Features

- **`fetch_url(url)`** — fetch a page and return clean Markdown (title as `#`, headings, lists, blockquotes, code blocks, `[text](href)` links with relative URLs resolved).
- **`extract_links(url)`** — return the page's links as a JSON array: de-duplicated, fragments stripped, only `http`/`https` kept.
- **Hard limits built in** — 10 s timeout, 1 MB response cap, non-HTML content types rejected with a clear error. No runaway fetches burning agent context.
- **Zero required dependencies** — fetching and HTML→Markdown conversion are standard-library only (`urllib` + `html.parser`). No readability/BeautifulSoup/html2text.
- **Official SDK when available, honest fallback when not** — if the [`mcp` Python SDK](https://pypi.org/project/mcp/) is installed it serves the tools (both the v1 `FastMCP` and v2 `MCPServer` APIs). Otherwise a built-in minimal JSON-RPC-over-stdio implementation speaks the same MCP subset (`initialize`, `tools/list`, `tools/call`, `ping`). Set `MCP_WEBFETCH_FORCE_STDIO=1` to force the fallback.
- **Tested** — 20 pytest cases against a local HTTP server and fixtures: conversion, link extraction, redirects, timeout, size limit, non-HTML rejection, and a full stdio protocol round-trip.

## Quick start

```bash
git clone https://github.com/Mahgal/mcp-webfetch.git && cd mcp-webfetch
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[sdk]"        # or plain `pip install -e .` — stdlib fallback, no SDK needed
python -m mcp_webfetch         # server runs on stdio; it waits for an MCP client
```

Run the local demo (serves a page on localhost, calls both tools, prints real results — no internet needed):

```bash
pip install -e ".[dev]"
python examples/demo.py
pytest
```

## Use it in an MCP client

### Claude Desktop

Add to `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\`):

```json
{
  "mcpServers": {
    "webfetch": {
      "command": "/absolute/path/to/mcp-webfetch/.venv/bin/python",
      "args": ["-m", "mcp_webfetch"],
      "cwd": "/absolute/path/to/mcp-webfetch"
    }
  }
}
```

### Cursor

Add to `.cursor/mcp.json` in your project (or the global MCP settings):

```json
{
  "mcpServers": {
    "webfetch": {
      "command": "/absolute/path/to/mcp-webfetch/.venv/bin/python",
      "args": ["-m", "mcp_webfetch"],
      "cwd": "/absolute/path/to/mcp-webfetch"
    }
  }
}
```

Restart the client, then ask your agent things like *"Fetch https://example.com and summarize it"* or *"List the links on this page"* — it will call `fetch_url` / `extract_links`.

## Real demo output

Verbatim output of `python examples/demo.py` (run 2026-11-09, Python 3.12, `mcp` SDK 2.3.0; the localhost port is ephemeral and differs per run):

```text
Serving demo page at http://127.0.0.1:40503/sample_page.html

(using the official MCP Python SDK client)

Tools: ['fetch_url', 'extract_links']

--- fetch_url(http://127.0.0.1:40503/sample_page.html) ---
# mcp-webfetch Demo Page

This page is served locally by `examples/demo.py` so the demo needs **no internet access** and touches no live website.

## What the tools do

- `fetch_url` turns this HTML into clean Markdown
- `extract_links` lists the links below, de-duplicated

Read the [MCP specification](https://modelcontextprotocol.io), browse [example.com](https://example.com), or revisit [the spec again](https://modelcontextprotocol.io) (duplicate on purpose).


--- extract_links(http://127.0.0.1:40503/sample_page.html) ---
[
  "https://modelcontextprotocol.io",
  "https://example.com"
]
```

Without the SDK installed, the same demo runs through the built-in JSON-RPC stdio client and prints the same tool results.

## Project layout

```text
mcp_webfetch/
  converter.py     HTML -> Markdown + link extraction (html.parser only)
  fetcher.py       urllib fetching with timeout / size / content-type limits
  tools.py         fetch_url + extract_links, transport-independent
  server.py        entry point: official SDK if present, fallback otherwise
  stdio_server.py  minimal MCP JSON-RPC-over-stdio implementation (stdlib)
tests/             pytest suite (fixtures + local HTTP server, no network)
examples/          demo.py + sample_page.html — local end-to-end demo
```

## Design notes & limits

- Conversion is intentionally simple: semantic HTML in, Markdown out. It does not run JavaScript, so JS-rendered pages return whatever the server-sent HTML contains.
- Tables are flattened to text lines rather than Markdown tables; images keep `![alt](src)` form.
- Only `http://` and `https://` URLs are accepted; `file:`, `ftp:` etc. are refused.
- Errors (timeout, HTTP status, non-HTML, oversized body) come back to the agent as MCP tool errors with a human-readable message.

## License

MIT — see [LICENSE](LICENSE).

TDQS

A3.8/5.0

Scored across 2 tools

Disambiguation4/5

The two tools differ mainly in output shape: fetch_url returns Markdown while extract_links returns a JSON link array. There is mild overlap because fetch_url explicitly preserves links, so an agent could plausibly get links via either tool, but the descriptions make the intended split reasonably clear.

Naming Consistency5/5

Both names follow a strict verb_noun snake_case pattern (fetch_url, extract_links) with no stylistic deviation. The convention is predictable and immediately readable.

Tool Count3/5

Two tools is thin for a general-purpose web-fetch server; a single narrow utility would be fine, but this surface leaves little room for variation in fetch behavior. It is not bloated, just minimal.

Completeness3/5

The core fetch-and-parse loop is covered, but notable gaps exist: no raw HTML/JSON fetch, no custom headers or method options, and no main-content/text-only extraction. Agents hitting non-HTML endpoints or auth-gated pages have no path forward.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues