mcp-webfetch
# mcp-webfetch
**Give your AI agent the web — as clean Markdown, not raw HTML.** A tiny [Model Context Protocol](https://modelcontextprotocol.io) server that fetches a page and returns readable Markdown plus its link list, in ~600 lines of Python with zero required dependencies.





## Why
Agents fed raw HTML waste context on tags, scripts and navigation chrome. `mcp-webfetch` does one job well: turn a URL into the part a model actually wants to read — title, headings, paragraphs, emphasis, code and links — and hand back the page's links when the agent wants to crawl onward.
## Features
- **`fetch_url(url)`** — fetch a page and return clean Markdown (title as `#`, headings, lists, blockquotes, code blocks, `[text](href)` links with relative URLs resolved).
- **`extract_links(url)`** — return the page's links as a JSON array: de-duplicated, fragments stripped, only `http`/`https` kept.
- **Hard limits built in** — 10 s timeout, 1 MB response cap, non-HTML content types rejected with a clear error. No runaway fetches burning agent context.
- **Zero required dependencies** — fetching and HTML→Markdown conversion are standard-library only (`urllib` + `html.parser`). No readability/BeautifulSoup/html2text.
- **Official SDK when available, honest fallback when not** — if the [`mcp` Python SDK](https://pypi.org/project/mcp/) is installed it serves the tools (both the v1 `FastMCP` and v2 `MCPServer` APIs). Otherwise a built-in minimal JSON-RPC-over-stdio implementation speaks the same MCP subset (`initialize`, `tools/list`, `tools/call`, `ping`). Set `MCP_WEBFETCH_FORCE_STDIO=1` to force the fallback.
- **Tested** — 20 pytest cases against a local HTTP server and fixtures: conversion, link extraction, redirects, timeout, size limit, non-HTML rejection, and a full stdio protocol round-trip.
## Quick start
```bash
git clone https://github.com/Mahgal/mcp-webfetch.git && cd mcp-webfetch
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[sdk]" # or plain `pip install -e .` — stdlib fallback, no SDK needed
python -m mcp_webfetch # server runs on stdio; it waits for an MCP client
```
Run the local demo (serves a page on localhost, calls both tools, prints real results — no internet needed):
```bash
pip install -e ".[dev]"
python examples/demo.py
pytest
```
## Use it in an MCP client
### Claude Desktop
Add to `claude_desktop_config.json` (macOS: `~/Library/Application Support/Claude/`, Windows: `%APPDATA%\Claude\`):
```json
{
"mcpServers": {
"webfetch": {
"command": "/absolute/path/to/mcp-webfetch/.venv/bin/python",
"args": ["-m", "mcp_webfetch"],
"cwd": "/absolute/path/to/mcp-webfetch"
}
}
}
```
### Cursor
Add to `.cursor/mcp.json` in your project (or the global MCP settings):
```json
{
"mcpServers": {
"webfetch": {
"command": "/absolute/path/to/mcp-webfetch/.venv/bin/python",
"args": ["-m", "mcp_webfetch"],
"cwd": "/absolute/path/to/mcp-webfetch"
}
}
}
```
Restart the client, then ask your agent things like *"Fetch https://example.com and summarize it"* or *"List the links on this page"* — it will call `fetch_url` / `extract_links`.
## Real demo output
Verbatim output of `python examples/demo.py` (run 2026-11-09, Python 3.12, `mcp` SDK 2.3.0; the localhost port is ephemeral and differs per run):
```text
Serving demo page at http://127.0.0.1:40503/sample_page.html
(using the official MCP Python SDK client)
Tools: ['fetch_url', 'extract_links']
--- fetch_url(http://127.0.0.1:40503/sample_page.html) ---
# mcp-webfetch Demo Page
This page is served locally by `examples/demo.py` so the demo needs **no internet access** and touches no live website.
## What the tools do
- `fetch_url` turns this HTML into clean Markdown
- `extract_links` lists the links below, de-duplicated
Read the [MCP specification](https://modelcontextprotocol.io), browse [example.com](https://example.com), or revisit [the spec again](https://modelcontextprotocol.io) (duplicate on purpose).
--- extract_links(http://127.0.0.1:40503/sample_page.html) ---
[
"https://modelcontextprotocol.io",
"https://example.com"
]
```
Without the SDK installed, the same demo runs through the built-in JSON-RPC stdio client and prints the same tool results.
## Project layout
```text
mcp_webfetch/
converter.py HTML -> Markdown + link extraction (html.parser only)
fetcher.py urllib fetching with timeout / size / content-type limits
tools.py fetch_url + extract_links, transport-independent
server.py entry point: official SDK if present, fallback otherwise
stdio_server.py minimal MCP JSON-RPC-over-stdio implementation (stdlib)
tests/ pytest suite (fixtures + local HTTP server, no network)
examples/ demo.py + sample_page.html — local end-to-end demo
```
## Design notes & limits
- Conversion is intentionally simple: semantic HTML in, Markdown out. It does not run JavaScript, so JS-rendered pages return whatever the server-sent HTML contains.
- Tables are flattened to text lines rather than Markdown tables; images keep `` form.
- Only `http://` and `https://` URLs are accepted; `file:`, `ftp:` etc. are refused.
- Errors (timeout, HTTP status, non-HTML, oversized body) come back to the agent as MCP tool errors with a human-readable message.
## License
MIT — see [LICENSE](LICENSE).
TDQS
Scored across 2 tools
The two tools differ mainly in output shape: fetch_url returns Markdown while extract_links returns a JSON link array. There is mild overlap because fetch_url explicitly preserves links, so an agent could plausibly get links via either tool, but the descriptions make the intended split reasonably clear.
Both names follow a strict verb_noun snake_case pattern (fetch_url, extract_links) with no stylistic deviation. The convention is predictable and immediately readable.
Two tools is thin for a general-purpose web-fetch server; a single narrow utility would be fine, but this surface leaves little room for variation in fetch behavior. It is not bloated, just minimal.
The core fetch-and-parse loop is covered, but notable gaps exist: no raw HTML/JSON fetch, no custom headers or method options, and no main-content/text-only extraction. Agents hitting non-HTML endpoints or auth-gated pages have no path forward.