fetch-mcp
This server gives AI assistants safe HTTP fetching and web-content extraction tools for reading, analyzing, and interacting with web resources.
Perform bare HTTP requests: GET, HEAD, OPTIONS, POST, PUT, PATCH, DELETE with headers, auth, timeouts, size caps, retries, and redirect handling.
Convert web pages to clean markdown or plain text for LLM consumption.
Extract article content in reader mode with title, byline, and word count.
Pull page metadata: title, description, Open Graph, Twitter cards, JSON-LD, icons, feeds, and links classified internal/external.
Parse sitemaps, RSS/Atom feeds, and robots.txt crawl rules.
Built-in SSRF protection blocks private/loopback/link-local targets by default, with an operator-controlled opt-in for private hosts.
Provides tools for fetching and parsing RSS 2.0 and Atom 1.0 feeds into structured entries, including titles, links, dates, authors, summaries, content, and categories.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@fetch-mcp Fetch https://example.com/blog and convert to markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
@yawlabs/fetch-mcp
One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.
A comprehensive HTTP fetch MCP server for AI assistants. Bring-your-own client: runs as a stdio MCP server so any MCP-compatible client (Claude Code, Claude Desktop, Cursor, mcph, …) can fetch web content safely.
What it gives the model
Tool | What it does |
| Bare HTTP requests with headers, auth, timeout, size cap, retry |
| Write-method HTTP with JSON or raw body |
| GET a page and convert to clean markdown (3–8× smaller than raw HTML) |
| GET a page and convert to plain text with block structure preserved |
| Reader-mode extraction — isolates the article body and returns title + markdown |
| Extract |
| Extract every outbound link, resolved to absolute URLs, classified internal/external |
| Parse |
| Parse an RSS 2.0 or Atom 1.0 feed into entries |
| Parse a site's |
Related MCP server: Crawlberg MCP Server
Safety
SSRF protection is on by default. The server refuses requests to:
Loopback (
127.0.0.0/8,::1)RFC1918 private ranges (
10/8,172.16/12,192.168/16)Link-local (
169.254/16,fe80::/10) — including the cloud metadata endpoint169.254.169.254CGNAT (
100.64/10)Unique-local IPv6 (
fc00::/7)Multicast / broadcast
IPv4-mapped IPv6 (
::ffff:0:0/96) re-checked against the IPv4 rules, and likewise the IPv4 embedded in IPv4-compatible (::/96), IPv4-translated (::ffff:0:0:0/96) and 6to4 (2002::/16) addressesTeredo (
2001::/32), NAT64 (64:ff9b::/96,64:ff9b:1::/48), site-local (fec0::/10) and discard-only (100::/64) IPv6Any other IPv6 outside global unicast (
2000::/3), and the non-global blocks inside it (3fff::/20,2001:2::/48,2001:10::/28)The Azure WireServer address
168.63.129.16Non-
http/httpsschemes (file://,gopher://,javascript:, …)Hostname
localhostand any*.localhost
Every redirect hop is re-validated against all of the above -- scheme, literal IP and localhost* -- before it is dialed, so a 302 from a public host to http://127.0.0.1, http://169.254.169.254 or ftp://… is caught. For hostnames, DNS is resolved once per hop, every returned address is checked, and the verified IP is pinned into the HTTP dispatcher so the subsequent TCP connection dials that exact address — closing the DNS-rebinding TOCTOU window. Authorization, Cookie and Proxy-Authorization headers are stripped on cross-origin redirects.
Reaching private hosts (development)
The model chooses tool arguments, and the model is not trusted: a prompt injected through a fetched page can ask for anything. So the per-call allow_private_hosts: true opt-in is refused unless the operator enables it by launching the server with FETCH_MCP_ALLOW_PRIVATE_HOSTS=1 (true/yes/on also work; any unrecognised value is treated as off and named on stderr). With the variable set, a call still has to ask: requests without allow_private_hosts stay guarded.
{
"mcpServers": {
"fetch": {
"command": "npx",
"args": ["-y", "@yawlabs/fetch-mcp@latest"],
"env": { "FETCH_MCP_ALLOW_PRIVATE_HOSTS": "1" }
}
}
}Only set it where the model may legitimately reach your internal network -- a local dev box, not a cloud VM with a metadata endpoint.
Install & run
# One-off (auto-updates each spawn via @latest)
npx -y @yawlabs/fetch-mcp@latest
# Or globally
npm i -g @yawlabs/fetch-mcp
fetch-mcpRequires Node 22.19 or newer when running on Node (the floor of its HTTP client, undici 8; the launcher refuses older Nodes with a clear message instead of crashing). Under oam, which the launcher prefers when installed, the Node version does not matter.
Configure in Claude Code / Claude Desktop
Add to your client's MCP config (usually claude_desktop_config.json or ~/.claude.json):
{
"mcpServers": {
"fetch": {
"command": "npx",
"args": ["-y", "@yawlabs/fetch-mcp@latest"]
}
}
}Or via mcph:
mcph add fetchTool reference
http_get, http_post, http_put, http_patch, http_delete, http_head, http_options
Common parameters:
Field | Type | Default | Meaning |
| string | — | Absolute URL |
| object | — | Custom request headers |
| int |
| Timeout for each attempt, covering DNS, every redirect hop and the body. Retries get a fresh budget; a whole call is capped at 5 minutes. Cancelling the tool call stops the request |
| int |
| Truncate body if larger |
| int |
| Redirect hops allowed |
| int |
| Retry count on 408/425/429/5xx with backoff (honors |
| string |
|
|
|
| — | Injects |
| string | — | Injects |
| bool |
| Opt this call into loopback / private / link-local targets. Refused unless the operator set |
| bool | auto | When unset, auto-detects by response Content-Type (text for text/*, JSON, XML, JS, form-urlencoded; binary otherwise). Set explicitly |
Body-capable tools (POST/PUT/PATCH/DELETE) also take:
Field | Type | Meaning |
| string | Raw request body |
| any | Structured body — encoded as JSON, |
| string | Overrides |
Response shape:
{
ok: boolean;
status: number;
statusText: string;
url: string; // final URL after redirects
headers: Record<string,string>;
body_text?: string;
body_base64?: string; // when decode_text=false
json?: unknown; // auto-parsed when response is application/json
truncated?: boolean; // set when max_bytes hit
redirects?: string[]; // chain of intermediate URLs
duration_ms: number;
error?: string;
}fetch_html_to_markdown
GET the URL, strip scripts/styles/iframes/svg/canvas plus <nav>, <footer>, <aside>, convert to atx-headed markdown with fenced code blocks and dash bullets. Intended for feeding pages into an LLM without blowing the context budget.
fetch_html_to_text
Same fetch, but emits plain text with block-level structure preserved as newlines. Useful when the model doesn't need markdown formatting.
fetch_reader
Isolates the main article body using, in order: <article>, <main>, itemprop="articleBody", common CMS class names (post-content, entry-content, etc.), then <body> as fallback. Returns:
{
url: string; // final URL after redirects
title?: string; // og:title, then <title>, then <h1>
byline?: string; // meta[name=author] / article:author
wordCount: number;
markdown: string; // main content converted to markdown
}fetch_meta
GET a URL and return its head metadata without downloading the full body (caps at 2 MiB by default):
{
url: string;
title?: string;
description?: string;
canonical?: string;
language?: string;
robots?: string;
og: Record<string, string>; // first value per key (og:title, og:image, og:type, ...)
twitter: Record<string, string>; // first value per key
article: Record<string, string>; // first value per key
ogAll: Record<string, string[]>; // only keys that appear > once (e.g. multiple og:image)
twitterAll: Record<string, string[]>;
articleAll: Record<string, string[]>;
icons: Array<{ rel: string; href: string; sizes?: string }>;
feeds: Array<{ href: string; title?: string; type?: string }>; // RSS/Atom
jsonLd: unknown[]; // parsed application/ld+json blocks
}fetch_links
GET a page and return every <a href> with text, resolved to absolute URLs. Respects <base href>. Skips #, javascript:, mailto:, tel:, data:, file:. Each link is classified internal or external vs. the page host. Optional filter/dedupe/limit.
fetch_sitemap
Fetch a sitemap.xml or sitemap-index and return the URL list:
{
sitemaps: string[]; // indexes followed, in order
urlCount: number;
truncated: boolean; // hit max_urls
childSitemaps: string[]; // listed but not fetched (past max_depth or max_sitemaps)
warnings: Array<{ url: string; error: string }>;
urls: Array<{
loc: string;
lastmod?: string;
changefreq?: string;
priority?: number;
}>;
}Gzipped .xml.gz sitemaps are auto-decompressed. max_depth controls how many levels of sitemap-index to follow (default 1). Setting max_depth: 0 on a sitemap-index returns the index's childSitemaps list without fetching any child (useful to discover structure cheaply). Partial failures — one child sitemap 500s while others succeed — are returned under warnings rather than aborting the whole call. max_sitemaps (default 50, max 1000) caps how many sitemap documents one call fetches, index included; children past it are listed under childSitemaps with a warning. The whole call is capped at 5 minutes. max_bytes (default 20 MiB) applies to each sitemap, and to the decompressed size of a gzipped one; a sitemap larger than that is an error (a warning for a child) rather than a silently partial URL list.
fetch_feed
Parse an RSS 2.0 or Atom 1.0 feed:
{
kind: "rss" | "atom" | "unknown";
title?: string;
description?: string;
link?: string;
updated?: string;
entryCount: number;
truncated: boolean; // hit limit
entries: Array<{
title?: string;
link?: string;
id?: string;
published?: string;
updated?: string;
author?: string;
summary?: string;
content?: string;
categories?: string[];
}>;
}fetch_robots
Fetches <origin>/robots.txt, parses it, and returns:
{
robotsUrl: string;
status: number;
userAgent: string;
path: string;
allowed: boolean;
matchedRule: string | null; // the longest-match Allow/Disallow that decided it
crawlDelay: number | null; // from the matched group
sitemaps: string[]; // top-level Sitemap: declarations
rawRobotsText: string; // first 512KB
note?: string; // only when robots.txt 404s
}A /robots.txt that returns 404 means there are no rules, so the verdict is allowed: true. The response keeps the same shape: status: 404, matchedRule: null, crawlDelay: null, sitemaps: [], rawRobotsText: "", plus the note. Any other non-2xx status is returned as an error.
The parser follows Google's rules: longest match wins, * is a wildcard segment, $ anchors the end of the path, specific user-agent group beats the * wildcard group when the UA matches (comparison uses the length of the actually-matched agent token, not the group's first agent). Allow beats Disallow on equal-length ties.
Development
npm install
npm run build
npm test # vitest
npm run lint # biome
npm run typecheckTests spin up a local loopback HTTP server on 127.0.0.1:0 to exercise the real request/response path — no mocking of HTTP. SSRF tests verify that the default-deny still applies to that local server unless the operator gate is on and the request opts into allow_private_hosts.
License
MIT © Yaw Labs
Links
Yaw Labs: https://yaw.sh
Available Tools
15 toolsfetch_feedARead-onlyIdempotent
Fetch and parse an RSS 2.0 or Atom 1.0 feed. Returns feed-level metadata (title, description, link, updated) plus a list of entries (title, link, id, published, updated, author, summary, content, contentType, categories). Auto-detects RSS vs Atom.
Inputs: url (required), limit, max_bytes, timeout_ms, +3 more.
fetch_html_to_markdownARead-onlyIdempotent
GET a URL, decode the HTML, and convert to clean markdown (headings, lists, links, code fences). Scripts, styles, iframes, nav, footer, and aside elements are stripped. Intended for feeding web pages into an LLM cheaply -- markdown is usually 3-8x smaller than raw HTML. Follows redirects, respects size/timeout limits, and blocks private-host requests by default.
Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.
fetch_html_to_textARead-onlyIdempotent
GET a URL, decode the HTML, and return plain text with block-level structure preserved as newlines. Scripts, styles, and comments stripped; HTML entities decoded. Lighter than markdown when you only need the reading content.
Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.
fetch_linksARead-onlyIdempotent
Extract every outbound link from an HTML page, resolved to absolute URLs. Each entry includes href, anchor text, optional rel/title, and an internal/external classification (bare-domain and www. treated as the same host). Only http and https links are returned: anchors (#) and every other scheme (javascript:, mailto:, tel:, data:, file:, ...) are skipped. Respects .
Inputs: url (required), limit, dedupe, filter, +5 more.
fetch_metaARead-onlyIdempotent
GET a URL and extract its head metadata: title, description, canonical, language, robots directive, Open Graph / Twitter Card / article: properties, icon links, RSS/Atom feed links, and any JSON-LD (schema.org) blocks. Keys that appear more than once (e.g. multiple og:image tags) are additionally returned via ogAll/twitterAll/articleAll arrays. Lists are capped: at most 100 icons, feeds, JSON-LD blocks and values per *All key, and 1000 keys per og/twitter/article; a JSON-LD block nested deeper than 64 levels is dropped. When anything is cut, truncated names the collections affected. Ideal for previewing a page before fully reading it.
Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.
fetch_readerARead-onlyIdempotent
GET a URL, locate the main article body (prefers , , itemprop=articleBody, or known CMS class names; falls back to ), strip navigation/footer/aside chrome, and convert to clean markdown. Returns { title, byline, markdown, wordCount }. Optimized for feeding long-form articles into an LLM without header/footer/sidebar noise.
Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.
fetch_robotsARead-onlyIdempotent
Fetch and parse the robots.txt for a given origin, then tell the caller whether a target URL is crawlable by a given user-agent. Follows the Google-style longest-match rule with Allow-wins-on-tie. Returns the raw robots.txt, parsed groups, sitemap references, and the allow/deny verdict with the matching rule.
Inputs: url (required), timeout_ms, user_agent, max_redirects, +1 more.
fetch_sitemapARead-onlyIdempotent
Fetch a sitemap.xml (or sitemap-index) and return the contained URLs with their lastmod / changefreq / priority. Follows sitemap-index chaining up to max_depth levels. Gzipped .xml.gz payloads are auto-decompressed. A single sitemap document over 16 MiB once decoded is refused (a 50,000-URL sitemap is ~11 MiB). Partial failures (one child sitemap 500s while others work) are returned under 'warnings' without aborting the whole request. SSRF-protected by default.
Inputs: url (required), max_urls, max_bytes, max_depth, +5 more.
http_deleteADestructiveIdempotent
Perform an HTTP DELETE. Idempotent (per spec): a repeated DELETE on an already-deleted resource typically returns 404/410.
Inputs: url (required), body, headers, retries, +10 more.
http_getARead-onlyIdempotent
Perform an HTTP GET. Returns status, headers, and body. Automatically parses JSON when the server responds with application/json. Follows redirects (each hop re-validated against SSRF rules). Refuses URLs that resolve to private/loopback/link-local addresses unless the operator enabled FETCH_MCP_ALLOW_PRIVATE_HOSTS and the call sets allow_private_hosts.
Inputs: url (required), headers, retries, max_bytes, +7 more.
http_headARead-onlyIdempotent
Perform an HTTP HEAD. Returns status + headers with no body. Useful for checking a resource exists, getting its size (Content-Length), or polling for changes cheaply.
Inputs: url (required), headers, retries, max_bytes, +7 more.
http_optionsARead-onlyIdempotent
Perform an HTTP OPTIONS. Returns the server's supported methods and CORS policy for a resource. Helpful for API discovery.
Inputs: url (required), headers, retries, max_bytes, +7 more.
http_patchADestructive
Perform an HTTP PATCH. Use for partial updates. Spec-wise NOT guaranteed idempotent — depends on the server's patch semantics.
Inputs: url (required), body, headers, retries, +10 more.
http_postADestructive
Perform an HTTP POST. Body can be given as a raw string (body) or as a JSON value (body_json — auto-sets content-type to application/json). NOT idempotent: calling twice submits twice.
Inputs: url (required), body, headers, retries, +10 more.
http_putADestructiveIdempotent
Perform an HTTP PUT. Use for full-resource replacement. Idempotent: the same PUT applied twice leaves the resource in the same state.
Inputs: url (required), body, headers, retries, +10 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 15 tools
The seven http_* verb tools are unambiguously separated by method, and fetch_meta/fetch_links/fetch_feed/fetch_sitemap/fetch_robots each target a distinct artifact. The only real overlap is the trio fetch_html_to_markdown, fetch_html_to_text, and fetch_reader, which all GET an HTML page and return readable content — the descriptions differentiate them (output format vs. article-body extraction), but an agent may still deliberate over which to pick.
All names are snake_case and follow a clean two-prefix scheme: http_<method> for raw requests and fetch_<artifact> for extraction helpers. The pattern is predictable and self-documenting across all 15 tools.
15 tools sits at the top of the well-scoped range and each one maps to a distinct HTTP method or extraction target. Nothing appears redundant filler, and the verb set (GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS) plus a small extraction suite is proportionate to a fetch server's purpose.
HTTP verb coverage is essentially complete (TRACE omitted, which is fine) and the parsing surface covers HTML, metadata, links, feeds, sitemaps, and robots.txt. Minor gaps: no explicit binary/raw-byte download tool, no form-urlencoded or multipart POST helper, and no cookie/session handling for multi-step flows.
Maintenance
Related MCP Connectors
Read any web page as clean Markdown for AI agents: fetch, search, metadata, links. SSRF-safe.
Unblocking and fresh web data for agents: URL to Markdown, YouTube, Maps, Amazon, jobs. Pay per call
Read a URL as clean markdown, screenshot a website, url to PDF. Web access for agents, no signup.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Related MCP Servers
- AlicenseBqualityDmaintenanceEnables fetching and converting web content to markdown with built-in prompt injection safeguards that detect and block malicious content attempting to manipulate the LLM.12MIT
- AlicenseNot gradedqualityAmaintenanceEnables AI agents to crawl and scrape websites, converting HTML to clean Markdown and structured metadata with support for JavaScript rendering, bot evasion, and SSRF protection.197MIT
- AlicenseAqualityCmaintenanceEnables AI agents to read RSS/Atom feeds, extract web page content as text or Markdown, search the web, and batch-fetch multiple URLs without building custom crawlers.5MIT
- AlicenseAqualityBmaintenanceEnables AI models to fetch real web pages while automatically bypassing anti-bot protections, login walls, and JS rendering, then returns clean, token-efficient article content.21MIT