Skip to main content
Glama

@yawlabs/fetch-mcp

Add to Yaw MCP

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

A comprehensive HTTP fetch MCP server for AI assistants. Bring-your-own client: runs as a stdio MCP server so any MCP-compatible client (Claude Code, Claude Desktop, Cursor, mcph, …) can fetch web content safely.

What it gives the model

Tool

What it does

http_get / http_head / http_options

Bare HTTP requests with headers, auth, timeout, size cap, retry

http_post / http_put / http_patch / http_delete

Write-method HTTP with JSON or raw body

fetch_html_to_markdown

GET a page and convert to clean markdown (3–8× smaller than raw HTML)

fetch_html_to_text

GET a page and convert to plain text with block structure preserved

fetch_reader

Reader-mode extraction — isolates the article body and returns title + markdown

fetch_meta

Extract <head> metadata: title, description, OpenGraph, Twitter cards, JSON-LD, feeds, icons

fetch_links

Extract every outbound link, resolved to absolute URLs, classified internal/external

fetch_sitemap

Parse sitemap.xml (including gzipped and sitemap-index chaining)

fetch_feed

Parse an RSS 2.0 or Atom 1.0 feed into entries

fetch_robots

Parse a site's robots.txt, return the verdict for a given path & user-agent

Related MCP server: Crawlberg MCP Server

Safety

SSRF protection is on by default. The server refuses requests to:

  • Loopback (127.0.0.0/8, ::1)

  • RFC1918 private ranges (10/8, 172.16/12, 192.168/16)

  • Link-local (169.254/16, fe80::/10) — including the cloud metadata endpoint 169.254.169.254

  • CGNAT (100.64/10)

  • Unique-local IPv6 (fc00::/7)

  • Multicast / broadcast

  • IPv4-mapped IPv6 (::ffff:0:0/96) re-checked against the IPv4 rules, and likewise the IPv4 embedded in IPv4-compatible (::/96), IPv4-translated (::ffff:0:0:0/96) and 6to4 (2002::/16) addresses

  • Teredo (2001::/32), NAT64 (64:ff9b::/96, 64:ff9b:1::/48), site-local (fec0::/10) and discard-only (100::/64) IPv6

  • Any other IPv6 outside global unicast (2000::/3), and the non-global blocks inside it (3fff::/20, 2001:2::/48, 2001:10::/28)

  • The Azure WireServer address 168.63.129.16

  • Non-http/https schemes (file://, gopher://, javascript:, …)

  • Hostname localhost and any *.localhost

Every redirect hop is re-validated against all of the above -- scheme, literal IP and localhost* -- before it is dialed, so a 302 from a public host to http://127.0.0.1, http://169.254.169.254 or ftp://… is caught. For hostnames, DNS is resolved once per hop, every returned address is checked, and the verified IP is pinned into the HTTP dispatcher so the subsequent TCP connection dials that exact address — closing the DNS-rebinding TOCTOU window. Authorization, Cookie and Proxy-Authorization headers are stripped on cross-origin redirects.

Reaching private hosts (development)

The model chooses tool arguments, and the model is not trusted: a prompt injected through a fetched page can ask for anything. So the per-call allow_private_hosts: true opt-in is refused unless the operator enables it by launching the server with FETCH_MCP_ALLOW_PRIVATE_HOSTS=1 (true/yes/on also work; any unrecognised value is treated as off and named on stderr). With the variable set, a call still has to ask: requests without allow_private_hosts stay guarded.

{
  "mcpServers": {
    "fetch": {
      "command": "npx",
      "args": ["-y", "@yawlabs/fetch-mcp@latest"],
      "env": { "FETCH_MCP_ALLOW_PRIVATE_HOSTS": "1" }
    }
  }
}

Only set it where the model may legitimately reach your internal network -- a local dev box, not a cloud VM with a metadata endpoint.

Install & run

# One-off (auto-updates each spawn via @latest)
npx -y @yawlabs/fetch-mcp@latest

# Or globally
npm i -g @yawlabs/fetch-mcp
fetch-mcp

Requires Node 22.19 or newer when running on Node (the floor of its HTTP client, undici 8; the launcher refuses older Nodes with a clear message instead of crashing). Under oam, which the launcher prefers when installed, the Node version does not matter.

Configure in Claude Code / Claude Desktop

Add to your client's MCP config (usually claude_desktop_config.json or ~/.claude.json):

{
  "mcpServers": {
    "fetch": {
      "command": "npx",
      "args": ["-y", "@yawlabs/fetch-mcp@latest"]
    }
  }
}

Or via mcph:

mcph add fetch

Tool reference

http_get, http_post, http_put, http_patch, http_delete, http_head, http_options

Common parameters:

Field

Type

Default

Meaning

url

string

—

Absolute URL

headers

object

—

Custom request headers

timeout_ms

int

10000

Timeout for each attempt, covering DNS, every redirect hop and the body. Retries get a fresh budget; a whole call is capped at 5 minutes. Cancelling the tool call stops the request

max_bytes

int

5242880 (5 MiB)

Truncate body if larger

max_redirects

int

5

Redirect hops allowed

retries

int

0

Retry count on 408/425/429/5xx with backoff (honors Retry-After)

user_agent

string

@yawlabs/fetch-mcp/<v>

User-Agent override

basic_auth

{username,password}

—

Injects Authorization: Basic …

bearer_token

string

—

Injects Authorization: Bearer …

allow_private_hosts

bool

false

Opt this call into loopback / private / link-local targets. Refused unless the operator set FETCH_MCP_ALLOW_PRIVATE_HOSTS=1 (details)

decode_text

bool

auto

When unset, auto-detects by response Content-Type (text for text/*, JSON, XML, JS, form-urlencoded; binary otherwise). Set explicitly true to force text decoding, false to force base64 in body_base64.

Body-capable tools (POST/PUT/PATCH/DELETE) also take:

Field

Type

Meaning

body

string

Raw request body

body_json

any

Structured body — encoded as JSON, Content-Type: application/json set automatically

content_type

string

Overrides Content-Type

Response shape:

{
  ok: boolean;
  status: number;
  statusText: string;
  url: string;              // final URL after redirects
  headers: Record<string,string>;
  body_text?: string;
  body_base64?: string;     // when decode_text=false
  json?: unknown;           // auto-parsed when response is application/json
  truncated?: boolean;      // set when max_bytes hit
  redirects?: string[];     // chain of intermediate URLs
  duration_ms: number;
  error?: string;
}

fetch_html_to_markdown

GET the URL, strip scripts/styles/iframes/svg/canvas plus <nav>, <footer>, <aside>, convert to atx-headed markdown with fenced code blocks and dash bullets. Intended for feeding pages into an LLM without blowing the context budget.

fetch_html_to_text

Same fetch, but emits plain text with block-level structure preserved as newlines. Useful when the model doesn't need markdown formatting.

fetch_reader

Isolates the main article body using, in order: <article>, <main>, itemprop="articleBody", common CMS class names (post-content, entry-content, etc.), then <body> as fallback. Returns:

{
  url: string;          // final URL after redirects
  title?: string;       // og:title, then <title>, then <h1>
  byline?: string;      // meta[name=author] / article:author
  wordCount: number;
  markdown: string;     // main content converted to markdown
}

fetch_meta

GET a URL and return its head metadata without downloading the full body (caps at 2 MiB by default):

{
  url: string;
  title?: string;
  description?: string;
  canonical?: string;
  language?: string;
  robots?: string;
  og:      Record<string, string>;       // first value per key (og:title, og:image, og:type, ...)
  twitter: Record<string, string>;       // first value per key
  article: Record<string, string>;       // first value per key
  ogAll:      Record<string, string[]>;  // only keys that appear > once (e.g. multiple og:image)
  twitterAll: Record<string, string[]>;
  articleAll: Record<string, string[]>;
  icons:   Array<{ rel: string; href: string; sizes?: string }>;
  feeds:   Array<{ href: string; title?: string; type?: string }>;   // RSS/Atom
  jsonLd:  unknown[];                    // parsed application/ld+json blocks
}

GET a page and return every <a href> with text, resolved to absolute URLs. Respects <base href>. Skips #, javascript:, mailto:, tel:, data:, file:. Each link is classified internal or external vs. the page host. Optional filter/dedupe/limit.

fetch_sitemap

Fetch a sitemap.xml or sitemap-index and return the URL list:

{
  sitemaps: string[];       // indexes followed, in order
  urlCount: number;
  truncated: boolean;       // hit max_urls
  childSitemaps: string[];  // listed but not fetched (past max_depth or max_sitemaps)
  warnings: Array<{ url: string; error: string }>;
  urls: Array<{
    loc: string;
    lastmod?: string;
    changefreq?: string;
    priority?: number;
  }>;
}

Gzipped .xml.gz sitemaps are auto-decompressed. max_depth controls how many levels of sitemap-index to follow (default 1). Setting max_depth: 0 on a sitemap-index returns the index's childSitemaps list without fetching any child (useful to discover structure cheaply). Partial failures — one child sitemap 500s while others succeed — are returned under warnings rather than aborting the whole call. max_sitemaps (default 50, max 1000) caps how many sitemap documents one call fetches, index included; children past it are listed under childSitemaps with a warning. The whole call is capped at 5 minutes. max_bytes (default 20 MiB) applies to each sitemap, and to the decompressed size of a gzipped one; a sitemap larger than that is an error (a warning for a child) rather than a silently partial URL list.

fetch_feed

Parse an RSS 2.0 or Atom 1.0 feed:

{
  kind: "rss" | "atom" | "unknown";
  title?: string;
  description?: string;
  link?: string;
  updated?: string;
  entryCount: number;
  truncated: boolean;       // hit limit
  entries: Array<{
    title?: string;
    link?: string;
    id?: string;
    published?: string;
    updated?: string;
    author?: string;
    summary?: string;
    content?: string;
    categories?: string[];
  }>;
}

fetch_robots

Fetches <origin>/robots.txt, parses it, and returns:

{
  robotsUrl: string;
  status: number;
  userAgent: string;
  path: string;
  allowed: boolean;
  matchedRule: string | null;  // the longest-match Allow/Disallow that decided it
  crawlDelay: number | null;   // from the matched group
  sitemaps: string[];          // top-level Sitemap: declarations
  rawRobotsText: string;       // first 512KB
  note?: string;               // only when robots.txt 404s
}

A /robots.txt that returns 404 means there are no rules, so the verdict is allowed: true. The response keeps the same shape: status: 404, matchedRule: null, crawlDelay: null, sitemaps: [], rawRobotsText: "", plus the note. Any other non-2xx status is returned as an error.

The parser follows Google's rules: longest match wins, * is a wildcard segment, $ anchors the end of the path, specific user-agent group beats the * wildcard group when the UA matches (comparison uses the length of the actually-matched agent token, not the group's first agent). Allow beats Disallow on equal-length ties.

Development

npm install
npm run build
npm test         # vitest
npm run lint     # biome
npm run typecheck

Tests spin up a local loopback HTTP server on 127.0.0.1:0 to exercise the real request/response path — no mocking of HTTP. SSRF tests verify that the default-deny still applies to that local server unless the operator gate is on and the request opts into allow_private_hosts.

License

MIT © Yaw Labs

Follow @YawLabs on X

Available Tools

15 tools
fetch_feedA
Read-onlyIdempotent

Fetch and parse an RSS 2.0 or Atom 1.0 feed. Returns feed-level metadata (title, description, link, updated) plus a list of entries (title, link, id, published, updated, author, summary, content, contentType, categories). Auto-detects RSS vs Atom.

Inputs: url (required), limit, max_bytes, timeout_ms, +3 more.

fetch_html_to_markdownA
Read-onlyIdempotent

GET a URL, decode the HTML, and convert to clean markdown (headings, lists, links, code fences). Scripts, styles, iframes, nav, footer, and aside elements are stripped. Intended for feeding web pages into an LLM cheaply -- markdown is usually 3-8x smaller than raw HTML. Follows redirects, respects size/timeout limits, and blocks private-host requests by default.

Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.

fetch_html_to_textA
Read-onlyIdempotent

GET a URL, decode the HTML, and return plain text with block-level structure preserved as newlines. Scripts, styles, and comments stripped; HTML entities decoded. Lighter than markdown when you only need the reading content.

Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.

fetch_metaA
Read-onlyIdempotent

GET a URL and extract its head metadata: title, description, canonical, language, robots directive, Open Graph / Twitter Card / article: properties, icon links, RSS/Atom feed links, and any JSON-LD (schema.org) blocks. Keys that appear more than once (e.g. multiple og:image tags) are additionally returned via ogAll/twitterAll/articleAll arrays. Lists are capped: at most 100 icons, feeds, JSON-LD blocks and values per *All key, and 1000 keys per og/twitter/article; a JSON-LD block nested deeper than 64 levels is dropped. When anything is cut, truncated names the collections affected. Ideal for previewing a page before fully reading it.

Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.

fetch_readerA
Read-onlyIdempotent

GET a URL, locate the main article body (prefers , , itemprop=articleBody, or known CMS class names; falls back to ), strip navigation/footer/aside chrome, and convert to clean markdown. Returns { title, byline, markdown, wordCount }. Optimized for feeding long-form articles into an LLM without header/footer/sidebar noise.

Inputs: url (required), max_bytes, timeout_ms, user_agent, +2 more.

fetch_robotsA
Read-onlyIdempotent

Fetch and parse the robots.txt for a given origin, then tell the caller whether a target URL is crawlable by a given user-agent. Follows the Google-style longest-match rule with Allow-wins-on-tie. Returns the raw robots.txt, parsed groups, sitemap references, and the allow/deny verdict with the matching rule.

Inputs: url (required), timeout_ms, user_agent, max_redirects, +1 more.

fetch_sitemapA
Read-onlyIdempotent

Fetch a sitemap.xml (or sitemap-index) and return the contained URLs with their lastmod / changefreq / priority. Follows sitemap-index chaining up to max_depth levels. Gzipped .xml.gz payloads are auto-decompressed. A single sitemap document over 16 MiB once decoded is refused (a 50,000-URL sitemap is ~11 MiB). Partial failures (one child sitemap 500s while others work) are returned under 'warnings' without aborting the whole request. SSRF-protected by default.

Inputs: url (required), max_urls, max_bytes, max_depth, +5 more.

http_deleteA
DestructiveIdempotent

Perform an HTTP DELETE. Idempotent (per spec): a repeated DELETE on an already-deleted resource typically returns 404/410.

Inputs: url (required), body, headers, retries, +10 more.

http_getA
Read-onlyIdempotent

Perform an HTTP GET. Returns status, headers, and body. Automatically parses JSON when the server responds with application/json. Follows redirects (each hop re-validated against SSRF rules). Refuses URLs that resolve to private/loopback/link-local addresses unless the operator enabled FETCH_MCP_ALLOW_PRIVATE_HOSTS and the call sets allow_private_hosts.

Inputs: url (required), headers, retries, max_bytes, +7 more.

http_headA
Read-onlyIdempotent

Perform an HTTP HEAD. Returns status + headers with no body. Useful for checking a resource exists, getting its size (Content-Length), or polling for changes cheaply.

Inputs: url (required), headers, retries, max_bytes, +7 more.

http_optionsA
Read-onlyIdempotent

Perform an HTTP OPTIONS. Returns the server's supported methods and CORS policy for a resource. Helpful for API discovery.

Inputs: url (required), headers, retries, max_bytes, +7 more.

http_patchA
Destructive

Perform an HTTP PATCH. Use for partial updates. Spec-wise NOT guaranteed idempotent — depends on the server's patch semantics.

Inputs: url (required), body, headers, retries, +10 more.

http_postA
Destructive

Perform an HTTP POST. Body can be given as a raw string (body) or as a JSON value (body_json — auto-sets content-type to application/json). NOT idempotent: calling twice submits twice.

Inputs: url (required), body, headers, retries, +10 more.

http_putA
DestructiveIdempotent

Perform an HTTP PUT. Use for full-resource replacement. Idempotent: the same PUT applied twice leaves the resource in the same state.

Inputs: url (required), body, headers, retries, +10 more.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4/5.0

    Scored across 15 tools

    Disambiguation4/5

    The seven http_* verb tools are unambiguously separated by method, and fetch_meta/fetch_links/fetch_feed/fetch_sitemap/fetch_robots each target a distinct artifact. The only real overlap is the trio fetch_html_to_markdown, fetch_html_to_text, and fetch_reader, which all GET an HTML page and return readable content — the descriptions differentiate them (output format vs. article-body extraction), but an agent may still deliberate over which to pick.

    Naming Consistency5/5

    All names are snake_case and follow a clean two-prefix scheme: http_<method> for raw requests and fetch_<artifact> for extraction helpers. The pattern is predictable and self-documenting across all 15 tools.

    Tool Count5/5

    15 tools sits at the top of the well-scoped range and each one maps to a distinct HTTP method or extraction target. Nothing appears redundant filler, and the verb set (GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS) plus a small extraction suite is proportionate to a fetch server's purpose.

    Completeness4/5

    HTTP verb coverage is essentially complete (TRACE omitted, which is fine) and the parsing surface covers HTML, metadata, links, feeds, sitemaps, and robots.txt. Minor gaps: no explicit binary/raw-byte download tool, no form-urlencoded or multipart POST helper, and no cookie/session handling for multi-step flows.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      B
      quality
      D
      maintenance
      Enables fetching and converting web content to markdown with built-in prompt injection safeguards that detect and block malicious content attempting to manipulate the LLM.
      1
      2
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents to crawl and scrape websites, converting HTML to clean Markdown and structured metadata with support for JavaScript rendering, bot evasion, and SSRF protection.
      197
      MIT
    • A
      license
      A
      quality
      C
      maintenance
      Enables AI agents to read RSS/Atom feeds, extract web page content as text or Markdown, search the web, and batch-fetch multiple URLs without building custom crawlers.
      5
      MIT
    • A
      license
      A
      quality
      B
      maintenance
      Enables AI models to fetch real web pages while automatically bypassing anti-bot protections, login walls, and JS rendering, then returns clean, token-efficient article content.
      21
      MIT