Skip to main content
Glama

@pipeworx/web-fetch

Fetch the exact contents of one public URL, live from its origin server, and get back the bytes (or text decoded from them) with a SHA-256 hash of exactly what the origin sent, plus the redirect chain and the response headers. Also lists and extracts named members from a ZIP archive at a public URL, without downloading the whole archive when the origin honours HTTP Range.

Part of Pipeworx — an MCP gateway connecting AI agents to 1867+ live data sources. This is an independent, unofficial integration — not affiliated with, endorsed by, or published by the upstream provider.

Tools

  • fetch_url(url, format?, max_bytes?, offset?) — one GET of one public http(s) URL. Returns status, url_final, redirects (present only when the URL redirected), headers (content-type, content-length, last-modified, etag, content-language, date), content_family (text | pdf | archive | binary), bytes_total, sha256 over the complete body, and a window of the body:

    • format: "auto" (default) — readable text for HTML and PDF, the raw body for everything else.

    • format: "raw" — the body itself: decoded text for text types, base64 for binary (zip, xlsx, gzip, PDF). sha256_returned hashes exactly the bytes in the window.

    • format: "text" — extracted text (text_extraction: html | pdf | none | failed). PDF extraction warnings are passed through in warnings.

    • max_bytes (default 100,000; at most 1,000,000 for text, 512,000 for base64) and offset page through a large body; follow next_offset.

    Text is decoded from the bytes, never guessed: the charset comes from a BOM, then the Content-Type header, then the page's own <meta charset> or XML declaration, then UTF-8 if the bytes are valid UTF-8, else windows-1252. charset_source says which rule decided and decode_replacements counts any characters lost in decoding (0 means none).

  • extract_archive_member(url, member?, format?, max_bytes?, offset?) — list or extract a named member from a ZIP archive at url.

    • No member: lists members — name, compressed_size, uncompressed_size, method (stored | deflate | method_N), crc32, local_header_offset, encrypted. members_truncated says if there were more than members_returned.

    • With member (exact name, falling back to a case-insensitive match — matched_by says which): decompresses that one entry and returns it — format: "auto"/"raw" give decoded text for a textual member and base64 for binary; "text" forces a text decode (falls back to base64 with a warning if the member looks binary). Windowed by max_bytes/offset exactly like fetch_url, over the DECOMPRESSED bytes. crc32_verified confirms the decompressed bytes match the archive's own checksum.

    • Reads only the end-of-central-directory record and the central directory — at the tail of the file — to list members, and only the wanted member's local header + payload to extract it, all via HTTP Range. A multi-GB archive costs the same few requests as a tiny one, as long as the origin honours Range (most do, including Internet Archive Wayback Machine captures); otherwise it falls back to a capped full download (range_supported: false).

    • Zip-bomb protection is two independent layers: the central directory's declared uncompressed size is checked against the cap BEFORE any byte is decompressed, and the real decompressed byte count is ALSO capped DURING decompression — so a member that lies about its own declared size is still caught. Encrypted entries, unsupported compression methods (only stored and deflate), and archives whose own end-of-central-directory record is ZIP64 (more than 65,535 entries, or a 4GB+ central directory) are refused by name rather than mis-read.

    • Refusals: member_not_found, encrypted_entry, unsupported_compression_method, decompressed_too_large (checked both before and during decompression — see above), too_many_members, directory_too_large, zip64_not_supported, too_large (Range not honoured and the archive exceeds the read cap), invalid_archive (not a ZIP, or corrupt), plus fetch_url's own blocked_url / upstream_refused / upstream_down / budget_exhausted for reaching the URL in the first place.

Checking "verbatim" yourself

sha256 is taken over the entity body after transfer compression is removed — the same bytes curl --compressed -s <url> | shasum -a 256 writes. Two caveats: a dynamic page can legitimately hash differently a second later, and some sites serve different content by location, so the answer is verbatim as served to this fetch (colo, when known, names the edge location).

Refusals — always loud, never an empty body

error

when

blocked_url

rule: scheme (not http/https), port (not 80/443), userinfo (user:pass@), private_address, dns_private (the name resolves to a private address), denylisted_host, too_many_redirects, invalid_redirect. Checked on the URL and every redirect hop.

upstream_refused

401 / 402 / 403 / 407 / 429 / 451, or a bot-wall page (reason: "bot_wall"). Carries status, a short body_summary, and a hint naming archive_evidence, which is never run automatically.

upstream_down

5xx, timeout, connection or DNS failure (phase).

upstream_status

any other non-2xx, e.g. 404.

unsupported_content_type

images, audio, video, fonts and executables (by declared type or by magic bytes).

too_large

an offset beyond the 10 MB read limit on an origin without byte ranges.

budget_exhausted

the per-site or hourly limit below; carries retry_after_s.

invalid_argument

a malformed argument.

Related MCP server: Web Resource Ledger

Limits

  • One GET, no cookies, no login, no caller headers, no JavaScript execution. likely_needs_js: true flags a page that renders its content with scripts.

  • 10 MB read per call; a larger body returns the first window with truncated: true, body_complete: false and sha256: null.

  • 25 s per call, at most 4 redirects.

  • Per site (registrable domain), across all callers: 30 requests a minute and 1,000 a day (sec.gov: 5 a minute). A 200 response is reused for 10 minutes (served_from: "cache", cache_age_s).

  • Identifies itself honestly as Pipeworx/1.0 (pipeworx.io); on a 403 it retries once as Pipeworx/1.0 (+https://pipeworx.io; support@pipeworx.io) and reports which it used in user_agent_used.

Auth

Keyless.

Data sources

  • Whatever public URL the caller names. Nothing is fetched except that URL and its redirects, plus a DNS-over-HTTPS lookup at https://cloudflare-dns.com/dns-query to refuse names that resolve to private addresses.

Quick Start

Add to your MCP client (Claude Desktop, Cursor, Windsurf, etc.):

{
  "mcpServers": {
    "web-fetch": {
      "url": "https://gateway.pipeworx.io/web-fetch/mcp"
    }
  }
}

What this endpoint actually serves

tools/list at https://gateway.pipeworx.io/web-fetch/mcp returns the tools in the table above plus the shared Pipeworx meta-tools — ask_pipeworx, discover_tools, search_within, remember/recall and the rest of the gateway-wide set. So the tool count you see is larger than this table: a single-pack endpoint currently lists roughly 30 shared tools alongside the pack's own. The connection's initialize response states its exact scope, and is the authoritative answer for a given day.

This is deliberate, not multiplexing by accident. The meta-tools are what let a scoped connection answer a question this pack does not cover — via ask_pipeworx, which routes across the whole catalog — without you adding a second MCP server. There is currently no way to mount a pack endpoint without them; if the extra schemas cost you more context than the routing is worth, connect to the full gateway once rather than to several pack endpoints.

Or connect to the full Pipeworx gateway to get every pack's tools listed directly, instead of just this one's:

{
  "mcpServers": {
    "pipeworx": {
      "url": "https://gateway.pipeworx.io/mcp"
    }
  }
}

Both URLs reach the same gateway and the same 1867+ data sources. The only difference is which pack's tools are listed directly; ask_pipeworx reaches all of them from either one.

No MCP client? Call it over HTTP

curl -X POST https://gateway.pipeworx.io/v1/tools/fetch_url \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv","max_bytes":2000}'

No account needed for the first calls. Inspect any tool: GET https://gateway.pipeworx.io/v1/tools/fetch_url. Find one: POST https://gateway.pipeworx.io/v1/tools/search_packs with {"query":"..."}.

Run it from npm

@pipeworx/mcp-web-fetch is a local stdio bridge to the Pipeworx gateway: your MCP client starts it with npx, and it forwards every call to the same https://gateway.pipeworx.io/web-fetch/mcp endpoint shown above. It needs network access; there is no offline mode.

{
  "mcpServers": {
    "web-fetch": {
      "command": "npx",
      "args": ["-y", "@pipeworx/mcp-web-fetch"]
    }
  }
}

Without a key it calls anonymously, capped at 50 calls a day. A free account raises that to 200 a day: get a key at https://pipeworx.io/signup and add "env": { "PIPEWORX_API_KEY": "<your key>" } to the server entry. The key is sent as a Bearer token, the same way a hosted connection sends it, so calls are metered and billed exactly as they are there.

It lists only this pack's tools by default. Set PIPEWORX_META_TOOLS=1 to also list ask_pipeworx and the other gateway-wide tools.

Behind an HTTPS proxy: Node's built-in fetch does not read HTTPS_PROXY by default. Newer Node releases honour it when NODE_USE_ENV_PROXY=1 is set; check your Node version's documentation.

Using with ask_pipeworx

Instead of calling tools directly, you can ask questions in plain English — this works on the pack endpoint above as well as on the full gateway:

ask_pipeworx({ question: "your question about Web Fetch data" })

The gateway picks the right tool and fills the arguments automatically.

More

License

MIT

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Securely fetches web content, extracts links and metadata, and downloads files through a sandboxed MCP server without JavaScript execution. Includes prompt-injection detection and comprehensive HTML sanitization for safe web data retrieval.
      2
      MIT
    • F
      license
      Not graded
      quality
      B
      maintenance
      📇 ☁️ - Capture web pages as cryptographically signed, tamper-evident evidence. Ed25519 signatures, RFC 3161 timestamps, and WACZ archives. Four tools: capture_url, get_capture, list_captures, verify_capture.
      2
      -
    • F
      license
      Not graded
      quality
      Not graded
      maintenance
      URL reality check for AI agents — returns HTTP status, SHA-256 content hash, classification, readability score, title, and wayback-machine fallback when dead, cached 10 minutes at $0.001 per call.
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      Read-only observation of a single live URL: parses static HTML to report security posture, forms, links, accessibility signals, and leaks, with described fixes. SSRF-gated and safe, never executes JavaScript.
      85 npm
      MIT