Skip to main content
Glama

TurboWebFetch

npm version npm downloads License GitHub stars

Real browsers. Real content. Full fidelity.

Your AI agents need to read web pages. Documentation, product info, articles, research. But standard fetch tools use plain HTTP - they cannot handle modern client-side rendering or bot mitigation layers, and return empty shells.

TurboWebFetch runs actual Chrome browsers. Your agents see what users see.

14 parallel browsers. Zero API keys. Runs locally.


Prerequisites

Before installing, verify you have:

node --version    # Need 18+
python3 --version # Need 3.8+

Google Chrome must be installed (not Chromium).


Related MCP server: crawler-mcp

Quick Start

claude mcp add turbowebfetch npx turbowebfetch

That's it. Your agents now have access to the fetch and fetch_batch tools.


What This Is (And Isn't)

TurboWebFetch helps your AI agents access content you have the right to access. It renders JavaScript-heavy pages that standard tools cannot handle.

It is for:

  • Fetching documentation that requires JS rendering (React, Stripe, etc.)

  • Product research on e-commerce sites

  • Reading articles and news behind JS walls

  • Multi-source research for your AI agents

It is not for:

  • Circumventing paywalls

  • Scraping data you don't have permission to collect

  • High-volume data harvesting (rate-limited by design)

  • Violating websites' Terms of Service

The challenge-handling exists because many legitimate sites use broad bot mitigation that affects even authorized access. If a site restricts access and you don't have permission, respect that.


WebFetch vs TurboWebFetch

Scenario

WebFetch

TurboWebFetch

Static HTML pages

Works

Works (overkill)

JavaScript SPAs

Empty content

Full render

Sites with JS challenges

Fails

Negotiates automatically

Bot mitigation layers

Fails

Negotiates automatically

Parallel agents

One at a time

14 simultaneous browsers

JS-heavy sites (docs, e-commerce)

Blocked or empty

Works

Rule of thumb: Use WebFetch for simple pages. Use TurboWebFetch when that fails.


Usage

Single page:

mcp__turbowebfetch__fetch(url: "https://react.dev/learn", format: "markdown")

Response:

{
  "success": true,
  "url": "https://react.dev/learn",
  "title": "Quick Start - React",
  "content": "# Quick Start\n\nWelcome to the React documentation...",
  "status": 200
}

Batch (parallel):

mcp__turbowebfetch__fetch_batch(
  urls: [
    "https://react.dev/learn",
    "https://nextjs.org/docs",
    "https://www.target.com/p/some-product"
  ],
  format: "text"
)

All three fetch simultaneously in separate browsers.


Parameters

Parameter

Default

Description

url

required

The URL to fetch

format

"text"

"text", "markdown", or "html"

timeout

60000

Milliseconds. Increase to 90000+ for slow sites

wait_for

-

CSS selector to wait for (rarely needed)

The tool auto-detects when content has loaded. Use wait_for only if auto-detection fails on a specific site.


Known Limitations

Sites that don't work:

  • Login-required content - This tool doesn't handle authentication

  • Interactive CAPTCHAs - It handles JS challenges, not image selection tasks

  • Zillow - Requires interactive verification

  • Bloomberg - Requires interactive verification

Performance:

  • Adds 5-10 seconds per page (browser startup + rendering + human-like behavior)

  • Memory usage: ~200-400MB per browser instance

  • For 14 parallel fetches, expect ~4GB RAM usage

Not for scale: This is a user assistant, not a scraping service. Rate-limited to 60 requests/minute per domain.


Configuration

Optional environment variables:

Variable

Default

Description

TURBOFETCH_MAX_PROCESSES

14

Max concurrent browsers

TURBOFETCH_HUMAN_MODE

true

Human-like scrolling/delays

TURBOFETCH_HEADLESS

true

Headless mode (auto-switches if blocked)

Most users won't need to change these.


Troubleshooting

"Python not found"

# macOS
brew install python3

# Ubuntu/Debian
sudo apt install python3 python3-venv

"Chrome not launching"

Install Google Chrome from https://google.com/chrome (not Chromium).

"Content is empty"

Some heavily lazy-loaded sites need an explicit selector:

mcp__turbowebfetch__fetch(
  url: "https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
  wait_for: "[class*=\"product\"]",
  timeout: 90000
)

"Page not loading on [site]"

Some sites require interactive verification that automated browsers cannot complete. Open an issue with the URL.


How It Works

  1. Your agent calls the MCP tool

  2. TurboWebFetch spawns a Python process with Chrome (via nodriver)

  3. Chrome loads the page, executes JavaScript, negotiates any browser challenges

  4. Content is extracted and returned as clean text/markdown/HTML

  5. Browser closes, process exits

Each fetch is isolated. No cookies or state persist between requests.


Development

Clone and build locally:

git clone https://github.com/aza-ali/turbowebfetch.git
cd turbowebfetch
npm install
npm run build

Python setup (virtual environment + nodriver) runs automatically during npm install. If it fails, run manually:

npm run setup:python

Then register with Claude Code:

claude mcp add turbowebfetch node /path/to/turbowebfetch/dist/index.js

License

MIT License. See LICENSE for details.

Copyright (c) 2026 Mourtaza Ali

Available Tools

2 tools
fetchA

Fetch and render a web page using a real Chrome browser. Handles JavaScript-heavy sites, anti-bot protection, and dynamic content. Auto-detects when content has loaded by monitoring DOM changes and network activity.

Inputs: url (required), format, wait_for, timeout, +1 more.

fetch_batchA

Fetch multiple URLs in parallel using real Chrome browsers. Each URL gets its own browser instance for isolation. Auto-detects content stabilization. Returns results in same order as input URLs.

Inputs: urls (required), format, timeout, human_mode.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 2 tools

    Disambiguation5/5

    The two tools have clearly distinct purposes: one fetches a single URL, the other fetches multiple URLs in parallel. Descriptions explicitly highlight the difference in scope and isolation, leaving no ambiguity.

    Naming Consistency5/5

    Both tool names follow the consistent pattern 'fetch' for single and 'fetch_batch' for batch, using a suffix to indicate the variant. No mixing of styles or conventions.

    Tool Count4/5

    With only 2 tools, the server is minimal but still covers the basic use cases of single and batch web fetching. The scope is narrow but appropriate for a focused web fetching server, though a few more tools (e.g., with options) could be justified.

    Completeness3/5

    The server covers the core functionality of fetching and rendering web pages, but lacks advanced features like custom headers, cookies, or wait conditions. Users needing fine-grained control may find the surface incomplete, though basic needs are met.

    Maintenance

    ActivityNo data
    ResponsivenessSyncing

    Related MCP Connectors

    Related MCP Servers