Skip to main content
Glama
aza-ali

turbowebfetch

by aza-ali
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Local

TurboWebFetch

npm version npm downloads License GitHub stars

Real browsers. Real content. Full fidelity.

Your AI agents need to read web pages. Documentation, product info, articles, research. But standard fetch tools use plain HTTP - they cannot handle modern client-side rendering or bot mitigation layers, and return empty shells.

TurboWebFetch runs actual Chrome browsers. Your agents see what users see.

14 parallel browsers. Zero API keys. Runs locally.

Prerequisites

Before installing, verify you have:

node --version    # Need 18+
python3 --version # Need 3.8+

Google Chrome must be installed (not Chromium).

Related MCP server: crawler-mcp

Quick Start

claude mcp add turbowebfetch npx turbowebfetch

That's it. Your agents now have access to the fetch and fetch_batch tools.

What This Is (And Isn't)

TurboWebFetch helps your AI agents access content you have the right to access. It renders JavaScript-heavy pages that standard tools cannot handle.

It is for:

  • Fetching documentation that requires JS rendering (React, Stripe, etc.)

  • Product research on e-commerce sites

  • Reading articles and news behind JS walls

  • Multi-source research for your AI agents

It is not for:

  • Circumventing paywalls

  • Scraping data you don't have permission to collect

  • High-volume data harvesting (rate-limited by design)

  • Violating websites' Terms of Service

The challenge-handling exists because many legitimate sites use broad bot mitigation that affects even authorized access. If a site restricts access and you don't have permission, respect that.

WebFetch vs TurboWebFetch

Scenario

WebFetch

TurboWebFetch

Static HTML pages

Works

Works (overkill)

JavaScript SPAs

Empty content

Full render

Sites with JS challenges

Fails

Negotiates automatically

Bot mitigation layers

Fails

Negotiates automatically

Parallel agents

One at a time

14 simultaneous browsers

JS-heavy sites (docs, e-commerce)

Blocked or empty

Works

Rule of thumb: Use WebFetch for simple pages. Use TurboWebFetch when that fails.

Usage

Single page:

mcp__turbowebfetch__fetch(url: "https://react.dev/learn", format: "markdown")

Response:

{
  "success": true,
  "url": "https://react.dev/learn",
  "title": "Quick Start - React",
  "content": "# Quick Start\n\nWelcome to the React documentation...",
  "status": 200
}

Batch (parallel):

mcp__turbowebfetch__fetch_batch(
  urls: [
    "https://react.dev/learn",
    "https://nextjs.org/docs",
    "https://www.target.com/p/some-product"
  ],
  format: "text"
)

All three fetch simultaneously in separate browsers.

Parameters

Parameter

Default

Description

url

required

The URL to fetch

format

"text"

"text", "markdown", or "html"

timeout

60000

Milliseconds. Increase to 90000+ for slow sites

wait_for

-

CSS selector to wait for (rarely needed)

The tool auto-detects when content has loaded. Use wait_for only if auto-detection fails on a specific site.

Known Limitations

Sites that don't work:

  • Login-required content - This tool doesn't handle authentication

  • Interactive CAPTCHAs - It handles JS challenges, not image selection tasks

  • Zillow - Requires interactive verification

  • Bloomberg - Requires interactive verification

Performance:

  • Adds 5-10 seconds per page (browser startup + rendering + human-like behavior)

  • Memory usage: ~200-400MB per browser instance

  • For 14 parallel fetches, expect ~4GB RAM usage

Not for scale: This is a user assistant, not a scraping service. Rate-limited to 60 requests/minute per domain.

Configuration

Optional environment variables:

Variable

Default

Description

TURBOFETCH_MAX_PROCESSES

14

Max concurrent browsers

TURBOFETCH_HUMAN_MODE

true

Human-like scrolling/delays

TURBOFETCH_HEADLESS

true

Headless mode (auto-switches if blocked)

Most users won't need to change these.

Troubleshooting

"Python not found"

# macOS
brew install python3

# Ubuntu/Debian
sudo apt install python3 python3-venv

"Chrome not launching"

Install Google Chrome from https://google.com/chrome (not Chromium).

"Content is empty"

Some heavily lazy-loaded sites need an explicit selector:

mcp__turbowebfetch__fetch(
  url: "https://www.bestbuy.com/site/searchpage.jsp?st=laptop",
  wait_for: "[class*=\"product\"]",
  timeout: 90000
)

"Page not loading on [site]"

Some sites require interactive verification that automated browsers cannot complete. Open an issue with the URL.

How It Works

  1. Your agent calls the MCP tool

  2. TurboWebFetch spawns a Python process with Chrome (via nodriver)

  3. Chrome loads the page, executes JavaScript, negotiates any browser challenges

  4. Content is extracted and returned as clean text/markdown/HTML

  5. Browser closes, process exits

Each fetch is isolated. No cookies or state persist between requests.

Development

Clone and build locally:

git clone https://github.com/aza-ali/turbowebfetch.git
cd turbowebfetch
npm install
npm run build

Python setup (virtual environment + nodriver) runs automatically during npm install. If it fails, run manually:

npm run setup:python

Then register with Claude Code:

claude mcp add turbowebfetch node /path/to/turbowebfetch/dist/index.js

License

MIT License. See LICENSE for details.

Copyright (c) 2026 Mourtaza Ali

Available Tools

2 tools
fetchA

Fetch and render a web page using a real Chrome browser. Handles JavaScript-heavy sites, anti-bot protection, and dynamic content. Auto-detects when content has loaded by monitoring DOM changes and network activity.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesThe URL to fetch (must be a valid HTTP/HTTPS URL)
formatNoOutput format: 'html' for raw HTML, 'text' for cleaned text content, 'markdown' for structured markdowntext
wait_forNoCSS selector to wait for before extracting content. Usually not needed - the tool auto-detects content stabilization. Use this only when auto-detection fails and you know the specific element to wait for. Examples: '[class*="product"]' for e-commerce, '.job-card' for job boards, '[data-testid="results"]' for search results.
timeoutNoTimeout in milliseconds (default: 60000, max: 120000). Increase to 90000+ for slow-loading e-commerce or search result pages.
human_modeNoEnable human-mode scrolling and delays for more natural browsing behavior (default: true)

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses use of real Chrome browser, anti-bot handling, auto-detection of content load via DOM/network, and human_mode for natural behavior. Lacks details on error handling, output format specifics, or memory/state implications.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences efficiently convey purpose and capabilities. Front-loaded with essential information, no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequately covers main behavior and parameter context. Lacks explicit mention of return format handling (though implied by output format parameter) and error/timeout behavior, but sufficient given the detailed schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with detailed descriptions. The tool description adds minor context (e.g., auto-detection for wait_for) but does not significantly enhance understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states 'Fetch and render a web page' with specific capabilities (JavaScript-heavy, anti-bot, dynamic content). Distinguishes from sibling 'fetch_batch' by focusing on single-page fetch.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for single pages but does not explicitly state when to use this vs. fetch_batch, nor when not to use it. No alternative or prerequisite guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

fetch_batchA

Fetch multiple URLs in parallel using real Chrome browsers. Each URL gets its own browser instance for isolation. Auto-detects content stabilization. Returns results in same order as input URLs.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYesArray of URLs to fetch (1-14 URLs)
formatNoOutput format for all fetched pagestext
timeoutNoTimeout in milliseconds per URL (default: 60000). Increase to 90000+ for slow-loading pages.
human_modeNoEnable human-mode scrolling and delays for more natural browsing behavior (default: true)

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses parallel browser instances, isolation, auto-detected stabilization, and same-order results, but lacks depth on resource usage, rate limiting, or meaning of 'stabilization'. With no annotations, more detail would benefit agent decision-making.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three concise sentences front-load critical purpose and behavior, with no redundant information. Every sentence earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and description does not detail return structure beyond order preservation. For a tool with 4 parameters and no output schema, more contextual information on results would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema covers 100% of parameters with descriptions. Description adds context about isolation for urls and auto-detection for human_mode, but this is marginal beyond schema. Baseline score appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it fetches multiple URLs in parallel using real Chrome browsers with isolation and auto-stabilization, distinguishing it from the sibling tool 'fetch' which likely handles single URLs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for multiple URLs needing parallel fetching and isolation, but does not explicitly state when to use versus alternatives, nor provide exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 2 tool updatesv1.1.0
    • First observedfetch
    • First observedfetch_batch
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A3.9/5.0

Scored across 2 tools

Disambiguation5/5

The two tools have clearly distinct purposes: one fetches a single URL, the other fetches multiple URLs in parallel. Descriptions explicitly highlight the difference in scope and isolation, leaving no ambiguity.

Naming Consistency5/5

Both tool names follow the consistent pattern 'fetch' for single and 'fetch_batch' for batch, using a suffix to indicate the variant. No mixing of styles or conventions.

Tool Count4/5

With only 2 tools, the server is minimal but still covers the basic use cases of single and batch web fetching. The scope is narrow but appropriate for a focused web fetching server, though a few more tools (e.g., with options) could be justified.

Completeness3/5

The server covers the core functionality of fetching and rendering web pages, but lacks advanced features like custom headers, cookies, or wait conditions. Users needing fine-grained control may find the surface incomplete, though basic needs are met.

Maintenance

ActivityStale
ResponsivenessSyncing

Resources

Related MCP Connectors

Related MCP Servers