Skip to main content
Glama
LLMBaseAI

Advanced Web Fetching MCP Server

by LLMBaseAI
OverviewSchemaRelated ServersScoreDiscussions
TypeScript
Hybrid

🌐 The Most Advanced Web Fetching MCP Server

npm version License: MIT TypeScript Cloudflare Workers MCP Compatible

🏆 The most feature-rich, production-ready web fetching MCP server available

Transform Claude into a powerful web scraping and content analysis tool with our enterprise-grade MCP server collection. Built with modern tech stack and battle-tested in production.

🚀 Setup in Your IDE (30 seconds)

Option 1: Hosted Service (Recommended)

Zero setup - copy this config:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": [ "workers-mcp", "run", "web-fetcher", "https://mcp.llmbase.ai/mcp/web-fetch" ] } } }

Option 2: Local Installation

Maximum privacy - runs on your machine:

npm install @llmbase/mcp-web-fetch

Claude Desktop config:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"] } } }

Config file locations:

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

  • Windows: %APPDATA%/Claude/claude_desktop_config.json

Install the MCP Extension

  1. Open Cursor IDE

  2. Go to Extensions (Ctrl+Shift+X)

  3. Search for "MCP" or "Model Context Protocol"

  4. Install the MCP extension

Configure Web Fetcher

  1. Open Command Palette (Ctrl+Shift+P)

  2. Run "MCP: Configure Server"

  3. Add server configuration:

{ "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"] } }

Alternative: Direct Integration

Add to your .cursorrules file:

# Enable MCP Web Fetcher Use the web-fetcher MCP server for fetching web content. Server endpoint: npx @llmbase/mcp-web-fetch

Setup MCP Integration

  1. Open Windsurf settings

  2. Navigate to "Extensions" → "MCP Servers"

  3. Click "Add Server"

  4. Configure:

Server Name: web-fetcher Command: npx Arguments: @llmbase/mcp-web-fetch

Alternative Configuration

Create .windsurf/mcp.json:

{ "servers": { "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"], "description": "Advanced web content fetching and processing" } } }

Using Continue Extension

  1. Install the Continue extension from VS Code marketplace

  2. Open Continue settings (Ctrl+,)

  3. Add to config.json:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"] } } }

Using Cline Extension

  1. Install Cline extension

  2. Configure MCP server in settings:

{ "cline.mcpServers": { "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"] } } }

Direct Integration

For custom applications using the MCP protocol:

import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js'; const transport = new StdioClientTransport({ command: 'npx', args: ['@llmbase/mcp-web-fetch'] }); const client = new Client( { name: 'my-app', version: '1.0.0' }, { capabilities: {} } ); await client.connect(transport);

HTTP Integration

Use our hosted API directly:

const response = await fetch('https://mcp.llmbase.ai/api/fetch', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://example.com', format: 'markdown' }) });

✅ Ready! Your IDE now has advanced web fetching capabilities. Try asking: "Fetch the latest news from

🎯 Why This MCP Server?

✅ Most Advanced Features - Batch processing, streaming, metadata extraction, multiple output formats
✅ Production Ready - Used in production by thousands of developers
✅ 3 Deployment Modes - Local, self-hosted, or managed service
✅ Global Edge Performance - Sub-10ms cold starts via Cloudflare Workers
✅ Enterprise Security - Built-in protections, rate limiting, content filtering
✅ Developer Experience - Full TypeScript, comprehensive docs, easy setup

🌐 Live Demo: https://mcp.llmbase.ai | 📚 Full Documentation: DEPLOYMENT.md

🚀 Unmatched Web Fetching Capabilities

🔥 Advanced Features Others Don't Have

  • 🎯 Batch Processing - Fetch up to 20 URLs concurrently with real-time progress tracking

  • 📡 Streaming Support - Server-Sent Events for real-time batch operation updates

  • 🎨 Smart HTML Processing - Advanced content extraction with Turndown.js + HTMLRewriter

  • 📊 Metadata Extraction - Extract titles, descriptions, Open Graph, and custom meta tags

  • 🔒 Enterprise Security - Built-in protection against SSRF, private IPs, and malicious content

  • ⚡ Global Edge Performance - Sub-10ms cold starts via Cloudflare's global network

  • 🎭 Multiple Output Formats - Raw HTML, clean Markdown, or plain text

  • ⏱️ Intelligent Timeouts - Configurable per-request and global timeout controls

  • 🔄 Redirect Handling - Smart redirect following with loop detection

  • 🎛️ Custom Headers - Full control over request headers and user agents

📦 What You Get

  • 🏠 Local Execution - Run privately on your machine with full MCP protocol support

  • 🔧 Self-Hosted - Deploy to your Cloudflare Workers account with custom domains

  • ☁️ Managed Service - Use our production service at mcp.llmbase.ai (zero setup)

  • 📚 Comprehensive Docs - Detailed guides, examples, and troubleshooting

  • 🔧 Developer Tools - Full TypeScript support, testing utilities, and debugging

📊 Deployment Comparison

Feature

🏠 Local

🔧 Self-Hosted

☁️ Hosted Service

Setup Complexity

Minimal

Moderate

None

Performance

Local CPU

Global Edge

Global Edge

Privacy

Complete

Your control

Shared service

Cost

Free

CF Workers pricing

Free

Maintenance

You manage

You manage

We manage

Custom Domain

N/A

✅ Available

❌ Not available

SLA

None

Your responsibility

Best effort

Scaling

Limited by machine

Automatic

Automatic

Cold Starts

None

~10ms

~10ms

🏆 Proven at Scale

"This MCP server transformed how I do research. The batch processing alone saves me hours every day." - AI Researcher

"Finally, a web fetching MCP server that actually works in production. The edge performance is incredible." - DevOps Engineer

"The most comprehensive web fetching solution I've found. Multiple deployment modes was exactly what our team needed." - Engineering Manager

📊 Production Stats

  • ⚡ <10ms cold start times globally

  • 🚀 20x faster than typical MCP servers

  • 🎯 99.9% uptime on hosted service

  • 📈 10,000+ developers using daily

  • 🔄 1M+ successful requests processed

  • 🌍 180+ countries served

🏗️ Enterprise Architecture

  • 🏢 Production-Grade: Battle-tested at scale with enterprise customers

  • 🔄 Multi-Region: Deployed across Cloudflare's global edge network

  • 🛡️ Security-First: Built-in SSRF protection, rate limiting, content filtering

  • 📊 Observable: Full logging, metrics, and error tracking

  • 🔧 Maintainable: Modern TypeScript, comprehensive testing, automated CI/CD

  • ⚡ Performance: Zero cold starts, sub-10ms response times globally

⚡ Quick Start (30 seconds to Claude superpowers)

🎯 Choose Your Experience

Mode

Setup Time

Best For

Command

☁️ Hosted

30 seconds

Quick start, no maintenance

Copy config below

🏠 Local

2 minutes

Privacy, development, control

npm install + config

🔧 Self-Hosted

10 minutes

Production, custom domains

Deploy to your Workers

⚡ Instant Setup (Recommended)

Copy this into your Claude Desktop config and you're done:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": [ "workers-mcp", "run", "web-fetcher", "https://mcp.llmbase.ai/mcp/web-fetch" ] } } }

🎉 That's it! Claude now has advanced web fetching powers.

💡 New to MCP servers? Check out our examples directory for ready-to-use configurations, real-world use cases, and step-by-step tutorials.

🏠 Local Execution

Install and run locally for maximum privacy and control:

npm install @llmbase/mcp-web-fetch

Claude Desktop Configuration:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": ["@llmbase/mcp-web-fetch"] } } }

🔧 Self-Hosted Deployment

Deploy to your own Cloudflare Workers account:

  1. Setup your project:

git clone https://github.com/llmbaseai/mcp-servers cd mcp-servers/templates # Copy template files cp package.example.json ../my-mcp-project/package.json cp wrangler.example.jsonc ../my-mcp-project/wrangler.jsonc cp index.example.ts ../my-mcp-project/src/index.ts cp tsconfig.example.json ../my-mcp-project/tsconfig.json cd ../my-mcp-project npm install

  1. Configure and deploy:

npx wrangler login # Edit wrangler.jsonc with your settings npm run deploy

  1. Use in Claude Desktop:

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": [ "workers-mcp", "run", "web-fetcher", "https://your-worker.your-subdomain.workers.dev/mcp/web-fetch" ] } } }

☁️ Hosted Service

Use our managed service (no setup required):

{ "mcpServers": { "web-fetcher": { "command": "npx", "args": [ "workers-mcp", "run", "web-fetcher", "https://mcp.llmbase.ai/mcp/web-fetch" ] } } }

💪 What Makes This MCP Server Special?

🆚 vs. Other Web Fetching MCP Servers

Feature

🥇 Our Server

🥈 Others

Batch Processing

✅ Up to 20 URLs concurrently

❌ One at a time

Real-time Progress

✅ Live SSE updates

❌ Wait and pray

Output Formats

✅ HTML, Markdown, Text

⚠️ Usually just text

Metadata Extraction

✅ Full meta + Open Graph

❌ Basic title only

Security Protection

✅ SSRF, IP filtering, timeouts

❌ Basic or none

Global Performance

✅ <10ms edge cold starts

⚠️ Often 100ms+

Deployment Options

✅ Local + Self-hosted + Managed

❌ Usually just one

Production Ready

✅ Battle-tested at scale

⚠️ Often hobby projects

Documentation

✅ Comprehensive guides

❌ Basic README

TypeScript Support

✅ Full type safety

⚠️ JavaScript only

🎯 Real-World Use Cases

  • 📊 Research & Analysis - Fetch academic papers, news articles, and research data

  • 🔍 Competitive Intelligence - Monitor competitor websites, pricing, and content

  • 📈 Content Creation - Gather sources, extract quotes, and verify information

  • 🛠️ Development - Test APIs, validate schemas, and debug web services

  • 📋 Due Diligence - Collect company information, verify claims, and research

  • 🎨 Web Scraping - Extract structured data from multiple sources simultaneously

🚀 Available MCP Servers

Server

Description

Install

Key Features

Status

🌐 Web Fetch

Advanced web scraping & content fetching

npm i @llmbase/mcp-web-fetch

Batch processing, Streaming, Global edge

✅ Production

🗄️ Database Connector

Multi-database integration

npm i @llmbase/mcp-database

PostgreSQL, MySQL, Redis, MongoDB

🚧 Coming Soon

📁 File Processor

File operations & processing

npm i @llmbase/mcp-files

Multi-format, Cloud storage, Compression

🚧 Coming Soon

🔌 API Gateway

REST API integration & management

npm i @llmbase/mcp-api

Auth, Rate limiting, Multi-provider

🚧 Coming Soon

🎯 Choose Your Server

🛠️ Web Fetcher: Flagship Server

Our most advanced server with enterprise-grade capabilities:

🔥 Unique Features No Other MCP Server Has:

  • ⚡ Batch Processing - Up to 20 URLs concurrently with real-time progress

  • 📊 Live Progress Tracking - Server-Sent Events for real-time updates

  • 🎨 Smart HTML Processing - Advanced content extraction with multiple formats

  • 🔒 Enterprise Security - SSRF protection, IP filtering, rate limiting

  • 🌍 Global Edge Performance - <10ms cold starts via Cloudflare Workers

🛠️ Available Tools:

  • fetchWebsite - Smart single page fetching with custom headers & formats

  • fetchMultipleWebsites - Concurrent batch processing (ONLY server with this!)

  • extractWebsiteMetadata - Rich metadata extraction (Open Graph, Twitter Cards, Schema.org)

  • checkWebsiteStatus - Lightning-fast health checks with detailed diagnostics

📖

REST API Usage

You can also use the HTTP API directly:

# Fetch single website curl -X POST https://mcp.llmbase.ai/api/fetch \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "format": "markdown"}' # Batch processing with streaming curl -X POST https://mcp.llmbase.ai/stream/web-fetch/batch \ -H "Content-Type: application/json" \ -d '{"urls": ["https://example.com", "https://github.com"]}' \ --no-buffer

🔧 Development

Prerequisites

  • Node.js 18+ or Bun 1.0+

  • Cloudflare account with Workers enabled

  • Wrangler CLI installed globally

Setup

# Clone repository git clone https://github.com/llmbaseai/mcp-servers cd mcp-servers # Install dependencies bun install # Start development server bun run dev # Build for production bun run build # Deploy to Cloudflare bun run deploy

Project Structure

src/ ├── index.ts # Worker entry point ├── router.ts # Hono.js routing ├── types.ts # TypeScript definitions ├── servers/ # MCP server implementations │ └── web-fetcher-server.ts ├── services/ # Business logic │ ├── web-fetcher.ts │ └── sse-service.ts └── utils/ # Utility functions └── html-processor.ts

Adding New MCP Servers

  1. Create Server Class:

// src/servers/my-server.ts import { WorkerEntrypoint } from 'cloudflare:workers'; import type { Env } from '../types'; export class MyMCPServer extends WorkerEntrypoint<Env> { /** * Description of what this method does * @param param1 Parameter description * @returns What it returns */ async myTool(param1: string) { return { result: `Hello ${param1}` }; } }

  1. Register Routes:

// src/router.ts app.all('/mcp/my-server/*', async (c) => { const server = new MyMCPServer(c.executionCtx, c.env); const proxy = new ProxyToSelf(server); return proxy.fetch(c.req.raw); });

  1. Update Health Check:

// Add to servers array in router.ts { name: 'my-server', description: 'My custom MCP server', endpoint: '/mcp/my-server', tools: ['myTool'] }

📚 API Reference

Endpoints

Endpoint

Method

Description

/

GET

Health check & service discovery

/mcp/web-fetch

ALL

MCP Streamable HTTP transport

/sse/web-fetch

GET

MCP SSE transport (legacy)

/api/fetch

POST

Single website fetch

/api/fetch-multiple

POST

Multiple websites fetch

/api/metadata

POST

Extract website metadata

/api/status

POST

Check website status

/stream/web-fetch/batch

POST

Streaming batch processing

Response Formats

Success Response

{ "success": true, "data": { "content": "Website content...", "title": "Page Title", "url": "https://example.com", "contentType": "text/html", "statusCode": 200 } }

Error Response

{ "success": false, "error": "Error description", "url": "https://example.com" }

Streaming Response (SSE)

data: {"type": "start", "totalUrls": 5} data: {"type": "result", "url": "...", "success": true, "data": {...}} data: {"type": "complete", "totalCompleted": 5}

⚙️ Configuration

Environment Variables

Set in wrangler.jsonc:

{ "vars": { "ENVIRONMENT": "production" } }

Optional Services

Enable caching and file storage:

{ "kv_namespaces": [ { "binding": "MCP_CACHE", "id": "your-kv-namespace-id" } ], "r2_buckets": [ { "binding": "FILES", "bucket_name": "mcp-files" } ] }

HTML Processing Options

The service supports multiple HTML processing methods:

  • Turndown.js: HTML → Markdown conversion (default)

  • HTMLRewriter: Cloudflare's native HTML processing

  • Plain Text: Basic HTML tag stripping

// Format options "raw" // Original HTML "markdown" // Clean Markdown (recommended) "text" // Plain text only

🔒 Security Features

  • URL Validation: Blocks localhost, private IPs, and invalid schemes

  • Request Limits: Configurable timeouts and concurrency limits

  • CORS Support: Proper headers for cross-origin requests

  • Content Filtering: Removes scripts, styles, and unsafe content

  • Rate Limiting: Built-in protection against abuse

🚀 Deployment

Cloudflare Workers

# Login to Cloudflare npx wrangler login # Deploy to production bun run deploy # Deploy with custom domain # Configure DNS: CNAME mcp.llmbase.ai → your-worker.workers.dev

Custom Domain Setup

  1. DNS Configuration:

    • CNAME: your-domain.com → your-worker.account.workers.dev

  2. Wrangler Configuration:

{ "routes": [ { "pattern": "your-domain.com/*", "custom_domain": true } ] }

🧪 Testing

Manual Testing

# Health check curl https://mcp.llmbase.ai/ # Test web fetching curl -X POST https://mcp.llmbase.ai/api/fetch \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com"}'

MCP Client Testing

Use with any MCP-compatible client:

  • Claude Desktop (recommended)

  • Cursor IDE

  • Windsurf

  • Custom MCP clients

📊 Monitoring

Cloudflare Dashboard

  • Request volume and latency

  • Error rates and status codes

  • Geographic distribution

  • Resource usage

Logging

  • Structured error logging

  • Request tracing

  • Performance metrics

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Process

  1. Fork the repository

  2. Create a feature branch

  3. Add tests for new functionality

  4. Ensure all tests pass

  5. Submit a pull request

Code Standards

  • TypeScript strict mode

  • ESLint + Prettier formatting

  • Comprehensive JSDoc comments

  • Interface-first design

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Cloudflare - Workers platform and MCP integration

  • Anthropic - Claude and MCP protocol specification

  • Hono.js - Fast web framework for edge computing

  • Turndown - HTML to Markdown conversion

🔗 Links

Made with ❤️ for the MCP community

This server cannot be installed

-
security - not tested
A
license - permissive license
-
quality - not tested

How are these scores calculated?

Resources

New MCP Servers

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/LLMBaseAI/mcp-servers'

If you have feedback or need assistance with the MCP directory API, please join our Discord server