Skip to main content
Glama
georgittanchev

web-tools-mcp-server

web-tools-mcp-server

Drop-in MCP replacement for Claude's native web_search and web_fetch tools. Solves the "failed to fetch" / "failed to crawl" errors caused by IP blocks and user-agent detection on Claude's default infrastructure.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│                     web-tools-mcp-server                         │
│                                                                  │
│  web_search ──────► Brave Search API ($5/1k queries)             │
│                     Your own API key = your own IP = no blocks   │
│                                                                  │
│  web_fetch  ──────► Strategy: AUTO (default)                     │
│                     │                                            │
│                     ├─ 1. Direct fetch                           │
│                     │   • Rotating real-browser User-Agents      │
│                     │   • Full browser headers (Sec-Ch-Ua, etc.) │
│                     │   • Optional proxy (Bright Data / any)     │
│                     │                                            │
│                     └─ 2. Jina Reader fallback (on 403/429/503)  │
│                        • Server-side JS rendering                │
│                        • Returns clean LLM-ready markdown        │
│                        • Free tier (no key needed)               │
└──────────────────────────────────────────────────────────────────┘

Related MCP server: ai-first-scraper-mcp

Why this beats native tools

Problem with native tools

How this MCP fixes it

Claude's IP is known and blocked by many sites

Your proxy IP (Bright Data residential) or at minimum your own server's IP

Claude's user agent is detected as a bot

Rotates through 12+ real browser UAs with matching Sec-Ch-Ua headers

JS-heavy sites return empty content

Jina Reader renders JS server-side, returns clean markdown

"Failed to fetch" with no fallback

Auto strategy tries direct → falls back to Jina automatically

Search results may be limited

Brave's 35B+ page index, you control the API key

Prerequisites

  • Node.js 18+

  • Brave Search API keyGet one here ($5 free credit/month)

  • Optional: Jina API key for higher rate limits — Get one here (free tier works without key)

  • Optional: Proxy URL (Bright Data, or any HTTP/SOCKS5 proxy)

Installation

# Clone or copy this directory
cd web-tools-mcp-server

# Install dependencies
npm install

# Build
npm run build

# Verify it compiled
ls dist/index.js

Environment Variables

Variable

Required

Description

BRAVE_API_KEY

Yes

Brave Search API subscription token

PROXY_URL

No

Proxy URL for direct fetches. Format: http://user:pass@host:port or socks5://user:pass@host:port

JINA_API_KEY

No

Jina Reader API key for higher rate limits. Free tier works without it.

HTTP_PROXY

No

Alternative to PROXY_URL (standard env var)

HTTPS_PROXY

No

Alternative to PROXY_URL (standard env var)

Bright Data proxy example

export PROXY_URL="http://brd-customer-XXXXX-zone-XXXXX:PASSWORD@brd.superproxy.io:22225"

Bright Data with residential IPs (best anti-detection)

export PROXY_URL="http://brd-customer-XXXXX-zone-residential:PASSWORD@brd.superproxy.io:22225"

Setup for Claude Code

Add to your Claude Code MCP configuration (~/.claude/claude_code_config.json or per-project .claude/config.json):

{
  "mcpServers": {
    "web-tools": {
      "command": "node",
      "args": ["/absolute/path/to/web-tools-mcp-server/dist/index.js"],
      "env": {
        "BRAVE_API_KEY": "your-brave-api-key-here",
        "PROXY_URL": "http://user:pass@proxy:port",
        "JINA_API_KEY": "optional-jina-key"
      }
    }
  }
}

Or using the CLI:

claude mcp add web-tools \
  -e BRAVE_API_KEY=your-key \
  -e PROXY_URL=http://user:pass@proxy:port \
  -- node /absolute/path/to/web-tools-mcp-server/dist/index.js

Setup for Claude AI (Desktop App)

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "web-tools": {
      "command": "node",
      "args": ["/absolute/path/to/web-tools-mcp-server/dist/index.js"],
      "env": {
        "BRAVE_API_KEY": "your-brave-api-key-here",
        "PROXY_URL": "http://user:pass@proxy:port",
        "JINA_API_KEY": "optional-jina-key"
      }
    }
  }
}

Restart the Claude desktop app after saving.

Tools

Search the web via Brave Search API.

Parameters:

  • query (string, required): Search query. 1-6 words recommended.

  • count (number, optional): Results count, 1-20. Default: 10.

  • offset (number, optional): Pagination offset. Default: 0.

  • country (string, optional): Two-letter country code (e.g., "us").

  • search_lang (string, optional): Language code (e.g., "en").

  • freshness (string, optional): Time filter — "pd", "pw", "pm", "py", or "YYYY-MM-DDtoYYYY-MM-DD".

web_fetch

Fetch a URL with anti-detection capabilities.

Parameters:

  • url (string, required): Full URL to fetch (must include https:// or http://).

  • strategy (string, optional): Fetch strategy. Default: "auto".

    • "auto": Direct fetch first, Jina Reader fallback on block.

    • "direct": Direct fetch only (UA rotation + proxy).

    • "jina": Jina Reader only (JS rendering, clean markdown output).

Fetch Strategy Decision Guide

Is the site JS-heavy or known for aggressive anti-bot?
├── Yes ──► Use strategy="jina"
└── No
    ├── Do you have a proxy configured?
    │   ├── Yes ──► Use strategy="auto" (default) — direct with proxy, Jina fallback
    │   └── No  ──► Use strategy="auto" — tries without proxy, Jina catches failures
    └── Is it a raw API/JSON endpoint?
        └── Yes ──► Use strategy="direct" — Jina would mangle the JSON

Updating User Agents

The user agent pool in src/constants.ts should be updated periodically to match current browser versions. Check whatismybrowser.com/guides/the-latest-user-agent for current strings.

Cost Estimate

Component

Cost

Notes

Brave Search API

$5/1k queries

$5 free credit/month

Jina Reader

Free tier

1M tokens free, no key needed

Bright Data (optional)

~$8-15/GB residential

Pay-as-you-go available

Typical monthly for light use

~$0-5

Under 1k searches + Jina free tier

Troubleshooting

"BRAVE_API_KEY environment variable is required" → Set the BRAVE_API_KEY env var in your MCP config.

Direct fetch always returns 403 → Configure PROXY_URL with a residential proxy (Bright Data), or rely on Jina fallback (strategy="auto").

Jina returns empty/garbage → Some sites block Jina too. Configure a Bright Data residential proxy and use strategy="direct".

"undici ProxyAgent could not be loaded" → Your Node.js version doesn't bundle undici properly. Run: npm install undici in the project directory.

License

MIT

Available Tools

3 tools
web_bulk_fetchWeb Bulk FetchA
Read-only

Fetch multiple URLs in parallel with anti-detection capabilities. Much faster than calling web_fetch multiple times.

Use this after web_search to fetch the top 3-4 results in a single call instead of making separate web_fetch calls.

Args:

  • urls (string[], required): Array of URLs to fetch (max 10). Each must include https:// or http://.

  • strategy (string, optional): "auto", "direct", or "jina". Applied to all URLs. Default: "auto".

Returns: Combined results for all URLs, each with page content, HTTP status, strategy used, and content size. Failed URLs are clearly marked.

Example: urls=["https://docs.python.org/3/library/asyncio.html", "https://nodejs.org/en/docs/guides/event-loop"]

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYesArray of URLs to fetch in parallel (max 10)
strategyNoFetch strategy applied to all URLs: 'auto' (default), 'direct', or 'jina'auto

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint, openWorldHint), the description adds meaningful behavioral details: fetches in parallel, uses anti-detection, returns combined results with status, strategy used, and content size, and clearly marks failed URLs. This gives the agent a solid mental model of execution and output.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with clear sections for arguments, returns, and an example. It is somewhat verbose but all content earns its place—the example is useful, and the returns section compensates for the missing output schema. Tight enough for its informative value.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity (parallel fetching, strategy selection) and rich annotations, the description is fully self-sufficient. It provides return format, failure behavior, usage context, and an example, leaving no critical gap for an agent to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema already provides full descriptions for both parameters (100% coverage), including max items and enum options. The description adds an example and clarifies that URLs must include http(s):// but largely repeats what the schema states, so it does not materially enhance parameter understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Fetch multiple URLs in parallel' with 'anti-detection capabilities.' It distinguishes itself from sibling web_fetch by emphasizing parallelism and speed, making its purpose unmistakable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly provides usage context: 'Use this after web_search to fetch the top 3-4 results in a single call instead of making separate web_fetch calls.' This directly names the alternative (web_fetch) and gives a concrete scenario, guiding the agent precisely when to choose this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_fetchWeb FetchA
Read-onlyIdempotent

Fetch the contents of a web page with anti-detection capabilities.

This tool replaces the native web_fetch. It uses rotating real-browser user agents, optional proxy (Bright Data or any HTTP/SOCKS5), and automatic fallback to Jina Reader API for sites that block direct access.

Three fetch strategies:

  • "auto" (default): Tries direct fetch first (with UA rotation + proxy if configured). If blocked (403/429/503/captcha), automatically falls back to Jina Reader.

  • "direct": Direct HTTP fetch only. Uses rotating browser user agents and your configured proxy. Returns raw HTML.

  • "jina": Uses Jina Reader API exclusively. Best for JS-heavy sites. Returns clean LLM-ready markdown. Handles rendering server-side.

For fetching multiple URLs at once, use web_bulk_fetch instead — it's faster and uses a single tool call.

Args:

  • url (string, required): The URL to fetch. Must include https:// or http://.

  • strategy (string, optional): "auto", "direct", or "jina". Default: "auto".

Returns: Page content (HTML for direct, markdown for Jina), HTTP status, final URL after redirects, and which strategy was used.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesFull URL to fetch, must include https:// or http://
strategyNoFetch strategy: 'auto' (try direct with proxy+UA rotation, fallback to Jina Reader), 'direct' (only direct fetch with rotating UA + optional proxy), 'jina' (only Jina Reader — best for JS-heavy sites and anti-bot bypassing)auto

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds significant behavioral detail beyond annotations: rotating user agents, optional proxy, fallback to Jina Reader on blocks, and the ability to choose strategies. It also discloses the return format (HTML vs markdown) and the included status/final URL/strategy, enriching the tool's operational model.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Despite being detailed, the description is well-structured with clear sections (purpose, strategies, alternative, args, returns). Each sentence contributes valuable information without redundancy, and the front-loaded core purpose makes it easy to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description compensates by detailing return values (page content, HTTP status, final URL, strategy used). It covers tool alternatives, strategy selection, and parameter semantics thoroughly, making it self-sufficient for an agent to select and invoke successfully.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds extra meaning by explaining the strategies in depth (auto, direct, jina) and how they behave, including fallback logic and output format differences. This goes beyond the schema's brief parameter descriptions, enhancing the agent's understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool fetches web page contents with anti-detection capabilities, specifying the verb 'Fetch' and resource 'contents of a web page'. It distinguishes itself from sibling web_bulk_fetch by noting that the latter is for multiple URLs, making the purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly advises using web_bulk_fetch for multiple URLs, and details when to use each strategy (e.g., 'jina' best for JS-heavy sites). The 'auto' strategy's fallback behavior is explained, providing clear when-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A4.7/5.0
Disambiguation5/5

Each tool has a distinct purpose: web_fetch retrieves a single page, web_bulk_fetch retrieves multiple pages in parallel, and web_search performs a web search. There is no overlap between these operations; even web_fetch and web_bulk_fetch are clearly differentiated by the number of URLs they handle.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern with a web_ prefix: web_fetch, web_search, web_bulk_fetch. The naming is predictable and clearly indicates the action and resource, with 'bulk' correctly modifying the fetch verb.

Tool Count5/5

With only three tools, the set is well-scoped for a web utility server. Each tool covers a distinct primary use case (single fetch, bulk fetch, search), and the size is appropriate without being bloated or too sparse.

Completeness5/5

The tool surface covers the core operations for the domain: searching the web and fetching one or multiple pages, with strategy options (auto/direct/jina) for different anti-blocking needs. No obvious gaps for typical web research workflows, as pagination via count/offset is available in web_search.

Maintenance

ActivityInactive
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Drop-in MCP replacement for the built-in WebFetch tool. Adds domain-scoped custom HTTP headers via YAML config, with bot-block detection, HTML-to-text extraction, retries, proxies, and prompt-injection sanitization.
    1
    MIT
  • A
    license
    A
    quality
    C
    maintenance
    Ad-free web scraping and search exposed as 3 MCP tools. fetch_page, fetch_pages_batch, search_web. Works with Claude Desktop, Cursor, Cline.
    3
    2
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Helps AI agents search the public web and fetch content with anti-bot measures, returning clean markdown outputs suitable for citation.
    2
    Apache 2.0
  • A
    license
    Not graded
    quality
    D
    maintenance
    Provides web search and page fetching tools for third-party models (e.g., DeepSeek, Qwen, Kimi) in Claude Code, with configurable search backends, Markdown output, and safety boundaries.
    63
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/georgittanchev/mcp-search'

If you have feedback or need assistance with the MCP directory API, please join our Discord server