Skip to main content
Glama

URL Text Fetcher MCP Server

by billallison

URL Text Fetcher MCP Server

A modern Model Context Protocol (MCP) server that provides URL text fetching, web scraping, and web search capabilities using the FastMCP framework for use with LM Studio and other MCP-compatible clients.

The server is built using the modern FastMCP framework, which provides:

  • Clean decorator-based tool definitions
  • Automatic schema generation from type hints
  • Simplified server setup and configuration
  • Better error handling and logging

All security features and functionality have been preserved while modernizing to follow MCP best practices.

Features

This MCP server enables AI models to:

  • Fetch text content from any URL by extracting all visible text
  • Extract links from web pages to discover related resources
  • Search the web using Brave Search and automatically fetch content from top results
  • Handle errors gracefully with proper timeout and exception handling

Security Features

Enterprise-grade security implementation:

  • SSRF Protection: Blocks requests to internal networks and metadata endpoints
  • Input Sanitization: Validates and cleans all URL and query inputs
  • Memory Protection: Content size limits prevent memory exhaustion
  • Rate Limiting: Thread-safe API rate limiting with configurable thresholds
  • Error Handling: Comprehensive exception handling without information leakage

Tools

The server provides three main tools:

fetch_url_text

  • Description: Downloads all visible text from a URL
  • Parameters:
    • url (string, required): The URL to fetch text from
  • Returns: Clean text content from the webpage
  • Description: Extracts all links from a web page
  • Parameters:
    • url (string, required): The URL to fetch links from
  • Returns: List of all href links found on the page

brave_search_and_fetch

  • Description: Search the web using Brave Search and automatically fetch content from the top results
  • Parameters:
    • query (string, required): The search query
    • max_results (integer, optional): Maximum number of results to fetch content for (default: 3, max: 10)
  • Returns: Search results with full text content from each result URL

Prerequisites

Brave Search API Key

To use the search functionality, you'll need a free Brave Search API key:

  1. Visit Brave Search API
  2. Sign up for a free account (2,000 queries/month, max 1 per second)
  3. Get your API key
  4. Copy .env.example to .env and add your API key:
    cp .env.example .env # Edit .env and set: BRAVE_API_KEY=your_actual_api_key

Installation

  1. Clone this repository
  2. Install dependencies:
    uv sync --dev --all-extras
  3. Configure your environment:
    cp .env.example .env # Edit .env file and set your BRAVE_API_KEY

Usage

With LM Studio

  1. Open LM Studio and navigate to the Integrations section
  2. Click "Install" then "Edit mcp.json"
  1. Option A: Use the configuration helper script
    ./configure_lmstudio_fastmcp.sh
    This will generate the correct configuration with the right paths for your system.
  2. Option B: Manual configuration - Add the server configuration:
{ "mcpServers": { "url-text-fetcher-fastmcp": { "command": "uv", "args": [ "run", "url-text-fetcher-fastmcp" ], "cwd": "/Users/wallison/TechProjects/mcp-server" } } }
Legacy Implementation (Low-Level)

For the legacy implementation:

./configure_lmstudio.sh # Generates config for legacy server

Note: The API key will be automatically loaded from your .env file in the project directory.

  1. Save the configuration and restart LM Studio
  2. The server will appear in the Integrations section

Standalone Usage

You can also run the server directly:

# FastMCP implementation (recommended) uv run url-text-fetcher-fastmcp # Legacy implementation uv run url-text-fetcher

Examples

Once configured with LM Studio, you can ask the AI to:

  • "Fetch the text content from https://example.com"
  • "Get all the links from https://news.example.com"
  • "Search for 'Python web scraping' and show me the content from the top 3 results"
  • "What's the latest news about AI? Search and get the full articles"
  • "Find information about MCP servers and fetch the detailed content"

Dependencies

  • mcp>=1.12.3 - Model Context Protocol framework
  • requests>=2.31.0 - HTTP library for web requests and Brave Search API
  • beautifulsoup4>=4.12.0 - HTML parsing and text extraction

Configuration

The server can be configured via the .env file:

# Required: Brave Search API Key BRAVE_API_KEY=your_api_key_here # Brave Search API Rate Limit (requests per second) # Free tier: 1 request per second (default) # Paid tier: 20 requests per second # Higher tier: 50 requests per second # Set this to match your subscription level BRAVE_RATE_LIMIT_RPS=1 # Optional: Request timeout in seconds (default: 10) REQUEST_TIMEOUT=10 # Optional: Content length limit in characters (default: 5000) CONTENT_LENGTH_LIMIT=5000 # Optional: Maximum response size in bytes (default: 10MB) MAX_RESPONSE_SIZE=10485760

Brave Search Subscription Tiers

The server automatically adjusts its rate limiting based on your Brave Search subscription:

  • Free Tier: 1 request per second (BRAVE_RATE_LIMIT_RPS=1)
  • Paid Tier: 20 requests per second (BRAVE_RATE_LIMIT_RPS=20)
  • Higher Tier: 50 requests per second (BRAVE_RATE_LIMIT_RPS=50)

The server will enforce the configured rate limit across all concurrent requests to ensure you stay within your API quota.

See .env.example for a template.

Development

This project uses:

  • Python 3.13+
  • uv for dependency management
  • MCP SDK for protocol implementation

To set up for development:

  1. Clone the repository
  2. Run uv sync --dev --all-extras
  3. Make your changes
  4. Test with MCP-compatible clients

Troubleshooting

LM Studio Configuration Issues

If you see errors like "Failed to spawn: url-text-fetcher" in LM Studio logs:

  1. Run the configuration helper:
    ./configure_lmstudio.sh
  2. Make sure you're using full paths:
    • Use the full path to uv (e.g., /Users/username/.local/bin/uv)
    • Include the cwd (current working directory) in your configuration
    • Set the BRAVE_API_KEY environment variable
  3. Test the server manually:
    uv run url-text-fetcher
    The server should start and wait for input (press Ctrl+C to exit).
  4. Check your API key:
    # Check if your .env file has the API key set cat .env | grep BRAVE_API_KEY
    Or test manually:
    export BRAVE_API_KEY=your_actual_api_key echo $BRAVE_API_KEY # Should show your key

Common Issues

  • "BRAVE_API_KEY environment variable is required": Make sure your .env file contains BRAVE_API_KEY=your_actual_api_key
  • "Network error": Check your internet connection and API key validity
  • "Content truncated": Normal behavior for very long web pages (content is limited to 5000 characters by default)

Error Handling

The server includes robust error handling for:

  • Network timeouts (10-second default)
  • Invalid URLs
  • HTTP errors (4xx, 5xx responses)
  • Parsing failures
  • Missing API keys
  • General exceptions

All errors are returned as descriptive text messages to help users understand what went wrong.

Development

This project uses:

  • Python 3.13+
  • uv for dependency management
  • MCP SDK for protocol implementation

To set up for development:

  1. Clone the repository
  2. Run uv sync --dev --all-extras
  3. Make your changes
  4. Test with MCP-compatible clients

Debugging

Since MCP servers run over stdio, debugging can be challenging. For the best debugging experience, we strongly recommend using the MCP Inspector.

You can launch the MCP Inspector via npm with this command:

npx @modelcontextprotocol/inspector uv --directory /Users/wallison/TechProjects/mcp-server run url-text-fetcher

Upon launching, the Inspector will display a URL that you can access in your browser to begin debugging.

License

MIT License - see LICENSE file for details

-
security - not tested
A
license - permissive license
-
quality - not tested

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

Enables AI models to fetch text content from URLs, extract links from web pages, and search the web using Brave Search with automatic content retrieval from top results. Provides comprehensive web scraping and search capabilities with robust error handling.

  1. Features
    1. Security Features
      1. Tools
        1. fetch_url_text
        2. fetch_page_links
        3. brave_search_and_fetch
      2. Prerequisites
        1. Brave Search API Key
      3. Installation
        1. Usage
          1. With LM Studio
          2. Standalone Usage
        2. Examples
          1. Dependencies
            1. Configuration
              1. Brave Search Subscription Tiers
            2. Development
              1. Troubleshooting
                1. LM Studio Configuration Issues
                2. Common Issues
              2. Error Handling
                1. Development
                  1. Debugging
                2. License

                  Related MCP Servers

                  • -
                    security
                    F
                    license
                    -
                    quality
                    Enables intelligent web scraping through a browser automation tool that can search Google, navigate to webpages, and extract content from various websites including GitHub, Stack Overflow, and documentation sites.
                    Last updated -
                    1
                    Python
                  • -
                    security
                    A
                    license
                    -
                    quality
                    Enables AI models to search the web for current information before generating responses, with features for conditional searching, geographic customization, and automatic citations.
                    Last updated -
                    JavaScript
                    MIT License
                  • A
                    security
                    A
                    license
                    A
                    quality
                    A powerful tool for fetching and extracting text content from web pages and APIs, supporting web scraping, REST API requests, and Google Custom Search integration.
                    Last updated -
                    4
                    2
                    JavaScript
                    MIT License
                    • Linux

                  View all related MCP servers

                  MCP directory API

                  We provide all the information about MCP servers via our MCP API.

                  curl -X GET 'https://glama.ai/api/mcp/v1/servers/billallison/brsearch-mcp-server'

                  If you have feedback or need assistance with the MCP directory API, please join our Discord server