Better Fetch - Advanced Web Content MCP Server

A powerful Model Context Protocol (MCP) server that intelligently fetches and processes web content with nested URL crawling capabilities. Transform any documentation site or web resource into clean, structured markdown files perfect for AI consumption and analysis.

🚀 Key Features

🕸️ Smart Web Crawling

• Nested URL Fetching: Automatically discovers and crawls linked pages up to configurable depth

• Single Page Mode: Option for simple single-page content extraction

• Domain Filtering: Stay within the same domain or allow cross-domain crawling

• Pattern Matching: Include/exclude URLs based on regex patterns

🧠 Intelligent Content Processing

• Content Cleaning: Removes ads, navigation, scripts, and other noise automatically

• Smart Section Detection: Identifies main content areas (<main>, <article>, .content)

• Automatic Titles: Generates meaningful section headers based on page titles and URL structure

• Table of Contents: Creates organized TOC with proper nesting

📝 Advanced Markdown Generation

• Clean Formatting: Converts HTML to well-structured markdown

• Code Block Preservation: Maintains formatting for code snippets and technical content

• Link Preservation: Keeps all important links with proper markdown syntax

• Metadata Integration: Includes source URLs, generation timestamps, and site information

⚙️ Highly Configurable

• Crawl Depth Control: Set maximum levels to crawl (default: 2)

• Page Limits: Control maximum pages to process (default: 50)

• Timeout Settings: Configurable request timeouts

• Respectful Crawling: Built-in delays between requests

• Error Handling: Graceful handling of failed requests and invalid URLs

📋 Available Tools

1. fetch_website_nested

Comprehensive web crawling with nested URL processing.

Parameters:

• url (required): Starting URL to crawl

• maxDepth (optional, default: 2): Maximum crawl depth

• maxPages (optional, default: 50): Maximum pages to process

• sameDomainOnly (optional, default: true): Restrict to same domain

• excludePatterns (optional): Array of regex patterns to exclude

• includePatterns (optional): Array of regex patterns to include

• timeout (optional, default: 10000): Request timeout in milliseconds

2. fetch_website_single

Simple single-page content extraction.

Parameters:

• url (required): URL to fetch

• timeout (optional, default: 10000): Request timeout in milliseconds

💡 Use Cases

📚 Documentation Processing

• API Documentation: Convert REST API docs, SDK guides, and technical references

• Framework Docs: Process React, Vue, Angular, or any framework documentation

• Library Guides: Extract comprehensive guides from library documentation sites

• Tutorial Series: Gather multi-part tutorials into single organized documents

🔍 Content Analysis & Research

• Competitive Analysis: Gather competitor documentation and feature descriptions

• Market Research: Extract product information from multiple related pages

• Academic Research: Collect and organize web-based research materials

• Knowledge Base Creation: Transform scattered web content into structured knowledge bases

🤖 AI Training & Context

• LLM Context Preparation: Create clean, structured content for AI model training

• RAG System Input: Generate high-quality documents for Retrieval-Augmented Generation

• Chatbot Knowledge: Build comprehensive knowledge bases for customer service bots

• Content Summarization: Prepare web content for automated summarization tasks

🛠️ Installation & Setup

Installing via Smithery

To install Better Fetch for Claude Desktop automatically via Smithery:

Prerequisites

• Node.js 18+

• npm or yarn

• MCP-compatible client (Claude Desktop, VS Code with MCP extension, etc.)

Step 1: Clone and Install

Step 2: Build the Project

Step 3: Test the Server (Optional)

Step 4: Configure Your MCP Client

For Claude Desktop:

Add to your claude_desktop_config.json:

For VS Code MCP Extension:

For Custom MCP Client:

📖 Usage Examples

Basic Documentation Crawling

Advanced Configuration

Single Page Extraction

📄 Sample Output

The server generates comprehensive markdown files with the following structure:

For a complete example, refer to output.md which demonstrates the server's output when processing a real documentation site.

🔧 Development

Project Structure

Available Scripts

Testing Your Changes

🚦 Performance & Limits

Default Limits

• Max Depth: 2 levels (configurable)

• Max Pages: 50 pages (configurable)

• Request Timeout: 10 seconds (configurable)

• Crawl Delay: 500ms between requests (respectful crawling)

Performance Tips

• Set appropriate maxPages limits for large sites

• Use includePatterns to focus on relevant content

• Enable sameDomainOnly to avoid external link crawling

• Adjust timeout based on target site response times

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

1. Fork the repository

2. Create a feature branch: git checkout -b feature/amazing-feature

3. Make your changes and add tests

4. Commit your changes: git commit -m 'Add amazing feature'

5. Push to the branch: git push origin feature/amazing-feature

6. Open a Pull Request

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🆘 Support & Issues

• Bug Reports: GitHub Issues

• Feature Requests: GitHub Discussions

• Documentation: Check the Wiki

🙏 Acknowledgments

• Built with the Model Context Protocol SDK

• Powered by Cheerio for HTML parsing

• Markdown conversion by Turndown

Made with ❤️ for the AI and developer community