Docs Fetch MCP Server

A Model Context Protocol (MCP) server for fetching web content with recursive exploration capabilities. This server enables LLMs to autonomously explore web pages and documentation to learn about specific topics.

Overview

The Docs Fetch MCP Server provides a simple but powerful way for LLMs to retrieve and explore web content. It enables:

Fetching clean, readable content from any web page
Recursive exploration of linked pages up to a specified depth
Same-domain link traversal to gather comprehensive information
Smart filtering of navigation links to focus on content-rich pages

This tool is particularly useful when users want an LLM to learn about a specific topic by exploring documentation or web content.

Features

Content Extraction: Cleanly extracts the main content from web pages, removing distractions like navigation, ads, and irrelevant elements
Link Analysis: Identifies and extracts links from the page, assessing their relevance
Recursive Exploration: Follows links to related content within the same domain, up to a specified depth
Parallel Processing: Efficiently crawls content with concurrent requests and proper error handling
Robust Error Handling: Gracefully handles network issues, timeouts, and malformed pages
Dual-Strategy Approach: Uses fast axios requests first with puppeteer as a fallback for more complex pages
Timeout Prevention: Implements global timeout handling to ensure reliable operation within MCP time limits
Partial Results: Returns available content even when some pages fail to load completely

Usage

The server exposes a single MCP tool:

`fetch_doc_content`

Fetches web page content with the ability to explore linked pages up to a specified depth.

Parameters:

url (string, required): URL of the web page to fetch
depth (number, optional, default: 1): Maximum depth of directory/link exploration (1-5)

Returns:

{
  "rootUrl": "https://example.com/docs",
  "explorationDepth": 2,
  "pagesExplored": 5,
  "content": [
    {
      "url": "https://example.com/docs",
      "title": "Documentation",
      "content": "Main page content...",
      "links": [
        {
          "url": "https://example.com/docs/topic1",
          "text": "Topic 1"
        },
        ...
      ]
    },
    ...
  ]
}

Installation

Clone this repository:

git clone https://github.com/wolfyy970/docs-fetch-mcp.git
cd docs-fetch-mcp

Install dependencies:

npm install

Build the project:

npm run build

Configure your MCP settings in your Claude Client:

{
  "mcpServers": {
    "docs-fetch": {
      "command": "node",
      "args": [
        "/path/to/docs-fetch-mcp/build/index.js"
      ],
      "env": {
        "MCP_TRANSPORT": "pipe"
      }
    }
  }
}

Dependencies

@modelcontextprotocol/sdk: MCP server SDK
puppeteer: Headless browser for web page interaction
axios: HTTP client for making requests

Development

To run the server in development mode:

npm run dev

License

MIT

This server cannot be installed

security - not tested

license - permissive license

quality - not tested

remote-capable server

The server can be hosted and run remotely because it primarily relies on remote services or has no dependency on the local environment.

Enables LLMs to autonomously retrieve and explore web content by fetching pages and recursively following links to a specified depth, particularly useful for learning about topics from documentation.

Related MCP Servers

MCP JinaAI Reader Server
spences10
A
security
A
license
A
quality
Integrates Jina.ai's Reader API with LLMs for efficient and structured web content extraction, optimized for documentation and web content analysis.
Last updated -
1
24
24
JavaScript
MIT License
Fetch MCP Server
ExactDoug
-
security
A
license
-
quality
Enables retrieval and processing of web page content for LLMs by converting HTML to markdown, with support for content truncation and pagination.
Last updated -
1
1
Python
MIT License
Tavily MCP Server with Proxy Support
tulong66
-
security
F
license
-
quality
Enables LLMs to perform sophisticated web searches through proxy servers using Tavily's API, supporting comprehensive web searches, direct question answering, and recent news article retrieval with AI-extracted content.
Last updated -
1
Python
SEO Tools MCP Server
Skobyn
-
security
F
license
-
quality
Enables LLMs to interact with DataForSEO and other SEO APIs through natural language, allowing for keyword research, SERP analysis, backlink analysis, and local SEO tasks.
Last updated -
9
TypeScript

View all related MCP servers

Docs Fetch MCP Server