Docs Fetch MCP Server
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Docs Fetch MCP Server fetch the React documentation starting from react.dev and explore links up to depth 3"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Docs Fetch MCP Server
A Bun-based Model Context Protocol (MCP) server for fetching documentation pages and bounded documentation crawls.
The server exposes one MCP tool, fetch_doc_content. It fetches a root URL, extracts readable Markdown, ranks links, and can crawl linked pages within explicit depth, page, timeout, and scope limits. Results are returned as structured JSON with crawl metadata, page content, ranked links, truncation flags, and per-page errors.
Features
Fast static fetch path with
axiosOptional Puppeteer rendering for client-rendered or thin pages
Markdown extraction with headings, lists, code blocks, tables, blockquotes, and links
Real crawl depth semantics:
depth: 1returns only the root pagedepth: 2includes direct child linksdepth: 3includes grandchildren, up to the maximum of5
URL normalization before dedupe: fragments, tracking params, default ports, and trailing slash duplicates are removed
Crawl scoping by same origin and optional
pathPrefixBounded
maxPages,maxConcurrency, global timeout, per-page timeout, and per-page content truncationPartial results with explicit per-page failures
Runtime argument validation shared with the advertised tool schema
Related MCP server: Driflyte MCP Server
Requirements
Bun
>=1.3.0Puppeteer's browser installation if
renderisautooralways
This project uses Bun for dependency management, runtime execution, builds, and tests.
Installation
bun install
bun run buildConfigure your MCP client:
{
"mcpServers": {
"docs-fetch": {
"command": "bun",
"args": ["/path/to/docs-fetch-mcp/build/index.js"]
}
}
}For local development without a build:
{
"mcpServers": {
"docs-fetch": {
"command": "bun",
"args": ["/path/to/docs-fetch-mcp/src/index.ts"]
}
}
}Tool
fetch_doc_content
Fetch one URL and optionally crawl linked pages.
Parameters:
Name | Type | Default | Limit | Description |
| string | required | HTTP/HTTPS only | Root URL to fetch. |
| number |
|
| Link distance from the root. |
| number |
|
| Maximum pages returned across the crawl. |
| number |
|
| Maximum pages fetched at once. |
| number |
|
| Global crawl timeout. |
| number |
|
| Per-page fetch timeout. |
| string |
|
| Browser rendering strategy. |
| boolean |
| n/a | Restrict crawled links to the same origin. |
| string | omitted | n/a | Optional path prefix crawl scope, such as |
| number |
|
| Markdown characters per page before truncation. |
| boolean |
| n/a | Include ranked links in returned page objects. |
Example request:
{
"url": "https://example.com/docs",
"depth": 2,
"maxPages": 8,
"pathPrefix": "/docs",
"render": "auto"
}Response shape:
{
"rootUrl": "https://example.com/docs",
"normalizedRootUrl": "https://example.com/docs",
"explorationDepth": 2,
"maxPages": 8,
"pagesExplored": 3,
"pagesFailed": 1,
"timedOut": false,
"durationMs": 1234,
"crawl": {
"sameOrigin": true,
"pathPrefix": "/docs",
"maxConcurrency": 3,
"render": "auto",
"perPageTimeoutMs": 10000
},
"content": [
{
"url": "https://example.com/docs",
"finalUrl": "https://example.com/docs",
"depth": 0,
"status": 200,
"title": "Documentation",
"description": "Example documentation",
"canonicalUrl": "https://example.com/docs",
"headings": ["Documentation"],
"content": "# Documentation\n\n...",
"contentLength": 2400,
"truncated": false,
"fetchedWith": "http",
"links": [
{
"url": "https://example.com/docs/api",
"text": "API reference",
"score": 18.5,
"internal": true
}
]
}
],
"errors": [
{
"url": "https://example.com/docs/missing",
"depth": 1,
"error": "Request failed with status code 404",
"status": 404
}
]
}Crawl Behavior
Crawling is breadth-first.
URLs are normalized before dedupe and enqueue.
sameOrigin: truerejects links whose origin differs from the normalized root URL.pathPrefixfurther restricts links to a path prefix on the root origin.includeLinks: falsehides links in returned page objects but does not disable link discovery for crawling.render: "never"uses only the HTTP fetch path.render: "always"uses Puppeteer for every fetched page.render: "auto"tries HTTP first, then uses Puppeteer when HTTP fails or extracted content is very thin.Browser fallback currently preserves the rendered response status and extracts the page body even for non-2xx pages.
Architecture
src/index.ts CLI entrypoint
src/server.ts MCP server wiring and tool registration
src/config/tool-options.ts Shared tool schema/default/range metadata
src/tool/fetch-doc-content-args.ts Runtime argument validation
src/crawler/docs-crawler.ts Crawl orchestration and queue management
src/crawler/page-fetcher.ts HTTP fetch, browser fallback, extraction coordination
src/browser/browser-manager.ts Reusable Puppeteer browser/page handling
src/content/content-extractor.ts Page metadata/content extraction facade
src/content/main-content-selector.ts Main content selection
src/content/link-extractor.ts Link normalization, dedupe, and ranking
src/content/markdown-renderer.ts HTML-to-Markdown rendering
src/content/text-cleanup.ts Shared text normalization helpers
src/utils/url.ts URL normalization and scope utilities
src/types/index.ts Shared TypeScript typesThe MCP schema and runtime option normalization share the same metadata source in src/config/tool-options.ts. Keep new options there first, then wire behavior through validation and crawler options.
Development
bun install
bun run dev
bun run test
bun run typecheck
bun run buildScripts:
bun run dev: run the MCP server from TypeScript source.bun run test: run Bun tests undersrc.bun run typecheck: run TypeScript with--noEmit.bun run build: emitbuild/index.jswith a Bun shebang.bun run start: run the built MCP server.
Testing
Tests are colocated with the modules they cover:
src/crawler/docs-crawler.test.ts: crawl depth, scoping, failures, and link visibility.src/crawler/page-fetcher.test.ts: fetch/render fallback characterization.src/content/content-extractor.test.ts: Markdown extraction and truncation.src/tool/fetch-doc-content-args.test.ts: boundary validation and schema/default sync.src/utils/url.test.ts: URL normalization and scope helpers.
When changing behavior, add or update characterization tests first. For refactors, keep bun run test, bun run typecheck, and bun run build green.
Notes
Use
render: "never"for fast static documentation crawls and tests.Use
pathPrefixfor documentation sites that share a domain with marketing pages, blogs, or apps.Puppeteer browser installation can be skipped only if callers use
render: "never".The custom Markdown renderer is intentionally covered by characterization tests; preserve output compatibility unless making an explicit behavior change.
License
MIT
Available Tools
1 toolfetch_doc_contentC
Fetch web page content with the ability to explore linked pages up to a specified depth
Inputs: url (required), depth.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 1 tool
With only one tool, there is no possibility of ambiguity or overlap between tools. The single tool 'fetch_doc_content' has a clear and distinct purpose, making it impossible for an agent to confuse it with other tools.
Since there is only one tool, it inherently follows a consistent naming pattern. The tool name 'fetch_doc_content' uses a verb_noun structure, which is clear and predictable, and there are no other tools to cause inconsistency.
A single tool is too few for a server named 'Docs Fetch MCP Server', which implies a domain involving document fetching and exploration. While the tool is well-described, one tool feels thin and insufficient for comprehensive operations like managing fetched content or handling different document types, limiting the server's utility.
The tool surface is severely incomplete for the inferred domain of document fetching. It only provides a fetch operation with depth control, missing essential functions such as listing fetched documents, updating or deleting cached content, searching within documents, or handling errors, which are necessary for a complete workflow.
Maintenance
Related MCP Connectors
Scrape websites by following links and extract structured data from natural language instructions.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Search the web, fetch any URL and map a site as clean, LLM-ready Markdown.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables LLMs to fetch and extract web content using browser automation, OCR, and multiple extraction methods, handling JavaScript rendering and anti-scraping techniques.17MIT
- AlicenseNot gradedqualityDmaintenanceEnables AI assistants to query and retrieve topic-specific knowledge from recursively crawled and indexed web pages.297 npm11MIT
- AlicenseNot gradedqualityCmaintenanceEnables agents to discover the llms.txt index covering any site, fetch its linked documentation pages as markdown, and search across parsed indexes by section heading, link title, and notes.MIT
- FlicenseNot gradedqualityAmaintenanceEnables AI agents and LLM pipelines to fetch webpages or crawl sites, extract main content, and convert it to clean, token-efficient Markdown with preserved links. It supports configurable crawling depth, caching, robots.txt compliance, and optional chunking for downstream processing.-