MD Webcrawl MCP

MD MCP Webcrawler Project

A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.

Features

Extract website content and save as markdown files
Map website structure and links
Batch processing of multiple URLs
Configurable output directory

Installation

Clone the repository:

git clone https://github.com/yourusername/webcrawler.git
cd webcrawler

Install dependencies:

pip install -r requirements.txt

Optional: Configure environment variables:

export OUTPUT_PATH=./output  # Set your preferred output directory

Output

Crawled content is saved in markdown format in the specified output directory.

Configuration

The server can be configured through environment variables:

OUTPUT_PATH: Default output directory for saved files
MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)
REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

Claude Set-Up

Install with FastMCP fastmcp install server.py

or user custom settings to run with fastmcp directly

"Crawl Server": {
      "command": "fastmcp",
      "args": [
        "run",
        "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
      ],
      "env": {
        "OUTPUT_PATH": "/Users/user/Webcrawl"
      }

Development

Live Development

fastmcp dev server.py --with-editable .

Debug

It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging

Examples

Example 1: Extract and Save Content

mcp call extract_content --url "https://example.com" --output_path "example.md"

Example 2: Create Content Index

mcp call scan_linked_content --url "https://example.com" | \
  mcp call create_index --content_map - --output_path "index.md"

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Requirements

Python 3.7+
FastMCP (uv pip install fastmcp)
Dependencies listed in requirements.txt

This server cannot be installed

security - not tested

license - permissive license

quality - not tested

How are these scores calculated?

hybrid server

The server is able to function both locally and remotely, depending on the configuration or use case.

基于 Python 的 MCP 服务器，可抓取网站以提取内容并将其保存为 markdown 文件，并具有映射网站结构和链接的功能。

Related Resources

Reddit Discussion about this server

Related MCP Servers

MCP NPX Fetch
tokenizin-agency
A
security
A
license
A
quality
A powerful MCP server for fetching and transforming web content into various formats (HTML, JSON, Markdown, Plain Text) with ease.
Last updated -
4
1,145
36
MIT License
UseScraper MCP Server
tanevanwifferen
A
security
A
license
A
quality
A TypeScript-based MCP server utilizing the UseScraper API to provide web scraping capabilities, allowing users to extract content from webpages in various formats.
Last updated -
2
MIT License
Mozilla Readability Parser MCP Server
jmh108
-
security
A
license
-
quality
A Python implementation of an MCP server that extracts webpage content, removes ads and non-essential elements, and transforms it into clean, LLM-optimized Markdown.
Last updated -
2
MIT License
Documentation Crawler & MCP Server
alizdavoodi
-
security
A
license
-
quality
Toolset that crawls websites, generates Markdown documentation, and makes that documentation searchable via a Model Context Protocol (MCP) server for integration with tools like Cursor.
Last updated -
24
MIT License

View all related MCP servers