MD Webcrawl MCP

MD MCP Webcrawler Project

A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.

Features

Extract website content and save as markdown files
Map website structure and links
Batch processing of multiple URLs
Configurable output directory

Installation

Clone the repository:

git clone https://github.com/yourusername/webcrawler.git
cd webcrawler

Install dependencies:

pip install -r requirements.txt

Optional: Configure environment variables:

export OUTPUT_PATH=./output  # Set your preferred output directory

Output

Crawled content is saved in markdown format in the specified output directory.

Configuration

The server can be configured through environment variables:

OUTPUT_PATH: Default output directory for saved files
MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)
REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

Claude Set-Up

Install with FastMCP fastmcp install server.py

or user custom settings to run with fastmcp directly

"Crawl Server": {
      "command": "fastmcp",
      "args": [
        "run",
        "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
      ],
      "env": {
        "OUTPUT_PATH": "/Users/user/Webcrawl"
      }

Development

Live Development

fastmcp dev server.py --with-editable .

Debug

It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging

Examples

Example 1: Extract and Save Content

mcp call extract_content --url "https://example.com" --output_path "example.md"

Example 2: Create Content Index

mcp call scan_linked_content --url "https://example.com" | \
  mcp call create_index --content_map - --output_path "index.md"

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Requirements

Python 3.7+
FastMCP (uv pip install fastmcp)
Dependencies listed in requirements.txt

This server cannot be installed

security - not tested

license - permissive license

quality - not tested

hybrid server

The server is able to function both locally and remotely, depending on the configuration or use case.

A Python-based MCP server that crawls websites to extract and save content as markdown files, with features for mapping website structure and links.

Related Resources

Reddit Discussion about this server

Related MCP Servers

MCP NPX Fetch
tokenizin-agency
A
security
A
license
A
quality
A powerful MCP server for fetching and transforming web content into various formats (HTML, JSON, Markdown, Plain Text) with ease.
Last updated -
4
146
12
TypeScript
MIT License
Markdown Downloader
dazeb
A
security
A
license
A
quality
An MCP server that enables users to download webpages as markdown files using r.jina.ai service, with features for configurable download directories and automatic date-stamped filenames.
Last updated -
5
2
25
JavaScript
MIT License
Mozilla Readability Parser MCP Server
jmh108
-
security
A
license
-
quality
A Python implementation of an MCP server that extracts webpage content, removes ads and non-essential elements, and transforms it into clean, LLM-optimized Markdown.
Last updated -
1
Python
MIT License
Website Scraper MCP Server
tolik-unicornrider
-
security
F
license
-
quality
An MCP server that extracts meaningful content from websites and converts HTML to high-quality Markdown, using Mozilla's Readability engine.
Last updated -
11,993
2
JavaScript

View all related MCP servers