MD MCP Webcrawler Project

A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.

Features

• Extract website content and save as markdown files

• Map website structure and links

• Batch processing of multiple URLs

• Configurable output directory

Related MCP server: UseScraper MCP Server

Installation

1. Clone the repository:

2. Install dependencies:

3. Optional: Configure environment variables:

Output

Crawled content is saved in markdown format in the specified output directory.

Configuration

The server can be configured through environment variables:

• OUTPUT_PATH: Default output directory for saved files

• MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)

• REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

Claude Set-Up

Install with FastMCP fastmcp install server.py

or user custom settings to run with fastmcp directly

Development

Live Development

Debug

It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging

Examples

Example 1: Extract and Save Content

Example 2: Create Content Index

Contributing

1. Fork the repository

2. Create a feature branch (git checkout -b feature/AmazingFeature)

3. Commit your changes (git commit -m 'Add some AmazingFeature')

4. Push to the branch (git push origin feature/AmazingFeature)

5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Requirements

• Python 3.7+

• FastMCP (uv pip install fastmcp)

• Dependencies listed in requirements.txt