Skip to main content
Glama
Sign In
enesjakozh

MD Webcrawl MCP

by jmh108
GitHub
Python
MIT License
1
OverviewInspectNewSchemaRelated ServersReviewsScore
Need Help?View Source CodeReport Issue

Integrations

  • Allows for cloning the repository from GitHub as part of the installation process

  • Extracts website content and saves it in markdown format, enabling structured documentation from crawled websites

MD MCP Webcrawler Project

A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.

Features

  • Extract website content and save as markdown files
  • Map website structure and links
  • Batch processing of multiple URLs
  • Configurable output directory

Installation

  1. Clone the repository:
git clone https://github.com/yourusername/webcrawler.git cd webcrawler
  1. Install dependencies:
pip install -r requirements.txt
  1. Optional: Configure environment variables:
export OUTPUT_PATH=./output # Set your preferred output directory

Output

Crawled content is saved in markdown format in the specified output directory.

Configuration

The server can be configured through environment variables:

  • OUTPUT_PATH: Default output directory for saved files
  • MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)
  • REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

Claude Set-Up

Install with FastMCP fastmcp install server.py

or user custom settings to run with fastmcp directly

"Crawl Server": { "command": "fastmcp", "args": [ "run", "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py" ], "env": { "OUTPUT_PATH": "/Users/user/Webcrawl" }

Development

Live Development

fastmcp dev server.py --with-editable .

Debug

It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging

Examples

Example 1: Extract and Save Content

mcp call extract_content --url "https://example.com" --output_path "example.md"

Example 2: Create Content Index

mcp call scan_linked_content --url "https://example.com" | \ mcp call create_index --content_map - --output_path "index.md"

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Requirements

  • Python 3.7+
  • FastMCP (uv pip install fastmcp)
  • Dependencies listed in requirements.txt

This server cannot be installed

-
security - not tested
A
license - permissive license
-
quality - not tested

local-only server

The server can only run on the client's local machine because it depends on local resources.

A Python-based MCP server that crawls websites to extract and save content as markdown files, with features for mapping website structure and links.

  1. Features
    1. Installation
      1. Output
        1. Configuration
          1. Claude Set-Up
            1. Development
              1. Live Development
              2. Debug
            2. Examples
              1. Example 1: Extract and Save Content
              2. Example 2: Create Content Index
            3. Contributing
              1. License
                1. Requirements

                  Related Resources

                  Related MCP Servers

                  • MCP NPX Fetch

                    tokenizin-agency
                    A
                    security
                    A
                    license
                    A
                    quality
                    A powerful MCP server for fetching and transforming web content into various formats (HTML, JSON, Markdown, Plain Text) with ease.
                    Last updated -
                    4
                    146
                    12
                    TypeScript
                    MIT License
                    • Apple
                    • Linux

                  • Markdown Downloader

                    dazeb
                    A
                    security
                    A
                    license
                    A
                    quality
                    An MCP server that enables users to download webpages as markdown files using r.jina.ai service, with features for configurable download directories and automatic date-stamped filenames.
                    Last updated -
                    5
                    11
                    23
                    JavaScript
                    MIT License

                  • Mozilla Readability Parser MCP Server

                    jmh108
                    -
                    security
                    A
                    license
                    -
                    quality
                    A Python implementation of an MCP server that extracts webpage content, removes ads and non-essential elements, and transforms it into clean, LLM-optimized Markdown.
                    Last updated -
                    1
                    Python
                    MIT License
                    • Linux
                    • Apple

                  • Website Scraper MCP Server

                    tolik-unicornrider
                    -
                    security
                    F
                    license
                    -
                    quality
                    An MCP server that extracts meaningful content from websites and converts HTML to high-quality Markdown, using Mozilla's Readability engine.
                    Last updated -
                    11,993
                    2
                    JavaScript

                  View all related MCP servers

                  Appeared in Searches

                  New MCP Servers

                  ID: saglkval3f