Website to Markdown MCP Server

README.md
# 🌐 Website to Markdown MCP Server

<div align="center">

**Language**: [English](README.md) | [繁體中文](README.zh-TW.md)

</div>

> A powerful Model Context Protocol (MCP) server designed for fetching website content and converting it to Markdown format, making it easier for AI to understand and process website information.

## ✨ Key Features

<div align="center">

| 🌟 Enhanced Processing | 📊 OpenAPI Support | ⚙️ Smart Analysis | 🎯 Advanced Extraction |
|:--------:|:-------------:|:----------:|:----------:|
| AI-powered content cleanup | OpenAPI 3.x/Swagger 2.0 | Reading time calculation | Main content detection |
| Auto ad removal | Professional validation | Word count statistics | Language detection |
| Content summarization | Structured API parsing | Smart retry mechanism | Multi-format support |

</div>

---

## 🆕 What's New in v1.2.0

<div align="center">

### 🚀 Major Enhancements

</div>

| Feature | Status | Description |
|:-----|:----------:|:-----|
| 🧠 **Enhanced Content Processor** | ✅ | AI-powered content cleaning and extraction |
| 📊 **Smart Analytics** | ✅ | Word count, reading time, content summary |
| 🌍 **Language Detection** | ✅ | Automatic language identification |
| 🎯 **Intelligent Retry** | ✅ | Smart retry mechanism with exponential backoff |
| 🔍 **Stealth Browser** | ✅ | Anti-detection browsing capabilities |
| ⚡ **Rate Limiting** | ✅ | Built-in rate limiting and concurrency control |
| 🧹 **Content Cleanup** | ✅ | Remove ads, navigation, and irrelevant content |
| 📝 **Enhanced Markdown** | ✅ | Support for strikethrough, underline, highlights |

---

## 🚀 Quick Start

### 🎯 Method 1: NPX Installation (🌟 Recommended)

> 💡 **Easiest way**: No local installation needed!

#### **Step 1**: Create Configuration File 📄

Create a `my-websites.json` file:

```json
{
  "websites": [
    {
      "name": "your_website",
      "url": "https://your-website.com",
      "description": "Your Project Website"
    },
    {
      "name": "api_docs",
      "url": "https://api.example.com/openapi.json",
      "description": "Your API Specification"
    }
  ]
}
```

#### **Step 2**: Configure MCP Server ⚙️

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "command": "npx",
      "args": ["-y", "website-to-markdown-mcp"],
      "disabled": false,
      "env": {
        "WEBSITES_CONFIG_PATH": "./my-websites.json"
      }
    }
  }
}
```

#### **Step 3**: Restart and Test 🔄

1. **Restart Cursor**
2. **Open Chat and use Agent mode**
3. **Test command**: `Please list all configured websites`

<div align="center">

**🎉 Done! No installation required!**

</div>

---

### 🎯 Method 2: Local Installation

> 💡 **Best Practice**: Use this method for development or customization!

#### **Step 1**: Clone and Build

```bash
git clone https://github.com/your-username/website-to-markdown-mcp.git
cd website-to-markdown-mcp
npm install
npm run build
```

#### **Step 2**: Configure MCP Server

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "command": "cmd",
      "args": ["/c", "node", "./website-to-markdown-mcp/dist/index.js"],
      "disabled": false,
      "env": {
        "WEBSITES_CONFIG_PATH": "./my-websites.json"
      }
    }
  }
}
```

---

## 🔥 Enhanced Output Features

### 📊 Rich Content Analysis

Every fetched content now includes:

- **📝 Content Summary**: AI-generated summary of the main content
- **⏱️ Reading Time**: Estimated reading time based on content length
- **🔢 Word Count**: Accurate word count for both English and Chinese
- **🌍 Language Detection**: Automatic language identification
- **🎯 Content Quality Score**: Assessment of content relevance

### 📋 Enhanced Markdown Output

```markdown
# 🚀 Example Website

**Source**: https://example.com
**Website**: example_site - Example Website
**📊 Reading Time**: 5 minutes
**🔢 Word Count**: 1,250 words
**🌍 Language**: English
**📝 Summary**: This article discusses the latest developments in web technology...

---

[Enhanced Markdown content with better formatting...]
```

---

## 🆕 Complete OpenAPI/Swagger Support

<div align="center">

### 🔥 Professional API Documentation

</div>

| Feature | OpenAPI 3.x | Swagger 2.0 | Description |
|:-----|:----------:|:-----------:|:-----|
| 🔍 **Auto Detection** | ✅ | ✅ | Support JSON/YAML formats |
| ✅ **Professional Validation** | ✅ | ✅ | Using `@readme/openapi-parser` |
| 📋 **Structured Parsing** | ✅ | ✅ | Endpoints, parameters, responses |
| 🔗 **Reference Resolution** | ✅ | ✅ | Auto handle `$ref` references |
| 📊 **Smart Summary** | ✅ | ✅ | Generate API overview |
| 📝 **Formatted Output** | ✅ | ✅ | Readable Markdown |

### 🌟 Pre-configured Example Websites

```json
{
  "websites": [
    {
      "name": "petstore_openapi",
      "url": "https://petstore3.swagger.io/api/v3/openapi.json",
      "description": "🐕 Swagger Petstore OpenAPI 3.0 Spec (Demo)"
    },
    {
      "name": "petstore_swagger",
      "url": "https://petstore.swagger.io/v2/swagger.json",
      "description": "🐱 Swagger Petstore Swagger 2.0 Spec (Demo)"
    },
    {
      "name": "github_api",
      "url": "https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json",
      "description": "🐙 GitHub REST API OpenAPI Spec"
    }
  ]
}
```

---

## 📦 Installation & Setup

### 🛠️ System Requirements

- **Node.js** 20.18.1+ (Recommended: v22.15.0 LTS) 
- **npm** 10.0.0+ or **yarn**
- **Cursor** Editor

> ⚠️ **Important**: Some dependencies require Node.js v20.18.1 or higher. Please update your Node.js version if you encounter engine compatibility warnings.

### ⚡ NPM Package Installation

```bash
# Global installation
npm install -g website-to-markdown-mcp

# Or use directly with npx (recommended)
npx website-to-markdown-mcp
```

### 🔧 Development Setup

```bash
# 1. Clone repository
git clone https://github.com/your-username/website-to-markdown-mcp.git
cd website-to-markdown-mcp

# 2. Install dependencies
npm install

# 3. Build project
npm run build
```

### 🎛️ Advanced Configuration Options

<div align="center">

#### Configuration Priority Order

</div>

```mermaid
graph TD
    A[🔍 Check Environment Variable<br/>WEBSITES_CONFIG_PATH] --> B{File exists?}
    B -->|Yes| C[✅ Load External Config File]
    B -->|No| D[🔍 Check Environment Variable<br/>WEBSITES_CONFIG]
    D --> E{Valid JSON?}
    E -->|Yes| F[✅ Load Embedded Config]
    E -->|No| G[🔍 Check config.json]
    G --> H{File exists?}
    H -->|Yes| I[✅ Load Local Config]
    H -->|No| J[🔧 Use Default Config]
```

---

## 🎨 Configuration Method Details

### 📋 Method 1: External Configuration File (🌟 Recommended)

> 💡 **Advantages**: Easy to edit, syntax highlighting, version control friendly

<details>
<summary><b>🔧 Detailed Setup Steps</b></summary>

1. **Create Configuration File**
   ```bash
   # Can be placed anywhere
   touch my-api-configs.json
   ```

2. **Edit Configuration Content**
   ```json
   {
     "websites": [
       {
         "name": "my_docs",
         "url": "https://docs.example.com",
         "description": "📚 My Documentation Website"
       }
     ]
   }
   ```

3. **Set Environment Variable**
   ```json
   {
     "env": {
       "WEBSITES_CONFIG_PATH": "./my-api-configs.json"
     }
   }
   ```

</details>

### 📋 Method 2: Embedded JSON (Backward Compatible)

<details>
<summary><b>🔧 Configuration Example</b></summary>

```json
{
  "mcpServers": {
    "website-to-markdown": {
      "command": "cmd",
      "args": ["/c", "node", "./website-to-markdown-mcp/dist/index.js"],
      "disabled": false,
      "env": {
        "WEBSITES_CONFIG": "{\"websites\":[{\"name\":\"example\",\"url\":\"https://example.com\",\"description\":\"Example Website\"}]}"
      }
    }
  }
}
```

</details>

### 📋 Method 3: Local config.json

<details>
<summary><b>🔧 Local Configuration</b></summary>

Directly edit `config.json` in the project root directory:

```json
{
  "websites": [
    {
      "name": "local_site",
      "url": "https://local.example.com",
      "description": "🏠 Local Test Website"
    }
  ]
}
```

</details>

---

## 🔧 Available Tools

### 🌐 General Tools

| Tool Name | Function | Parameters | Example |
|:--------|:-----|:-----|:-----|
| `fetch_website` | Fetch any website | `url`: Website URL | Fetch OpenAPI spec files |
| `list_configured_websites` | List configured websites | None | View all available websites |

### 🎯 Dedicated Tools

Each configured website automatically generates corresponding dedicated tools:

- `fetch_petstore_openapi` - Fetch Petstore OpenAPI 3.0 spec
- `fetch_petstore_swagger` - Fetch Petstore Swagger 2.0 spec  
- `fetch_github_api` - Fetch GitHub API spec
- `fetch_tailwind_css` - Fetch Tailwind CSS documentation

---

## 📊 Enhanced Output Format Examples

### 🌐 General Website Content with Analytics

```markdown
# Website Title

**Source**: https://example.com
**Website**: example_site - Example Website
**📊 Reading Time**: 3 minutes
**🔢 Word Count**: 650 words
**🌍 Language**: English
**📝 Summary**: This article provides a comprehensive overview of modern web development practices, covering frontend frameworks, backend technologies, and deployment strategies.

---

[Enhanced cleaned Markdown content with ads removed and main content extracted...]
```

### 📋 OpenAPI 3.x Specification File

```markdown
# 🚀 Example API (v2.1.0)

**Source**: https://api.example.com/openapi.json
**OpenAPI Version**: 3.0.3
**Validation Status**: ✅ Valid
**📊 Processing Time**: 1.2 seconds
**🔢 Endpoints**: 25 endpoints
**🌍 Server Locations**: 3 servers

---

## 📋 API Basic Information

- **API Name**: Example API
- **Version**: 2.1.0
- **OpenAPI Version**: 3.0.3
- **Description**: A powerful example API for modern applications

## 🌐 Servers

1. **https://api.example.com**
   - 🏢 Production server
2. **https://staging-api.example.com**
   - 🧪 Testing server

## 🛠️ API Endpoints

Total of **25** endpoints:

### 👥 `/users`
- **GET**: Get user list
- **POST**: Create new user

### 🔍 `/users/{id}`
- **GET**: Get specific user
- **PUT**: Update user information
- **DELETE**: Delete user

## 🧩 Components

- **Schemas**: 12 data models
- **Parameters**: 8 reusable parameters  
- **Responses**: 15 reusable responses
- **Security Schemes**: 3 security mechanisms
```

---

## 🎯 Usage Examples

### 💻 Basic Usage

```
Please fetch the content from https://docs.example.com and convert to markdown
```

### 🔍 OpenAPI Specification Fetching

```
Please use the fetch_petstore_openapi tool to fetch Petstore OpenAPI specification
```

### 📚 Documentation Website Fetching

```
Please fetch React official documentation content
```

---

## 🚨 Troubleshooting

> 📋 **Complete Troubleshooting Guide**: See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for detailed solutions to common issues.

### ❓ Quick Solutions

<details>
<summary><b>🔧 Node.js Version Issues</b></summary>

**Error**: `npm WARN EBADENGINE Unsupported engine`
- **Solution**: Update Node.js to v20.18.1 or higher
- **Download**: [Node.js Official Website](https://nodejs.org/)
- **Verify**: `node --version`

</details>

<details>
<summary><b>🌐 Module Not Found Issues</b></summary>

**Error**: `Cannot find module './db.json'`
- **Solution 1**: Clear npm cache: `npm cache clean --force`
- **Solution 2**: Update Node.js version
- **Solution 3**: Use local installation instead of npx

</details>

<details>
<summary><b>⚙️ Configuration Issues</b></summary>

**Q: Configuration changes not taking effect?**
- ✅ Confirm JSON format is correct
- ✅ Restart Cursor
- ✅ Check environment variable names

**Q: JSON format errors?**
- 🛠️ Use [JSON Validator](https://jsonlint.com/)
- 🛠️ Confirm using double quotes
- 🛠️ Check for extra commas

</details>

### 🔍 Debug Mode

Detailed logs are output to stderr at startup:

```bash
# View debug messages
npm run dev 2> debug.log
```

---

## 📈 Performance & Optimization

### ⚡ Performance Features

- 🚀 **Smart Retry**: Intelligent retry with exponential backoff
- 💾 **Rate Limiting**: Built-in rate limiting to prevent overload
- 🎯 **Content Filtering**: Remove irrelevant content for faster processing
- 🧹 **Ad Removal**: Automatic ad and popup removal
- 📊 **Stealth Mode**: Anti-detection browsing capabilities

### 🛡️ Security Considerations

- 🔒 HTTPS websites only (recommended)
- 🛠️ Auto filter malicious scripts
- 📝 Limit output content length
- 🔐 Stealth browsing to avoid detection

---

## 📦 Dependencies

<div align="center">

| Package | Version | Purpose |
|:-----|:----:|:-----|
| `@modelcontextprotocol/sdk` | ^1.0.0 | MCP Core Framework |
| `@readme/openapi-parser` | ^4.1.0 | Professional OpenAPI Parsing |
| `axios` | ^1.6.0 | HTTP Request Handling |
| `cheerio` | ^1.0.0 | HTML Parsing Engine |
| `turndown` | ^7.1.2 | HTML to Markdown |
| `yaml` | ^2.8.0 | YAML Format Support |
| `zod` | ^3.22.0 | Data Validation Framework |
| `playwright` | ^1.40.0 | Browser automation |

</div>

---

## 📝 Changelog

### 🎉 v1.2.0 (Latest)

<div align="center">

**🚀 Major Feature Updates**

</div>

- ✨ **Added** Enhanced content processing with AI-powered cleanup
- ✨ **Added** Smart analytics: word count, reading time, content summary
- ✨ **Added** Language detection and multi-language support
- ✨ **Added** Stealth browser capabilities for anti-detection
- ✨ **Added** Built-in rate limiting and retry mechanisms
- ✨ **Added** Advanced content filtering and ad removal
- 🔧 **Enhanced** Markdown processing with more HTML element support
- 📊 **Improved** Output format with rich metadata
- 🎯 **Fixed** Various technical issues and dependencies

### 🎯 v1.1.0 (Previous)

<div align="center">

**🚀 Major Feature Updates**

</div>

- ✨ **Added** Full OpenAPI 3.x/Swagger 2.0 support
- ✨ **Added** JSON/YAML format auto-detection  
- ✨ **Added** Professional-grade spec validation and reference resolution
- ✨ **Added** Version auto-adaptation mechanism
- ✨ **Added** Structured API documentation summary
- 🔧 **Pre-configured** Multiple OpenAPI/Swagger examples
- 📦 **Added** NPM package distribution with npx support
- 🎯 **Enhanced** Installation methods for better user experience

### 🎯 v1.0.0 (Stable)

- 🎉 **Initial Release**
- 🌐 **Basic Functions** Website content fetching
- 📝 **Core Functions** Markdown conversion  
- ⚙️ **Configuration Support** Multi-website management

---

## 🤝 Contributing

### 💡 How to Contribute

1. **🍴 Fork** this project
2. **🌟 Create** feature branch (`git checkout -b feature/AmazingFeature`)
3. **📝 Commit** changes (`git commit -m 'Add some AmazingFeature'`)
4. **📤 Push** to branch (`git push origin feature/AmazingFeature`)
5. **🔄 Open** Pull Request

### 🐛 Issue Reporting

Report issues on the [Issues](https://github.com/your-repo/issues) page, please include:

- 🔍 **Issue Description**
- 🔄 **Reproduction Steps**  
- 💻 **Environment Information**
- 📸 **Screenshots or Logs**

---

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

---

<div align="center">

### 🌟 If this project helps you, please give it a Star!

**💬 Have questions or suggestions? Feel free to open an Issue!**

---

**Made by Sun** ❤️ **for the Developer Community**

</div>
Deploy Server
One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd
TDQS

B3/5.0
Scored 2026-04-18 07:36 across 4 tools
Disambiguation2/5
There is significant overlap and ambiguity among the fetch tools. fetch_example_site and fetch_test_site are essentially special cases of fetch_website, making it unclear when to use which. An agent might struggle to choose between them, as the descriptions don't clarify distinct use cases beyond different hardcoded URLs.
Naming Consistency5/5
Tool names follow a consistent verb_noun pattern throughout (fetch_example_site, fetch_test_site, fetch_website, list_configured_websites). All use snake_case and start with descriptive verbs, making the naming predictable and easy to understand.
Tool Count3/5
With 4 tools, the count is borderline for the server's purpose of converting websites to markdown. It feels slightly thin, as there could be more operations like updating configurations or handling errors, but it covers basic fetching and listing. It's not extreme, but could be more robust.
Completeness2/5
There are significant gaps in the tool surface. The server lacks tools for configuring or managing websites beyond listing them (e.g., add_website, remove_website, update_configuration). This incompleteness will likely cause agent failures when trying to perform full lifecycle operations, as the current set is limited to fetching and listing only.
Maintenance

ActivityInactive
ResponsivenessNo issues
Resources

Need Help?
Related Servers