# 🌐 Website to Markdown MCP Server
<div align="center">
**Language**: [English](README.md) | [繁體中文](README.zh-TW.md)
</div>
> A powerful Model Context Protocol (MCP) server designed for fetching website content and converting it to Markdown format, making it easier for AI to understand and process website information.
## ✨ Key Features
<div align="center">
| 🌟 Enhanced Processing | 📊 OpenAPI Support | ⚙️ Smart Analysis | 🎯 Advanced Extraction |
|:--------:|:-------------:|:----------:|:----------:|
| AI-powered content cleanup | OpenAPI 3.x/Swagger 2.0 | Reading time calculation | Main content detection |
| Auto ad removal | Professional validation | Word count statistics | Language detection |
| Content summarization | Structured API parsing | Smart retry mechanism | Multi-format support |
</div>
---
## 🆕 What's New in v1.2.0
<div align="center">
### 🚀 Major Enhancements
</div>
| Feature | Status | Description |
|:-----|:----------:|:-----|
| 🧠 **Enhanced Content Processor** | ✅ | AI-powered content cleaning and extraction |
| 📊 **Smart Analytics** | ✅ | Word count, reading time, content summary |
| 🌍 **Language Detection** | ✅ | Automatic language identification |
| 🎯 **Intelligent Retry** | ✅ | Smart retry mechanism with exponential backoff |
| 🔍 **Stealth Browser** | ✅ | Anti-detection browsing capabilities |
| ⚡ **Rate Limiting** | ✅ | Built-in rate limiting and concurrency control |
| 🧹 **Content Cleanup** | ✅ | Remove ads, navigation, and irrelevant content |
| 📝 **Enhanced Markdown** | ✅ | Support for strikethrough, underline, highlights |
---
## 🚀 Quick Start
### 🎯 Method 1: NPX Installation (🌟 Recommended)
> 💡 **Easiest way**: No local installation needed!
#### **Step 1**: Create Configuration File 📄
Create a `my-websites.json` file:
```json
{
"websites": [
{
"name": "your_website",
"url": "https://your-website.com",
"description": "Your Project Website"
},
{
"name": "api_docs",
"url": "https://api.example.com/openapi.json",
"description": "Your API Specification"
}
]
}
```
#### **Step 2**: Configure MCP Server ⚙️
Add to `.cursor/mcp.json`:
```json
{
"mcpServers": {
"website-to-markdown": {
"command": "npx",
"args": ["-y", "website-to-markdown-mcp"],
"disabled": false,
"env": {
"WEBSITES_CONFIG_PATH": "./my-websites.json"
}
}
}
}
```
#### **Step 3**: Restart and Test 🔄
1. **Restart Cursor**
2. **Open Chat and use Agent mode**
3. **Test command**: `Please list all configured websites`
<div align="center">
**🎉 Done! No installation required!**
</div>
---
### 🎯 Method 2: Local Installation
> 💡 **Best Practice**: Use this method for development or customization!
#### **Step 1**: Clone and Build
```bash
git clone https://github.com/your-username/website-to-markdown-mcp.git
cd website-to-markdown-mcp
npm install
npm run build
```
#### **Step 2**: Configure MCP Server
Add to `.cursor/mcp.json`:
```json
{
"mcpServers": {
"website-to-markdown": {
"command": "cmd",
"args": ["/c", "node", "./website-to-markdown-mcp/dist/index.js"],
"disabled": false,
"env": {
"WEBSITES_CONFIG_PATH": "./my-websites.json"
}
}
}
}
```
---
## 🔥 Enhanced Output Features
### 📊 Rich Content Analysis
Every fetched content now includes:
- **📝 Content Summary**: AI-generated summary of the main content
- **⏱️ Reading Time**: Estimated reading time based on content length
- **🔢 Word Count**: Accurate word count for both English and Chinese
- **🌍 Language Detection**: Automatic language identification
- **🎯 Content Quality Score**: Assessment of content relevance
### 📋 Enhanced Markdown Output
```markdown
# 🚀 Example Website
**Source**: https://example.com
**Website**: example_site - Example Website
**📊 Reading Time**: 5 minutes
**🔢 Word Count**: 1,250 words
**🌍 Language**: English
**📝 Summary**: This article discusses the latest developments in web technology...
---
[Enhanced Markdown content with better formatting...]
```
---
## 🆕 Complete OpenAPI/Swagger Support
<div align="center">
### 🔥 Professional API Documentation
</div>
| Feature | OpenAPI 3.x | Swagger 2.0 | Description |
|:-----|:----------:|:-----------:|:-----|
| 🔍 **Auto Detection** | ✅ | ✅ | Support JSON/YAML formats |
| ✅ **Professional Validation** | ✅ | ✅ | Using `@readme/openapi-parser` |
| 📋 **Structured Parsing** | ✅ | ✅ | Endpoints, parameters, responses |
| 🔗 **Reference Resolution** | ✅ | ✅ | Auto handle `$ref` references |
| 📊 **Smart Summary** | ✅ | ✅ | Generate API overview |
| 📝 **Formatted Output** | ✅ | ✅ | Readable Markdown |
### 🌟 Pre-configured Example Websites
```json
{
"websites": [
{
"name": "petstore_openapi",
"url": "https://petstore3.swagger.io/api/v3/openapi.json",
"description": "🐕 Swagger Petstore OpenAPI 3.0 Spec (Demo)"
},
{
"name": "petstore_swagger",
"url": "https://petstore.swagger.io/v2/swagger.json",
"description": "🐱 Swagger Petstore Swagger 2.0 Spec (Demo)"
},
{
"name": "github_api",
"url": "https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json",
"description": "🐙 GitHub REST API OpenAPI Spec"
}
]
}
```
---
## 📦 Installation & Setup
### 🛠️ System Requirements
- **Node.js** 20.18.1+ (Recommended: v22.15.0 LTS)
- **npm** 10.0.0+ or **yarn**
- **Cursor** Editor
> ⚠️ **Important**: Some dependencies require Node.js v20.18.1 or higher. Please update your Node.js version if you encounter engine compatibility warnings.
### ⚡ NPM Package Installation
```bash
# Global installation
npm install -g website-to-markdown-mcp
# Or use directly with npx (recommended)
npx website-to-markdown-mcp
```
### 🔧 Development Setup
```bash
# 1. Clone repository
git clone https://github.com/your-username/website-to-markdown-mcp.git
cd website-to-markdown-mcp
# 2. Install dependencies
npm install
# 3. Build project
npm run build
```
### 🎛️ Advanced Configuration Options
<div align="center">
#### Configuration Priority Order
</div>
```mermaid
graph TD
A[🔍 Check Environment Variable<br/>WEBSITES_CONFIG_PATH] --> B{File exists?}
B -->|Yes| C[✅ Load External Config File]
B -->|No| D[🔍 Check Environment Variable<br/>WEBSITES_CONFIG]
D --> E{Valid JSON?}
E -->|Yes| F[✅ Load Embedded Config]
E -->|No| G[🔍 Check config.json]
G --> H{File exists?}
H -->|Yes| I[✅ Load Local Config]
H -->|No| J[🔧 Use Default Config]
```
---
## 🎨 Configuration Method Details
### 📋 Method 1: External Configuration File (🌟 Recommended)
> 💡 **Advantages**: Easy to edit, syntax highlighting, version control friendly
<details>
<summary><b>🔧 Detailed Setup Steps</b></summary>
1. **Create Configuration File**
```bash
# Can be placed anywhere
touch my-api-configs.json
```
2. **Edit Configuration Content**
```json
{
"websites": [
{
"name": "my_docs",
"url": "https://docs.example.com",
"description": "📚 My Documentation Website"
}
]
}
```
3. **Set Environment Variable**
```json
{
"env": {
"WEBSITES_CONFIG_PATH": "./my-api-configs.json"
}
}
```
</details>
### 📋 Method 2: Embedded JSON (Backward Compatible)
<details>
<summary><b>🔧 Configuration Example</b></summary>
```json
{
"mcpServers": {
"website-to-markdown": {
"command": "cmd",
"args": ["/c", "node", "./website-to-markdown-mcp/dist/index.js"],
"disabled": false,
"env": {
"WEBSITES_CONFIG": "{\"websites\":[{\"name\":\"example\",\"url\":\"https://example.com\",\"description\":\"Example Website\"}]}"
}
}
}
}
```
</details>
### 📋 Method 3: Local config.json
<details>
<summary><b>🔧 Local Configuration</b></summary>
Directly edit `config.json` in the project root directory:
```json
{
"websites": [
{
"name": "local_site",
"url": "https://local.example.com",
"description": "🏠 Local Test Website"
}
]
}
```
</details>
---
## 🔧 Available Tools
### 🌐 General Tools
| Tool Name | Function | Parameters | Example |
|:--------|:-----|:-----|:-----|
| `fetch_website` | Fetch any website | `url`: Website URL | Fetch OpenAPI spec files |
| `list_configured_websites` | List configured websites | None | View all available websites |
### 🎯 Dedicated Tools
Each configured website automatically generates corresponding dedicated tools:
- `fetch_petstore_openapi` - Fetch Petstore OpenAPI 3.0 spec
- `fetch_petstore_swagger` - Fetch Petstore Swagger 2.0 spec
- `fetch_github_api` - Fetch GitHub API spec
- `fetch_tailwind_css` - Fetch Tailwind CSS documentation
---
## 📊 Enhanced Output Format Examples
### 🌐 General Website Content with Analytics
```markdown
# Website Title
**Source**: https://example.com
**Website**: example_site - Example Website
**📊 Reading Time**: 3 minutes
**🔢 Word Count**: 650 words
**🌍 Language**: English
**📝 Summary**: This article provides a comprehensive overview of modern web development practices, covering frontend frameworks, backend technologies, and deployment strategies.
---
[Enhanced cleaned Markdown content with ads removed and main content extracted...]
```
### 📋 OpenAPI 3.x Specification File
```markdown
# 🚀 Example API (v2.1.0)
**Source**: https://api.example.com/openapi.json
**OpenAPI Version**: 3.0.3
**Validation Status**: ✅ Valid
**📊 Processing Time**: 1.2 seconds
**🔢 Endpoints**: 25 endpoints
**🌍 Server Locations**: 3 servers
---
## 📋 API Basic Information
- **API Name**: Example API
- **Version**: 2.1.0
- **OpenAPI Version**: 3.0.3
- **Description**: A powerful example API for modern applications
## 🌐 Servers
1. **https://api.example.com**
- 🏢 Production server
2. **https://staging-api.example.com**
- 🧪 Testing server
## 🛠️ API Endpoints
Total of **25** endpoints:
### 👥 `/users`
- **GET**: Get user list
- **POST**: Create new user
### 🔍 `/users/{id}`
- **GET**: Get specific user
- **PUT**: Update user information
- **DELETE**: Delete user
## 🧩 Components
- **Schemas**: 12 data models
- **Parameters**: 8 reusable parameters
- **Responses**: 15 reusable responses
- **Security Schemes**: 3 security mechanisms
```
---
## 🎯 Usage Examples
### 💻 Basic Usage
```
Please fetch the content from https://docs.example.com and convert to markdown
```
### 🔍 OpenAPI Specification Fetching
```
Please use the fetch_petstore_openapi tool to fetch Petstore OpenAPI specification
```
### 📚 Documentation Website Fetching
```
Please fetch React official documentation content
```
---
## 🚨 Troubleshooting
> 📋 **Complete Troubleshooting Guide**: See [TROUBLESHOOTING.md](TROUBLESHOOTING.md) for detailed solutions to common issues.
### ❓ Quick Solutions
<details>
<summary><b>🔧 Node.js Version Issues</b></summary>
**Error**: `npm WARN EBADENGINE Unsupported engine`
- **Solution**: Update Node.js to v20.18.1 or higher
- **Download**: [Node.js Official Website](https://nodejs.org/)
- **Verify**: `node --version`
</details>
<details>
<summary><b>🌐 Module Not Found Issues</b></summary>
**Error**: `Cannot find module './db.json'`
- **Solution 1**: Clear npm cache: `npm cache clean --force`
- **Solution 2**: Update Node.js version
- **Solution 3**: Use local installation instead of npx
</details>
<details>
<summary><b>⚙️ Configuration Issues</b></summary>
**Q: Configuration changes not taking effect?**
- ✅ Confirm JSON format is correct
- ✅ Restart Cursor
- ✅ Check environment variable names
**Q: JSON format errors?**
- 🛠️ Use [JSON Validator](https://jsonlint.com/)
- 🛠️ Confirm using double quotes
- 🛠️ Check for extra commas
</details>
### 🔍 Debug Mode
Detailed logs are output to stderr at startup:
```bash
# View debug messages
npm run dev 2> debug.log
```
---
## 📈 Performance & Optimization
### ⚡ Performance Features
- 🚀 **Smart Retry**: Intelligent retry with exponential backoff
- 💾 **Rate Limiting**: Built-in rate limiting to prevent overload
- 🎯 **Content Filtering**: Remove irrelevant content for faster processing
- 🧹 **Ad Removal**: Automatic ad and popup removal
- 📊 **Stealth Mode**: Anti-detection browsing capabilities
### 🛡️ Security Considerations
- 🔒 HTTPS websites only (recommended)
- 🛠️ Auto filter malicious scripts
- 📝 Limit output content length
- 🔐 Stealth browsing to avoid detection
---
## 📦 Dependencies
<div align="center">
| Package | Version | Purpose |
|:-----|:----:|:-----|
| `@modelcontextprotocol/sdk` | ^1.0.0 | MCP Core Framework |
| `@readme/openapi-parser` | ^4.1.0 | Professional OpenAPI Parsing |
| `axios` | ^1.6.0 | HTTP Request Handling |
| `cheerio` | ^1.0.0 | HTML Parsing Engine |
| `turndown` | ^7.1.2 | HTML to Markdown |
| `yaml` | ^2.8.0 | YAML Format Support |
| `zod` | ^3.22.0 | Data Validation Framework |
| `playwright` | ^1.40.0 | Browser automation |
</div>
---
## 📝 Changelog
### 🎉 v1.2.0 (Latest)
<div align="center">
**🚀 Major Feature Updates**
</div>
- ✨ **Added** Enhanced content processing with AI-powered cleanup
- ✨ **Added** Smart analytics: word count, reading time, content summary
- ✨ **Added** Language detection and multi-language support
- ✨ **Added** Stealth browser capabilities for anti-detection
- ✨ **Added** Built-in rate limiting and retry mechanisms
- ✨ **Added** Advanced content filtering and ad removal
- 🔧 **Enhanced** Markdown processing with more HTML element support
- 📊 **Improved** Output format with rich metadata
- 🎯 **Fixed** Various technical issues and dependencies
### 🎯 v1.1.0 (Previous)
<div align="center">
**🚀 Major Feature Updates**
</div>
- ✨ **Added** Full OpenAPI 3.x/Swagger 2.0 support
- ✨ **Added** JSON/YAML format auto-detection
- ✨ **Added** Professional-grade spec validation and reference resolution
- ✨ **Added** Version auto-adaptation mechanism
- ✨ **Added** Structured API documentation summary
- 🔧 **Pre-configured** Multiple OpenAPI/Swagger examples
- 📦 **Added** NPM package distribution with npx support
- 🎯 **Enhanced** Installation methods for better user experience
### 🎯 v1.0.0 (Stable)
- 🎉 **Initial Release**
- 🌐 **Basic Functions** Website content fetching
- 📝 **Core Functions** Markdown conversion
- ⚙️ **Configuration Support** Multi-website management
---
## 🤝 Contributing
### 💡 How to Contribute
1. **🍴 Fork** this project
2. **🌟 Create** feature branch (`git checkout -b feature/AmazingFeature`)
3. **📝 Commit** changes (`git commit -m 'Add some AmazingFeature'`)
4. **📤 Push** to branch (`git push origin feature/AmazingFeature`)
5. **🔄 Open** Pull Request
### 🐛 Issue Reporting
Report issues on the [Issues](https://github.com/your-repo/issues) page, please include:
- 🔍 **Issue Description**
- 🔄 **Reproduction Steps**
- 💻 **Environment Information**
- 📸 **Screenshots or Logs**
---
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
---
<div align="center">
### 🌟 If this project helps you, please give it a Star!
**💬 Have questions or suggestions? Feel free to open an Issue!**
---
**Made by Sun** ❤️ **for the Developer Community**
</div>
