Glama
Sign In

Website Scraper MCP Server

by tolik-unicornrider
Verified
npmGitHub
JavaScript
11,993
2
RedditDiscord
OverviewInspectNewSchemaRelated ServersReviewsScore
Need Help?View Source CodeReport Issue

hybrid server

The server is able to function both locally and remotely, depending on the configuration or use case.

Integrations

  • Utilizes Mozilla's Readability library (the same engine used in Firefox's Reader View) to extract meaningful content from web pages for conversion to Markdown

  • Converts clean HTML to high-quality Markdown with TurndownService, supporting both web scraping and direct conversion of local HTML files

  • Leverages Mozilla's Readability library to extract the main content from web pages while removing clutter and navigation elements

Website Scraper

A command-line tool and MCP server for scraping websites and converting HTML to Markdown.

Features

  • Extracts meaningful content from web pages using Mozilla's Readability library (the same engine used in Firefox's Reader View)
  • Converts clean HTML to high-quality Markdown with TurndownService
  • Securely handles HTML by removing potentially harmful script tags
  • Works as both a command-line tool and an MCP server
  • Supports direct conversion of local HTML files to Markdown

Installation

# Install dependencies npm install # Build the project npm run build # Optionally, install globally npm install -g .

Usage

CLI Mode

# Print output to console scrape https://example.com # Save output to a file scrape https://example.com output.md # Convert a local HTML file to Markdown scrape --html-file input.html # Convert a local HTML file and save output to a file scrape --html-file input.html output.md # Show help scrape --help # Or run via npm script npm run start:cli -- https://example.com

MCP Server Mode

This tool can be used as a Model Context Protocol (MCP) server:

# Start in MCP server mode npm start

Code Structure

  • src/index.ts - Core functionality and MCP server implementation
  • src/cli.ts - Command-line interface implementation
  • src/data_processing.ts - HTML to Markdown conversion functionality

API

The tool exports the following functions:

// Scrape a website and convert to Markdown import { scrapeToMarkdown } from './build/index.js'; // Convert HTML string to Markdown directly import { htmlToMarkdown } from './build/data_processing.js'; async function example() { // Web scraping const markdown = await scrapeToMarkdown('https://example.com'); console.log(markdown); // Direct HTML conversion const html = '<h1>Hello World</h1><p>This is <strong>bold</strong> text.</p>'; const md = htmlToMarkdown(html); console.log(md); }

License

ISC

This server cannot be installed

-
security - not tested
F
license - not found
-
quality - not tested

An MCP server that extracts meaningful content from websites and converts HTML to high-quality Markdown, using Mozilla's Readability engine.

  1. Features
    1. Installation
      1. Usage
        1. CLI Mode
        2. MCP Server Mode
      2. Code Structure
        1. API
          1. License