Skip to main content
Glama
1abhishekpandey

scrape-mcp

by 1abhishekpandey
OverviewSchemaSourceRelated ServersDiscussionsAdmin
JavaScript

Scrape CLI

Standalone web scraper that converts pages to markdown. No Docker, Redis, or API server needed.

Install

npm install
npm link    # makes `scrape` available globally

Quick Start

scrape https://example.com                   # markdown to stdout
scrape https://example.com -o out.md         # save to file
scrape url1 url2 url3 -o ./pages/            # multiple URLs to directory
cat urls.txt | scrape -o ./output/           # pipe URLs from stdin
scrape map https://example.com               # discover all URLs on domain
scrape crawl https://example.com --depth 2   # crawl and scrape pages (BFS)

Features

  • Multiple output formats — markdown (default), html, json (-f html, -f json)

  • Smart content extraction — expanded selector list with scoring heuristic fallback

  • Map — sitemap + HTML link discovery, filtering, dedup

  • Crawl — BFS traversal, depth control, robots.txt respect, delay, include/exclude filters

  • Metadata — title, description, author via --metadata

  • curl fallback — handles bot-blocking sites (e.g. StackOverflow)

  • Full page mode — --full to skip content extraction

File Structure

src/scrape.mjs         CLI entry point, subcommand routing
lib/
  constants.mjs        Headers, selectors, blocked extensions
  html.mjs             HTML cleanup, markdown conversion, content scoring
  scraper.mjs          HTTP fetch + curl fallback, scrape orchestrator
  markdown-twin.mjs    Fetches the site's own .md version of a page
  twitter.mjs          Twitter/X handler via FxTwitter API + oEmbed
  links.mjs            Link extraction, URL filtering, normalization
  sitemap.mjs          Sitemap/robots.txt parsing
  output.mjs           File/directory output helpers
worker/                Cloudflare Workers MCP server
docs/                  Documentation

Tech Stack

Node.js ESM, cheerio, turndown + GFM plugin, native fetch with curl fallback.

See docs/ for detailed usage, architecture, and contributing guides.

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityMaintained
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • Puppeteer Vision MCP Server

    djannot
    A
    license
    Not graded
    quality
    C
    maintenance
    Scrapes webpages and converts them to markdown using AI-powered interaction to automatically handle cookie banners, CAPTCHAs, paywalls, and other blocking elements before extracting clean content.
    8 npm
    48
    Apache 2.0

  • UnWeb MCP Server

    mbsoft-systems
    A
    license
    Not graded
    quality
    D
    maintenance
    Enables conversion of webpages to clean markdown with content quality scoring and multi-page crawling for documentation sites. Supports Claude Code, Cursor, and Windsurf with native LangChain and LlamaIndex export formats.
    26 npm
    MIT

  • Web Reader

    PPParticle
    A
    license
    Not graded
    quality
    B
    maintenance
    Converts web pages to Markdown for MCP clients like Claude, with support for single and batch reads, caching, fallback rendering, and SSRF protection.
    13 npm
    MIT

  • icecrawl MCP server

    MrGreenlaw
    F
    license
    Not graded
    quality
    C
    maintenance
    Enables scraping single pages or crawling entire websites, converting content to markdown and optionally extracting structured data with Claude.
    -