# Scrape CLI

Standalone web scraper that converts pages to markdown. No Docker, Redis, or API server needed.

# Install

```bash
npm install
npm link    # makes `scrape` available globally
```

# Quick Start

```bash
scrape https://example.com                   # markdown to stdout
scrape https://example.com -o out.md         # save to file
scrape url1 url2 url3 -o ./pages/            # multiple URLs to directory
cat urls.txt | scrape -o ./output/           # pipe URLs from stdin
scrape map https://example.com               # discover all URLs on domain
scrape crawl https://example.com --depth 2   # crawl and scrape pages (BFS)
```

# Features

- **Multiple output formats** — markdown (default), html, json (`-f html`, `-f json`)
- **Smart content extraction** — expanded selector list with scoring heuristic fallback
- **Map** — sitemap + HTML link discovery, filtering, dedup
- **Crawl** — BFS traversal, depth control, robots.txt respect, delay, include/exclude filters
- **Metadata** — title, description, author via `--metadata`
- **curl fallback** — handles bot-blocking sites (e.g. StackOverflow)
- **Full page mode** — `--full` to skip content extraction

# File Structure

```
src/scrape.mjs         CLI entry point, subcommand routing
lib/
  constants.mjs        Headers, selectors, blocked extensions
  html.mjs             HTML cleanup, markdown conversion, content scoring
  scraper.mjs          HTTP fetch + curl fallback, scrape orchestrator
  markdown-twin.mjs    Fetches the site's own .md version of a page
  twitter.mjs          Twitter/X handler via FxTwitter API + oEmbed
  links.mjs            Link extraction, URL filtering, normalization
  sitemap.mjs          Sitemap/robots.txt parsing
  output.mjs           File/directory output helpers
worker/                Cloudflare Workers MCP server
docs/                  Documentation
```

# Tech Stack

Node.js ESM, [cheerio](https://github.com/cheeriojs/cheerio), [turndown](https://github.com/mixmark-io/turndown) + GFM plugin, native `fetch` with curl fallback.

See [docs/](docs/) for detailed usage, architecture, and contributing guides.