opencrawl

Local, keyless web scraping for agents and scripts. One Node process gives you an MCP server, a v2-style HTTP API, a CLI and a TypeScript library that turn web pages into clean markdown. No account, no credits, no data leaving your machine except the requests you make to the sites you scrape.

• MCP server with scrape, batch_scrape, map, crawl, check_crawl_status, search, parse, extract, feed, search_local and watch tools for Claude Code and any other MCP client.

• Feeds: parse RSS 2.0, RSS 1.0, Atom and JSON Feed, or discover a site's feed from a page; the cheapest way to ask "what's new".

• Documents: PDF, DOCX and XLSX, from disk or from a URL.

• Polite and resilient: retries 429/503 with backoff and Retry-After, honours robots.txt Crawl-delay, and can write every page to disk as markdown.

• Structured extraction without a model: JSON-LD, microdata, Open Graph, Twitter card and meta tags, values picked by CSS selectors (with attributes, lists and nested fields), and HTML tables as JSON rows or CSV. Exact values, no hallucination.

• Local store (SQLite, no server): every scrape and crawl is saved. maxAge serves repeat requests from it, search_local full-text searches everything you have ever scraped (offline), crawl jobs survive restarts, and watch tells you what changed on a page since you last looked, with a unified diff.

• HTTP API (/v2/scrape, /v2/map, /v2/crawl, /v2/search, /v2/parse) with { success, data } envelopes, so clients that let you override their API base URL can point at it.

• CLI and library for direct use.

• Plain HTTP first; headless Chromium only when a page is a JavaScript shell or when you ask for a screenshot or waitFor. Uses your installed Google Chrome when Playwright's bundled Chromium is missing.

Install

git clone https://github.com/ktcod/opencrawl
cd opencrawl
npm install         # Node 22.13 or newer (built-in SQLite)
npm run build
npm link            # puts `opencrawl` on your PATH
opencrawl doctor    # checks browser + search backend

If doctor reports no browser, either install Google Chrome or run npx playwright install chromium.

Related MCP server: scrapyard

Use it from Claude Code (MCP)

claude mcp add --scope user opencrawl -- opencrawl mcp

Or edit ~/.claude.json:

{
  "mcpServers": {
    "opencrawl": { "command": "opencrawl", "args": ["mcp"] }
  }
}

Tools are registered as scrape, batch_scrape, map, crawl, check_crawl_status, search, parse, extract, feed, search_local and watch. crawl and batch_scrape accept outDir to write pages to disk instead of returning them. Set OPENCRAWL_TOOL_PREFIX to prefix them if an existing prompt expects other names.

Use the HTTP API

opencrawl serve                # http://127.0.0.1:3002
curl -s http://127.0.0.1:3002/v2/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown","links"]}'

A Document is { markdown?, html?, rawHtml?, links?, images?, screenshot?, metadata } where metadata carries title, description, language, sourceURL, url, statusCode, contentType and the usual Open Graph fields.

The API refuses to bind a non-loopback address without OPENCRAWL_API_KEY; with a key, every /v2 request needs Authorization: Bearer <token>. Browser requests (any Origin header), unexpected Host headers and non-JSON bodies are refused, so a web page cannot drive the scraper. /v2/parse only accepts a raw filePath from loopback callers; remote callers upload the file as multipart form-data.

Use the CLI directly

opencrawl scrape https://example.com                     # markdown to stdout
opencrawl scrape https://example.com -f markdown,links --json
opencrawl scrape https://app.example.com --render always  # force a browser render
opencrawl map https://docs.example.com --search "auth" --limit 20
opencrawl crawl https://docs.example.com --limit 50 --include-paths "^/docs" -o crawl.json
opencrawl search "query" --limit 5 --scrape
opencrawl parse ./report.pdf                               # also .docx and .xlsx
opencrawl scrape https://a.example https://b.example --out-dir ./pages   # batch, one folder per page
opencrawl crawl https://docs.example.com --limit 50 --out-dir ./docs      # <dir>/<host>/<path>/index.md
opencrawl feed https://example.com/blog --limit 5           # discovers and parses the feed
opencrawl feed https://example.com/feed.xml --scrape --json # items plus each page's markdown
opencrawl scrape https://example.com --max-age 3600000   # reuse a snapshot under an hour old
opencrawl search-local "pricing tiers" --site docs.example.com
opencrawl watch https://example.com/changelog              # new | same | changed (+diff) | removed
opencrawl watch --list
opencrawl extract https://example.com/product             # JSON-LD, microdata, OG, meta, tables
opencrawl extract https://example.com/list --select "names=.product .name*" --select "links=.product a@href*"
opencrawl extract https://example.com/stats --csv         # every table as CSV

Feeds, documents and files on disk

• Feeds. feed accepts a feed URL or any page. For a page it reads <link rel="alternate"> tags, then probes /feed, /rss.xml, /atom.xml, /index.xml, /feed.json and friends. Items are normalized across RSS 2.0, RSS 1.0 (RDF), Atom and JSON Feed: title, url, published, updated, author, summary (text), content (markdown), categories. since filters by date, scrape: true attaches each item's page.

• DOCX and XLSX. parse and scrape handle them next to PDF. DOCX goes through the same HTML-to-markdown pipeline as web pages (headings, bold, lists survive). Each XLSX sheet becomes a markdown table whose first row is the header; ask for the extract format to get the sheets as table objects too.

• Out-dir. crawl --out-dir, scrape <urls...> --out-dir, and outDir on the MCP crawl and batch_scrape tools write one folder per page at <dir>/<host>/<path>/ with index.md, plus index.html, links.txt, extract.json, screenshot.png and metadata.json when those formats were requested. Query strings become a short hash so variants do not collide; nothing is written outside the directory.

• Retries and politeness. Plain HTTP fetches retry 429 and 503 up to OPENCRAWL_RETRIES (2) times, honouring Retry-After and capping each wait at OPENCRAWL_MAX_RETRY_WAIT_MS (30 s). A crawl with no explicit delay uses the start host's robots.txt Crawl-delay, which also serializes its requests.

Structured extraction

extract reads what the page already states in machine-readable form, so an agent gets exact values before it reads any prose:

• Embedded data (structured, default all): jsonLd (every application/ld+json block, invalid ones reported under errors), microdata (items as JSON-LD-style objects with @type, nested items and repeated properties as arrays), openGraph (og:*, article:*, product:* and friends, repeated keys as arrays), twitter and meta (title, lang, canonical, description, author, and so on).

• Selectors (selectors): { "price": ".price" } returns the first match's text. A full spec is { selector, attribute?, type?: "text" | "html" | "attribute", multiple?, fields? }. multiple returns every match; fields evaluates sub-selectors inside each match so a list of cards becomes a list of objects; href/src style attributes come back absolute. Invalid selectors land in errors, not exceptions.

• Tables (tables, default on): each <table> becomes { caption, headers, rows, rowCount, columnCount }. Header rows (<thead> or all-<th> rows) become keys, duplicate headers get _2 suffixes, colspan/rowspan are expanded, and { csv: true } adds RFC 4180 CSV. selector limits which tables, minRows and maxRows bound them.

The same options work as a scrape format, { type: "extract", ... }, so a crawl can extract from every page it visits (the result lands in document.extract).

The local store

Every scrape, crawl page and watch snapshot is written to a SQLite file (~/.opencrawl/opencrawl.db by default, OPENCRAWL_DB to move it). Screenshots are not stored. What it gives you:

• Cache. maxAge (milliseconds) on a scrape or crawl returns the newest snapshot fetched within that window instead of hitting the network; the result carries metadata.cacheState: "hit" and cachedAt. The default window is 0 (always fresh); set OPENCRAWL_DEFAULT_MAX_AGE_MS to change it. storeInCache: false skips the write.

• Local search. search_local / opencrawl search-local runs SQLite FTS5 over titles and markdown of everything stored, one hit per URL, with a snippet. No network.

• Persistent crawls. Job status and pages are read from the store, so a crawl started by one process can be read by another and survives restarts (a crawl interrupted mid-run is marked failed). Finished jobs are kept for OPENCRAWL_JOB_RETENTION_DAYS (7).

• Change tracking. watch re-scrapes a URL and compares it with its last snapshot. The first call records a baseline (new); later calls answer same, changed (with a unified diff and added/removed line counts), removed (404/410 after a baseline) or error. Nothing is scheduled inside opencrawl; call it from cron, launchd or an agent.

Delete the file to start over.

Use the library

import { scrape, map, crawl, search, parseFile } from "opencrawl";

const doc = await scrape("https://example.com", { formats: ["markdown", "links"] });
const { links } = await map("https://example.com", { search: "pricing" });
const job = await crawl("https://docs.example.com", { limit: 20, includePaths: ["^/docs"] });
const results = await search({ query: "self hosted scraper", limit: 5 });

What it does and doesn't do

Search backends

Force one with OPENCRAWL_SEARCH_BACKEND=duckduckgo|searxng|brave.

Configuration

Development

npm test             # vitest against a local fixture site; no network, no browser
npm run typecheck
OPENCRAWL_TEST_BROWSER=1 npm test   # also runs the Chrome rendering tests

Design decisions and known gaps are in DESIGN.md.

License

MIT