# opencrawl

Local, keyless web scraping for agents and scripts. One Node process gives you an MCP
server, a v2-style HTTP API, a CLI and a TypeScript library that turn web pages into clean
markdown. No account, no credits, no data leaving your machine except the requests you
make to the sites you scrape.

- **MCP server** with `scrape`, `batch_scrape`, `map`, `crawl`, `check_crawl_status`, `search`,
  `parse`, `extract`, `feed`, `search_local` and `watch` tools for Claude Code and any other
  MCP client.
- **Feeds**: parse RSS 2.0, RSS 1.0, Atom and JSON Feed, or discover a site's feed from a
  page; the cheapest way to ask "what's new".
- **Documents**: PDF, DOCX and XLSX, from disk or from a URL.
- **Polite and resilient**: retries 429/503 with backoff and `Retry-After`, honours
  robots.txt `Crawl-delay`, and can write every page to disk as markdown.
- **Structured extraction without a model**: JSON-LD, microdata, Open Graph, Twitter card
  and meta tags, values picked by CSS selectors (with attributes, lists and nested fields),
  and HTML tables as JSON rows or CSV. Exact values, no hallucination.
- **Local store** (SQLite, no server): every scrape and crawl is saved. `maxAge` serves
  repeat requests from it, `search_local` full-text searches everything you have ever
  scraped (offline), crawl jobs survive restarts, and `watch` tells you what changed on a
  page since you last looked, with a unified diff.
- **HTTP API** (`/v2/scrape`, `/v2/map`, `/v2/crawl`, `/v2/search`, `/v2/parse`) with
  `{ success, data }` envelopes, so clients that let you override their API base URL can
  point at it.
- **CLI** and **library** for direct use.
- Plain HTTP first; headless Chromium only when a page is a JavaScript shell or when you ask
  for a screenshot or `waitFor`. Uses your installed Google Chrome when Playwright's bundled
  Chromium is missing.

## Install

```bash
git clone https://github.com/ktcod/opencrawl
cd opencrawl
npm install         # Node 22.13 or newer (built-in SQLite)
npm run build
npm link            # puts `opencrawl` on your PATH
opencrawl doctor    # checks browser + search backend
```

If `doctor` reports no browser, either install Google Chrome or run
`npx playwright install chromium`.

## Use it from Claude Code (MCP)

```bash
claude mcp add --scope user opencrawl -- opencrawl mcp
```

Or edit `~/.claude.json`:

```json
{
  "mcpServers": {
    "opencrawl": { "command": "opencrawl", "args": ["mcp"] }
  }
}
```

Tools are registered as `scrape`, `batch_scrape`, `map`, `crawl`, `check_crawl_status`,
`search`, `parse`, `extract`, `feed`, `search_local` and `watch`. `crawl` and `batch_scrape`
accept `outDir` to write pages to disk instead of returning them. Set `OPENCRAWL_TOOL_PREFIX` to prefix them if an existing
prompt expects other names.

## Use the HTTP API

```bash
opencrawl serve                # http://127.0.0.1:3002
curl -s http://127.0.0.1:3002/v2/scrape \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown","links"]}'
```

| Endpoint | Body | Response |
| --- | --- | --- |
| `POST /v2/scrape` | `{ url, formats?, onlyMainContent?, includeTags?, excludeTags?, waitFor?, timeout?, mobile?, headers?, render? }` | `{ success, data: Document }` |
| `POST /v2/batch/scrape` | `{ urls[], concurrency?, ignoreInvalidURLs?, scrapeOptions? }` | `{ success, data: Document[], errors, total, completed }` (input order) |
| `POST /v2/feed` | `{ url, limit?, since?, discover?, scrape?, scrapeOptions? }` | `{ success, data: { type, title, link, items: [{ title, url, published, author, summary, content, categories }], discovered? } }` |
| `POST /v2/map` | `{ url, search?, sitemap?, includeSubdomains?, ignoreQueryParameters?, limit? }` | `{ success, links: [{ url, title? }] }` |
| `POST /v2/crawl` | `{ url, limit?, maxDiscoveryDepth?, includePaths?, excludePaths?, sitemap?, crawlEntireDomain?, allowSubdomains?, allowExternalLinks?, delay?, maxConcurrency?, scrapeOptions? }` | `{ success, id, url }` |
| `GET /v2/crawl/:id?skip=0&limit=200` | | `{ success, status, total, completed, data: Document[], next }`; `next` is the URL of the following page or `null` |
| `GET /v2/crawl/:id/errors` | | `{ success, errors, robotsBlocked }` |
| `DELETE /v2/crawl/:id` | | `{ success, status: "cancelled" }` |
| `POST /v2/search` | `{ query, limit?, sources?, tbs?, includeDomains?, excludeDomains?, scrapeOptions? }` | `{ success, data: { web, news, images } }` |
| `POST /v2/parse` | multipart `file` (+ `options` JSON), or `{ filePath }` from localhost; PDF, DOCX, XLSX, HTML, text | `{ success, data: Document }` |
| `POST /v2/extract` | `{ url, structured?, selectors?, tables?, scrapeOptions? }` | `{ success, data: { jsonLd, microdata, openGraph, twitter, meta, selected, tables, errors } }` |
| `POST /v2/watch` | `{ url, diff?, scrapeOptions? }` | `{ success, status: new \| same \| changed \| removed \| error, previous, current, diff?, linesAdded?, linesRemoved? }` |
| `GET /v2/watches` | | `{ success, watches: [{ url, checkCount, lastCheckedAt, lastChangedAt }] }` |
| `POST /v2/local-search` | `{ query, limit?, site?, since? }` | `{ success, data: [{ url, title, snippet, fetchedAt, score }] }` |

A `Document` is `{ markdown?, html?, rawHtml?, links?, images?, screenshot?, metadata }`
where `metadata` carries `title`, `description`, `language`, `sourceURL`, `url`,
`statusCode`, `contentType` and the usual Open Graph fields.

The API refuses to bind a non-loopback address without `OPENCRAWL_API_KEY`; with a key,
every `/v2` request needs `Authorization: Bearer <token>`. Browser requests (any `Origin`
header), unexpected `Host` headers and non-JSON bodies are refused, so a web page cannot
drive the scraper. `/v2/parse` only accepts a raw `filePath` from loopback callers; remote
callers upload the file as multipart form-data.

## Use the CLI directly

```bash
opencrawl scrape https://example.com                     # markdown to stdout
opencrawl scrape https://example.com -f markdown,links --json
opencrawl scrape https://app.example.com --render always  # force a browser render
opencrawl map https://docs.example.com --search "auth" --limit 20
opencrawl crawl https://docs.example.com --limit 50 --include-paths "^/docs" -o crawl.json
opencrawl search "query" --limit 5 --scrape
opencrawl parse ./report.pdf                               # also .docx and .xlsx
opencrawl scrape https://a.example https://b.example --out-dir ./pages   # batch, one folder per page
opencrawl crawl https://docs.example.com --limit 50 --out-dir ./docs      # <dir>/<host>/<path>/index.md
opencrawl feed https://example.com/blog --limit 5           # discovers and parses the feed
opencrawl feed https://example.com/feed.xml --scrape --json # items plus each page's markdown
opencrawl scrape https://example.com --max-age 3600000   # reuse a snapshot under an hour old
opencrawl search-local "pricing tiers" --site docs.example.com
opencrawl watch https://example.com/changelog              # new | same | changed (+diff) | removed
opencrawl watch --list
opencrawl extract https://example.com/product             # JSON-LD, microdata, OG, meta, tables
opencrawl extract https://example.com/list --select "names=.product .name*" --select "links=.product a@href*"
opencrawl extract https://example.com/stats --csv         # every table as CSV
```

## Feeds, documents and files on disk

- **Feeds.** `feed` accepts a feed URL or any page. For a page it reads
  `<link rel="alternate">` tags, then probes `/feed`, `/rss.xml`, `/atom.xml`, `/index.xml`,
  `/feed.json` and friends. Items are normalized across RSS 2.0, RSS 1.0 (RDF), Atom and
  JSON Feed: `title`, `url`, `published`, `updated`, `author`, `summary` (text), `content`
  (markdown), `categories`. `since` filters by date, `scrape: true` attaches each item's page.
- **DOCX and XLSX.** `parse` and `scrape` handle them next to PDF. DOCX goes through the same
  HTML-to-markdown pipeline as web pages (headings, bold, lists survive). Each XLSX sheet
  becomes a markdown table whose first row is the header; ask for the `extract` format to get
  the sheets as table objects too.
- **Out-dir.** `crawl --out-dir`, `scrape <urls...> --out-dir`, and `outDir` on the MCP
  `crawl` and `batch_scrape` tools write one folder per page at
  `<dir>/<host>/<path>/` with `index.md`, plus `index.html`, `links.txt`, `extract.json`,
  `screenshot.png` and `metadata.json` when those formats were requested. Query strings
  become a short hash so variants do not collide; nothing is written outside the directory.
- **Retries and politeness.** Plain HTTP fetches retry 429 and 503 up to
  `OPENCRAWL_RETRIES` (2) times, honouring `Retry-After` and capping each wait at
  `OPENCRAWL_MAX_RETRY_WAIT_MS` (30 s). A crawl with no explicit `delay` uses the start
  host's robots.txt `Crawl-delay`, which also serializes its requests.

## Structured extraction

`extract` reads what the page already states in machine-readable form, so an agent gets
exact values before it reads any prose:

- **Embedded data** (`structured`, default all): `jsonLd` (every `application/ld+json`
  block, invalid ones reported under `errors`), `microdata` (items as JSON-LD-style
  objects with `@type`, nested items and repeated properties as arrays), `openGraph`
  (`og:*`, `article:*`, `product:*` and friends, repeated keys as arrays), `twitter`
  and `meta` (title, lang, canonical, description, author, and so on).
- **Selectors** (`selectors`): `{ "price": ".price" }` returns the first match's text.
  A full spec is `{ selector, attribute?, type?: "text" | "html" | "attribute",
  multiple?, fields? }`. `multiple` returns every match; `fields` evaluates sub-selectors
  inside each match so a list of cards becomes a list of objects; `href`/`src` style
  attributes come back absolute. Invalid selectors land in `errors`, not exceptions.
- **Tables** (`tables`, default on): each `<table>` becomes `{ caption, headers, rows,
  rowCount, columnCount }`. Header rows (`<thead>` or all-`<th>` rows) become keys,
  duplicate headers get `_2` suffixes, `colspan`/`rowspan` are expanded, and
  `{ csv: true }` adds RFC 4180 CSV. `selector` limits which tables, `minRows` and
  `maxRows` bound them.

The same options work as a scrape format, `{ type: "extract", ... }`, so a crawl can
extract from every page it visits (the result lands in `document.extract`).

## The local store

Every scrape, crawl page and watch snapshot is written to a SQLite file
(`~/.opencrawl/opencrawl.db` by default, `OPENCRAWL_DB` to move it). Screenshots are not
stored. What it gives you:

- **Cache.** `maxAge` (milliseconds) on a scrape or crawl returns the newest snapshot
  fetched within that window instead of hitting the network; the result carries
  `metadata.cacheState: "hit"` and `cachedAt`. The default window is 0 (always fresh); set
  `OPENCRAWL_DEFAULT_MAX_AGE_MS` to change it. `storeInCache: false` skips the write.
- **Local search.** `search_local` / `opencrawl search-local` runs SQLite FTS5 over titles
  and markdown of everything stored, one hit per URL, with a snippet. No network.
- **Persistent crawls.** Job status and pages are read from the store, so a crawl started by
  one process can be read by another and survives restarts (a crawl interrupted mid-run is
  marked `failed`). Finished jobs are kept for `OPENCRAWL_JOB_RETENTION_DAYS` (7).
- **Change tracking.** `watch` re-scrapes a URL and compares it with its last snapshot.
  The first call records a baseline (`new`); later calls answer `same`, `changed` (with a
  unified diff and added/removed line counts), `removed` (404/410 after a baseline) or
  `error`. Nothing is scheduled inside opencrawl; call it from cron, launchd or an agent.

Delete the file to start over.

## Use the library

```ts
import { scrape, map, crawl, search, parseFile } from "opencrawl";

const doc = await scrape("https://example.com", { formats: ["markdown", "links"] });
const { links } = await map("https://example.com", { search: "pricing" });
const job = await crawl("https://docs.example.com", { limit: 20, includePaths: ["^/docs"] });
const results = await search({ query: "self hosted scraper", limit: 5 });
```

## What it does and doesn't do

| Feature | opencrawl |
| --- | --- |
| Scrape formats markdown, html, rawHtml, links, images, screenshot, attributes | Yes |
| LLM-backed formats (json, summary, query, changeTracking, branding) | No. Returned as a `warning`, markdown still delivered |
| `onlyMainContent`, `includeTags`, `excludeTags`, `waitFor`, `mobile`, `headers`, `removeBase64Images`, `skipTlsVerification` | Yes |
| PDF scraping and parsing (`pdfOptions.maxPages`) | Yes (text layer only, no OCR) |
| Map with sitemap modes, keyword ranking, subdomains, query stripping, limit | Yes |
| Crawl with limit, depth, include/exclude paths, subdomains, external links, entire-domain, delay, concurrency, sitemap modes, robots.txt | Yes. Default limit 100 pages |
| Search sources web/news/images, time filters, domain filters, per-result scraping, github/pdf/research categories | Yes. News and images need SearXNG or Brave |
| Parse PDF, DOCX, XLSX, HTML, text files | Yes |
| Batch scrape, feeds (RSS/Atom/JSON), pages written to disk | Yes |
| Retries on 429/503 with Retry-After, robots.txt Crawl-delay | Yes |
| Cache (`maxAge`), offline full-text search, persistent crawl jobs, change tracking | Yes, via the local SQLite store |
| Structured extraction: JSON-LD, microdata, Open Graph, meta, CSS selectors, tables to JSON/CSV | Yes, no model involved |
| Browser actions, interactive sessions, agents, scheduled monitors, webhooks | No |
| Proxies, stealth, caching options | Accepted and ignored |

## Search backends

| Backend | Configure | Sources |
| --- | --- | --- |
| DuckDuckGo HTML (default) | nothing | web only; rate limited under heavy use |
| SearXNG | `SEARXNG_URL=http://localhost:8080` | web, news, images |
| Brave Search API | `BRAVE_SEARCH_API_KEY=...` | web, news, images |

Force one with `OPENCRAWL_SEARCH_BACKEND=duckduckgo|searxng|brave`.

## Configuration

| Variable | Default | Meaning |
| --- | --- | --- |
| `OPENCRAWL_BROWSER` | `auto` | `auto` tries bundled Chromium, then Chrome, then Edge. `chromium`, `chrome`, `msedge`, or `none` |
| `OPENCRAWL_TIMEOUT_MS` | `30000` | Per-request timeout |
| `OPENCRAWL_CRAWL_LIMIT` | `100` | Default crawl page cap |
| `OPENCRAWL_CRAWL_CONCURRENCY` | `5` | Parallel pages per crawl (max 20) |
| `OPENCRAWL_MAP_LIMIT` | `5000` | Default map URL cap |
| `OPENCRAWL_MAP_DEPTH` / `OPENCRAWL_MAP_MAX_PAGES` | `1` / `20` | How far map follows links when the sitemap is thin |
| `OPENCRAWL_USER_AGENT` | Chrome desktop UA | User agent for HTTP and browser requests |
| `OPENCRAWL_ROBOTS_UA` | `opencrawl` | Token checked against robots.txt during crawls |
| `OPENCRAWL_TOOL_PREFIX` | empty | MCP tool name prefix |
| `OPENCRAWL_DB` | `~/.opencrawl/opencrawl.db` | SQLite store path (`:memory:` for a throwaway store); `OPENCRAWL_DATA_DIR` moves the folder |
| `OPENCRAWL_DEFAULT_MAX_AGE_MS` | `0` | Cache window used when a request sets no `maxAge` |
| `OPENCRAWL_CRAWL_PAGE_SIZE` | `200` | Documents per page on `GET /v2/crawl/:id` |
| `OPENCRAWL_JOB_RETENTION_DAYS` | `7` | How long finished crawl jobs stay in the store |
| `OPENCRAWL_RETRIES` | `2` | Retries on HTTP 429/503 per fetch |
| `OPENCRAWL_MAX_RETRY_WAIT_MS` | `30000` | Longest wait between retries, even if Retry-After asks for more |
| `OPENCRAWL_BATCH_CONCURRENCY` | `5` | Parallel pages for batch scrapes |
| `OPENCRAWL_API_KEY` | unset | Bearer token required by the HTTP API. Required to bind a non-loopback `HOST` (or set `OPENCRAWL_INSECURE=1`) |
| `OPENCRAWL_ALLOWED_HOSTS` | bind host | Extra `Host` header values the HTTP API accepts (DNS-rebinding guard); `*` disables the check |
| `OPENCRAWL_ALLOWED_ORIGINS` | none | Browser origins allowed to call the HTTP API; any other `Origin` header is refused |
| `OPENCRAWL_MAX_RESPONSE_BYTES` | `52428800` | Largest page or sitemap body buffered (50 MB) |
| `OPENCRAWL_MAX_SITEMAP_FETCHES` | `50` | Sitemap files fetched per map or crawl |
| `OPENCRAWL_MAX_BROWSER_CONTEXTS` | `4` | Browser pages open at once across all scrapes |
| `PORT` / `HOST` | `3002` / `127.0.0.1` | HTTP API bind |

## Development

```bash
npm test             # vitest against a local fixture site; no network, no browser
npm run typecheck
OPENCRAWL_TEST_BROWSER=1 npm test   # also runs the Chrome rendering tests
```

Design decisions and known gaps are in [DESIGN.md](DESIGN.md).

## License

MIT