# RAG Documentation Scraper

**Turn any documentation site into a vector-database-ready knowledge base in one click.**

Crawls documentation sites with Playwright, strips HTML navigation boilerplate, splits content on heading boundaries, and outputs clean Markdown or JSON chunks with embedded source metadata — ready to drop into Pinecone, Chroma, pgvector, or any LangChain / LlamaIndex / Haystack pipeline.

---

## Why AI engineers choose this over generic scrapers

| Generic scraper output | What you get here |
|------------------------|-------------------|
| Raw HTML with nav bars, sidebars, cookie banners | Clean text extracted from the main content area only — no noise |
| One blob per page | Heading-boundary chunks that map naturally to retrieval units |
| No metadata | Every chunk carries `source_url`, `page_title`, `section` breadcrumb, `token_count`, `content_hash` |
| Re-processes every page on every run | Optional DynamoDB delta tracking skips unchanged pages — up to **80% cost reduction** on re-runs |
| Duplicate content from versioned docs | SHA-256 deduplication silently drops identical chunks regardless of URL |

---

## Benchmarks

Tested against the Crawlee.dev documentation (production run, 2026-10-18):

- **25 pages crawled → 94 chunks** produced, **0 errors**
- All chunks under 512 tokens (compatible with text-embedding-3-small, nomic-embed-text, bge-m3)
- Per-page time: ~1.8 s average (Playwright, images/fonts blocked)
- Noise stripped: nav, sidebar, footer, cookie banners, pagination, breadcrumbs, "edit this page" widgets

---

## 3-line Python quickstart

```python
from apify_client import ApifyClient
chunks = ApifyClient("YOUR_APIFY_TOKEN").actor("YOUR_USERNAME/apify-rag-docs-scraper").call(
    run_input={"startUrls": [{"url": "https://YOUR_DOCS_SITE.com/docs/"}], "chunkSize": 500, "outputFormat": "json-chunks"}
)["defaultDatasetId"]
items = list(ApifyClient("YOUR_APIFY_TOKEN").dataset(chunks).iterate_items())
# items[0] → {"text": "...", "metadata": {"source": "...", "title": "...", "section": "...", "token_count": 487}}
```

See `developer_quickstart/colab_rag_example.ipynb` for a full end-to-end walkthrough: actor run → dataset pull → ChromaDB insert → semantic search query.

---

## Key features

- **3 RAG-native output formats** — `markdown` (default), `json-chunks` (Pinecone/Chroma/pgvector ready), `text` (bare text)
- **4 framework-specific formats** — LangChain `Document`, LlamaIndex `TextNode`, Haystack `Document`, raw JSONL
- **Flexible chunk size** — Set `chunkSize` (tokens) to control granularity; works with any embedding model
- **4 chunking strategies** — heading boundary (recommended), fixed token window with overlap, paragraph, whole-page
- **PDF support** — basic pdfjs extraction (free) or AWS Textract OCR for scanned/image PDFs
- **Incremental crawling** — DynamoDB delta tracking; re-runs skip unchanged pages entirely
- **Version-aware URL filtering** — restrict to a specific version path (e.g. `/v2/`) with one input field
- **MCP-compatible** — includes an MCP server wrapper so Claude Desktop and Cursor can call this actor as a live tool

---

## Output formats

### `markdown` (default) — generic RAG, vector-DB agnostic

```json
{
  "markdown": "## Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "source": "https://crawlee.dev/docs/introduction",
  "title": "Introduction | Crawlee for JavaScript",
  "section": "Introduction > Quick Start",
  "chunk_index": 0,
  "chunk_total": 12,
  "token_count": 312,
  "content_hash": "c4a7e891f23b5d10",
  "scraped_at": "2026-10-18T14:45:22.341Z"
}
```

### `json-chunks` — Pinecone / Chroma / pgvector

```json
{
  "text": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "title": "Introduction | Crawlee for JavaScript",
    "section": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "last_modified": null,
    "content_hash": "c4a7e891f23b5d10",
    "scraped_at": "2026-10-18T14:45:22.341Z"
  }
}
```

### `langchain` — drop-in LangChain `Document`

```json
{
  "page_content": "Quick Start\n\nCrawlee is available as a set of NPM packages...",
  "metadata": {
    "source": "https://crawlee.dev/docs/introduction",
    "heading_path": "Introduction > Quick Start",
    "chunk_index": 0,
    "chunk_total": 12,
    "token_count": 312,
    "content_hash": "c4a7e891f23b5d10"
  }
}
```

---

## Input parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `startUrls` | array | — | **Required.** Root URL(s) to crawl |
| `urlPatterns` | string[] | `[]` | Glob patterns to restrict crawl scope (e.g. `/docs/**`). Empty = all same-domain URLs |
| `maxDepth` | integer | `5` | Link levels to follow (0 = start pages only) |
| `outputFormat` | enum | `markdown` | `markdown`, `json-chunks`, `text`, `langchain`, `llamaindex`, `haystack`, `raw_jsonl` |
| `chunkSize` | integer | `512` | Target token count per chunk. 500 works well for most embedding models |
| `chunkStrategy` | enum | `heading_boundary` | `heading_boundary`, `fixed_tokens`, `paragraph`, `page` |
| `chunkOverlapTokens` | integer | `50` | Token overlap between adjacent chunks (for `fixed_tokens` strategy) |
| `deduplicateContent` | boolean | `true` | SHA-256 dedup — skip chunks already seen in this run |
| `versionFilter` | string | `""` | Only crawl URLs containing this path fragment (e.g. `/v3/`) |
| `useDeltaTracking` | boolean | `false` | DynamoDB incremental crawling — re-runs skip unchanged pages |
| `pdfHandling` | enum | `skip` | `skip`, `extract_text`, `aws_textract` |
| `maxRequestsPerCrawl` | integer | `200` | Hard page cap per run. `0` = unlimited |
| `excludeSelectors` | string[] | `[]` | Extra CSS selectors to strip (e.g. `.announcement-bar`) |

---

## Pricing

This actor uses **Pay-Per-Result** pricing — you pay per chunk pushed to the dataset.

| Docs site size | Typical chunks | Estimated cost |
|----------------|----------------|----------------|
| 50 pages | 250–400 chunks | ~$1.25–$2.00 |
| 100 pages | 500–800 chunks | ~$2.50–$4.00 |
| 500 pages | 2,500–4,000 chunks | ~$12.50–$20.00 |

**Cut costs by up to 80%:** Enable `useDeltaTracking: true` so re-runs only process pages that changed since the last run. Stable documentation sites rarely change more than 10–20% of pages per day.

---

## Pre-configured Published Tasks

Five ready-to-run tasks are available in the Apify Store for the most popular documentation niches:

1. **LangChain Documentation Scraper** — Full LangChain Python docs, Markdown + RAG chunks
2. **Stripe API Docs Ingestion Tool** — Stripe API reference in `json-chunks` format for payment bots
3. **Docusaurus / ReadTheDocs Crawler** — Universal adapter for Docusaurus and ReadTheDocs sites
4. **Cursor / Claude MCP Documentation Extractor** — MCP spec + Cursor AI docs into structured chunks
5. **GitHub Wiki & Markdown Repository Scraper** — GitHub Wikis and Markdown repos

Run `python scripts/create_published_tasks.py` (after `apify push`) to create these tasks programmatically.

---

## AWS Integration (optional)

AWS features — Textract OCR for PDFs and DynamoDB delta tracking — are fully optional. Set these environment variables in Apify Console → Actor Settings → Environment Variables:

| Variable | Required for |
|----------|-------------|
| `AWS_ACCESS_KEY_ID` | Delta tracking + Textract |
| `AWS_SECRET_ACCESS_KEY` | Delta tracking + Textract |
| `AWS_S3_BUCKET` | Textract only (temp staging) |

The DynamoDB table is created automatically on first use (PAY_PER_REQUEST billing, 90-day TTL).

---

## MCP / Agent integration

The repository ships a standalone MCP server (`mcp-server/`) that wraps the Apify actor as a live tool. Once configured, Claude Desktop, Cursor, or any MCP-compatible AI client can call it directly during a conversation — no API calls or copy-pasting required.

### Tool exposed

| Tool | Description |
|------|-------------|
| `scrape_documentation_for_rag` | Crawl a documentation URL and return semantically chunked, LLM-ready text with full source metadata. Accepts `url`, `outputFormat`, `chunkSize`, `maxPages`, `maxDepth`, `urlPattern`, and `versionFilter`. |

### Prerequisites

```bash
# Node 18+ required
git clone https://github.com/LernowaAdmin/rag-docs-scraper.git
cd rag-docs-scraper/mcp-server
npm install
npm run build          # compiles TypeScript → dist/index.js
```

### Claude Desktop (local stdio)

Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}
```

Restart Claude Desktop. The tool appears automatically in the tool palette.

### Cursor (local stdio)

Edit `~/.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "rag-docs-scraper": {
      "command": "node",
      "args": ["/absolute/path/to/rag-docs-scraper/mcp-server/dist/index.js"],
      "env": {
        "APIFY_TOKEN": "apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}
```

Reload Cursor (`Cmd/Ctrl + Shift + P` → "Reload Window").

### Remote / Docker (HTTP + SSE transport)

For hosted deployments, the server supports SSE over HTTP:

```bash
cd mcp-server
MCP_TRANSPORT=http MCP_PORT=3001 APIFY_TOKEN=apify_api_... node dist/index.js
```

Then point your MCP client at `http://your-host:3001/sse`.

The `mcp-server/Dockerfile` builds a production-ready image:

```bash
cd mcp-server
docker build -t rag-docs-mcp .
docker run -p 3001:3001 -e APIFY_TOKEN=apify_api_... -e MCP_TRANSPORT=http -e MCP_PORT=3001 rag-docs-mcp
```

### Apify-hosted MCP gateway (no local server needed)

Apify exposes every actor's API as an MCP-compatible tool via their MCP gateway. Add this to your client config for a zero-install option:

```json
{
  "mcpServers": {
    "apify": {
      "type": "sse",
      "url": "https://mcp.apify.com/sse",
      "headers": {
        "Authorization": "Bearer apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxx"
      }
    }
  }
}
```

See `developer_quickstart/mcp_client_config_example.json` for all three variants side by side.

---

## FAQ

**The actor crawls only one page — why aren't more pages enqueued?**

A redirect often changes the URL structure after navigation. Check the actual URL in the logs (it may be `/en/docs/` instead of `/docs/`) and update `startUrls` and `urlPatterns` to match the post-redirect path.

**Chunks are empty or very short on a JavaScript-rendered site.**

The page content may not be rendered when Playwright first loads it. Add the main content selector to `excludeSelectors`'s inverse — or simply let the actor wait by setting `navigationTimeoutSecs` to a higher value (default: 30 s).

**I'm seeing nav menu text in every chunk.**

Add the offending element's CSS selector to `excludeSelectors` (e.g. `[".custom-nav", "#site-sidebar"]`). The actor already strips the most common patterns automatically.

**Versioned docs produce duplicate chunks.**

Use `versionFilter` to restrict crawling to one version (e.g. `/v3/`), or leave `deduplicateContent: true` to drop identical chunks regardless of which version URL they came from.