# OnMind-RAG

A **complementary and independent** project from OnMind-PUB: a lightweight knowledge base (light RAG) that exposes an **MCP server** so an agent (MCP client) can query the content of one or more sites.

It is not part of the `init → index → build → publish` workflow. It only **reads** the `_index.json` already generated by a site and, on demand, the markdown.

## Idea

1. On startup, the server loads `docs/public/_index.json` from each configured site.
2. Normalizes in memory (stable IDs, clean tags, `visibility`, path to `.md`).
3. **Body cache:** reads each `.md` once, stores `bodyText` (no frontmatter) and logs time/size to stderr.
4. **Search engine:** **Orama** BM25 full-text by default; `hybrid`/`vector` modes opt-in with embeddings.
5. Exposes compact MCP tools: summary → search (meta + body) → document reading.

No `_catalog.json` is written. If the site re-indexes externally, use the `reload_catalog` tool.

## Requirements

- Node.js ≥ 18 (or Bun with Node compatibility)
- At least one site with `docs/public/_index.json` (e.g. `sites/know` from OnMind-PUB)

## Setup

```bash
cd rag
cp .env.example .env   # optional; you can also export vars
bun install            # or: npm install
```

Variables (see `.env.example`):

| Variable | Meaning |
|----------|---------|
| `RAG_SITES` | Paths to site roots, comma-separated (required) |
| `RAG_SITE_NAMES` | Optional labels (same order) |
| `RAG_VISIBILITY` | `public` (default) \| `protected` \| `all` |
| `RAG_MAX_BODY` | Max chars returned by `read_document` (default 50000; increase for very long docs) |
| `RAG_CACHE_BODY` | `1` (default) caches body at startup → Orama indexes body → **full-text on markdown**. `0` = metadata only (title/desc/tags) → **no body search**; `read_document` falls back to disk. |
| `RAG_SEARCH_MODE` | `fulltext` (default, BM25) \| `hybrid` \| `vector` (last two require `RAG_EMBEDDINGS=1`) |
| `RAG_EMBEDDINGS` | `1` enables local hash embeddings for hybrid/vector; `0` (default) |
| `RAG_EMBED_DIMS` | Vector dimensions (default 384) |
| `RAG_SNAPSHOT` | `1` saves/loads Orama snapshot to disk; `0` (default) |
| `RAG_SNAPSHOT_PATH` | Snapshot path (default `./data/orama-snapshot.json`) |

Example:

```bash
export RAG_SITES=../sites/know
# or multiple:
# export RAG_SITES=../sites/know,../sites/andrey
```

## Smoke test (no MCP)

```bash
RAG_SITES=../sites/know bun run smoke
```

With embeddings + hybrid:

```bash
RAG_SITES=../sites/know RAG_EMBEDDINGS=1 RAG_SEARCH_MODE=hybrid bun run smoke
```

With snapshot (2nd load ~100 ms):

```bash
RAG_SITES=../sites/know RAG_SNAPSHOT=1 bun run smoke   # first: saved
RAG_SITES=../sites/know RAG_SNAPSHOT=1 bun run smoke   # second: loaded
```

## Start MCP server (stdio)

```bash
RAG_SITES=../sites/know bun run mcp
```

### MCP Client (Cursor / Claude Desktop / Grok / Jan)

```json
{
  "mcpServers": {
    "onmind-rag": {
      "command": "bun",
      "args": ["/absolute/path/to/pub/rag/src/server.js"],
      "env": {
        "RAG_SITES": "/absolute/path/to/pub/sites/know",
        "RAG_VISIBILITY": "public",
        "RAG_SEARCH_MODE": "fulltext",
        "RAG_EMBEDDINGS": "0",
        "RAG_SNAPSHOT": "0"
      }
    }
  }
}
```

> You can also use `node` instead of `bun`

```json
{
  "command": "node",
  "args": ["/absolute/path/to/pub/rag/src/server.js"],
  "env": { "RAG_SITES": "/absolute/path/to/pub/sites/know" }
}
```

## Tools

| Tool | Purpose |
|------|---------|
| `list_sites` | Loaded sites, body-cache stats, search-engine status (Orama, embeddings, snapshot) |
| `catalog_summary` | Stats by category / language / tags (orientation) |
| `search_content` | **Full-text Orama (BM25)** by default on title/description/tags/body. `mode: fulltext\|hybrid\|vector` optional. Returns cards with `match.score` and `match.snippet`. |
| `get_entry` | One record by `id` (`site:url`) |
| `read_document` | Markdown body (preferably from cache; truncatable) |
| `list_series` | Series ordered by `filename` within a category |
| `reload_catalog` | Re-reads `_index.json` + body cache + rebuilds index |

Typical agent flow:

```text
catalog_summary → search_content → get_entry / list_series → read_document
```

## Relation to OnMind-PUB

| OnMind-PUB | onmind-rag |
|------------|------------|
| Generates `_index.json` and the static site | Only consumes it |
| Publish workflow | Outside that flow |
| Lives in monorepo for convenience | Movable: just point `RAG_SITES` to any folder with `docs/public/_index.json` |

## Body cache (startup)

In stderr you'll see something like:

```text
[onmind-rag] index: 289 entries from 1 site(s) in 3ms at ...
[onmind-rag] body cache: 289 docs, 3533.3 KiB in 59ms (missing path=0, read errors=0)
[onmind-rag] search engine: orama · mode=fulltext · embeddings=off · 289 docs · 450ms
```

**With `RAG_CACHE_BODY=1` (default):** Orama indexes the `body` field → `search_content` performs **full-text BM25 over the entire markdown**. `read_document` answers from memory (`fromCache: true`).

**With `RAG_CACHE_BODY=0`:** Orama **only indexes metadata** (title, description, tags) → `search_content` **does not find matches in the document body**. `read_document` falls back to disk reads.

Disable only if RAM is very tight or corpus > 50 MB and you accept metadata-only search.

## Search engine (Orama)

| Mode | What it does |
|------|--------------|
| `fulltext` (default) | BM25 with stemming, typo tolerance, field boosting. Fast, no external deps. |
| `hybrid` | BM25 + vector (cosine) — needs `RAG_EMBEDDINGS=1` |
| `vector` | Vector similarity only — needs `RAG_EMBEDDINGS=1` |

### Embeddings (opt-in)

`RAG_EMBEDDINGS=1` uses **local feature-hashing** (no TensorFlow, no API keys) — a portable baseline to exercise the vector/hybrid path. Not a SOTA semantic model. For real quality, replace `embed.js` with a provider (OpenAI, Xenova/transformers.js, etc.) keeping the `embedText(text, { dims })` interface.

### Snapshot (opt-in)

`RAG_SNAPSHOT=1` serializes the Orama index to disk (`data/orama-snapshot.json` + `.meta.json`). On subsequent starts, if the **corpus fingerprint** (ids, titles, body length, hide, embeddings flag/dims) matches, it loads the snapshot in ~100 ms instead of re-indexing (~450 ms).

## Limits (by design)

- Lexical/BM25 retrieval + hash vectors (no SOTA embeddings unless you plug them in).
- No knowledge graph / neighbors yet (next step: markdown links).
- Designed as a useful, portable base — not a managed vector engine.

## Why Spanish tokenizer?

The corpus is predominantly Spanish (~80%). Orama's default English tokenizer treats short technical codes like "XDB", "IAM", "UCDM" as stop-words/noise. Setting `language: 'spanish'` in the tokenizer indexes them correctly.

---

*Based on onmind-rag v0.2.0 — MCP server for knowledge retrieval with Orama BM25 + opt-in hybrid/vector + snapshot*