<div align="center">

# RAG, Simplified

**Learn how Retrieval-Augmented Generation actually works — then run one.**

An interactive, animated lab that takes RAG apart stage by stage, over a real
index you can query. Every knob is live; the whole engine is a few hundred
readable lines. No cloud, no API keys, runs entirely on your machine.

[Features](#features) · [Quickstart](#quickstart) · [How it works](#how-it-works) ·
[Web UI](#the-web-ui) · [MCP for agents](#use-it-from-an-agent-mcp) ·
[Repo tour](#repo-tour--your-learning-path) · [Roadmap](#roadmap)

</div>

---

## What this is

Most RAG explainers show a diagram and a black box. This one shows the diagram,
then lets you open the box: query a real index, toggle dense vs. keyword search,
watch documents climb through fusion and reranking, and *see* the embedding space
the retriever searches.

It is two things at once:

- **A course.** Seven short, precise chapters covering chunking, embeddings,
  vector search, hybrid retrieval, fusion, reranking, and the failure modes.
- **A working product.** A real hybrid RAG engine (dense + BM25 + RRF + a
  cross-encoder), a FastAPI server, an animated web UI, and an MCP server so
  agents can query it as a tool.

```
documents ──✂──▶ chunks ──🧮──▶ vectors
                                   │
                                   ▼
                                [ index ]                ← the offline half
                                   ▲
                                   │
question ──🎯──▶ retrieve ─────────┘
                   │  dense + BM25 → RRF → rerank
                   ▼
             augment prompt ──▶ LLM ──▶ answer        ← the online half
```

**The one rule this project teaches:** *the retrieval is the product.* Everything
else is plumbing around it.

## Features

- **Real hybrid retrieval** — dense embeddings **+** BM25 keyword search, fused
  with **Reciprocal Rank Fusion**, then re-scored by a **cross-encoder reranker**.
- **Live playground** — toggle each technique on/off and watch the ranking change.
- **Pipeline view** — every query shows all four stages (dense / BM25 / fused /
  reranked) side by side, so you *see* fusion and reranking do their job.
- **Embedding-space map** — the real 384-dim vectors projected to 2D with PCA;
  your query lands and its nearest neighbours light up.
- **Index any website** — point it at a docs site and crawl it into the corpus.
- **Pluggable sources** — bundled docs, a website crawler, and two Learn Anything
  adapters (a local export, and the mirrored `linsa-io` markdown corpus); add your
  own in ~15 lines.
- **MCP server** — expose search/answer as tools to Claude, Cursor, Hermes, any
  MCP client.
- **Works with zero setup** — no LLM? It still answers by returning the best
  passages. Add a local LLM and generation turns on automatically.

## Quickstart

```bash
git clone https://github.com/Nilabh-code/rag-simplified
cd rag-simplified

python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt

python -m engine.cli ingest                       # build the index
python -m engine.cli serve --port 8320            # open http://127.0.0.1:8320
```

First run downloads two small models (an embedding model ~90 MB and a reranker
~90 MB). After that it is fully offline.

Prefer the terminal?

```bash
python -m engine.cli query "why is chunk size a trade-off?"
python -m engine.cli query "combine keyword and semantic search" --trace   # show every stage
python -m engine.cli query "what is reranking?" --answer                  # retrieve + generate
```

## How it works

The pipeline is a funnel. Cheap, high-recall stages first; an expensive,
high-precision stage last.

| # | Stage | File | What it does |
|---|-------|------|--------------|
| 1 | **Ingest** | `engine/ingest.py` | sources → documents → chunks → embeddings → index |
| 2 | **Chunk** | `engine/chunking.py` | split text on paragraph boundaries, ~900-char windows with overlap |
| 3 | **Embed** | `engine/embeddings.py` | text → 384-dim vector where *meaning is distance* (L2-normalised) |
| 4 | **Store** | `engine/store.py` | `chunks.jsonl` + `vectors.npy`, kept row-for-row in lockstep |
| 5 | **Retrieve** | `engine/retriever.py` | dense **+** BM25 → RRF fuse → cross-encoder rerank → diversify |
| 6 | **Answer** | `engine/answer.py` | paste the top chunks into the prompt; answer only from them |
| 7 | **Visualise** | `engine/visualize.py` | PCA of the real vectors so you can see the space |

Why each technique exists:

- **Dense search** matches *meaning* ("stale price" finds "outdated quote") but
  blurs exact tokens like `alg=none`.
- **BM25** nails exact tokens but cannot match synonyms.
- **RRF** fuses the two ranked lists by *rank*, so two incomparable score scales
  can be combined fairly.
- **Reranking** runs a cross-encoder over the shortlist — reading query and
  document *together* — so the genuinely best chunk lands at rank 1.

## The web UI

`python -m engine.cli serve` then open `http://127.0.0.1:8320`. You get:

- a **playground** with per-technique toggles and a top-k slider,
- the **pipeline view** animating all four retrieval stages for your query,
- the **embedding-space map** with your query and its nearest neighbours,
- a **corpus** panel that lists what's indexed and lets you crawl a website,
- and an **answer** panel showing the exact augmented prompt sent to the LLM.

## Index learn-anything.xyz (the real corpus)

[learn-anything.xyz](https://learn-anything.xyz/) is a JavaScript SPA behind Clerk
auth — every route returns the same empty shell, there is no public API and no
sitemap, so the *site itself* cannot be crawled. But the knowledge it is built from
is public markdown: the `linsa-io` GitHub org (where `github.com/learn-anything`
301-redirects) holds the curated-list repos (books, courses, websites, research
papers, …) **and** the archived Obsidian topic vault — thousands of interlinked
notes forming a genuine link graph.

`scripts/fetch_learn_anything.py` mirrors that graph: it seeds from the hub and the
vault, downloads each file's raw markdown, and **follows every markdown link it
finds**, recursing until the frontier is empty (explicit `.md` links and repo
readmes; pass `--follow-repos` to also chase every bare repo link).

```bash
export GITHUB_TOKEN=$(gh auth token)          # lifts GitHub API rate limits
python scripts/fetch_learn_anything.py        # -> data/corpus/learn-anything/
python -m engine.cli ingest                   # bundles it into the index
```

The mirror lands as `<repo>/<path>.md` with `_manifest.jsonl` (file→URL map) and
`_links.jsonl` (the link-graph edge list). `engine/sources/learn_anything_vault.py`
reads it as the `learn-anything-vault` source, which the CLI, API and MCP server all
include automatically once a mirror exists.

## Index any website

```bash
python -m engine.cli ingest --url https://en.wikipedia.org/wiki/Retrieval-augmented_generation --max-pages 25
```

Or use the corpus panel in the web UI. The crawler is same-domain and capped;
bundled docs stay in the index.

## Use it from an agent (MCP)

The engine ships an MCP server, so any MCP-capable agent can search and answer
over your knowledge base as tools.

```bash
python -m engine.mcp_server            # stdio (for local clients)
python -m engine.mcp_server --http --port 8321   # HTTP for remote clients
```

Register it (e.g. `claude_desktop_config.json`):

```json
{
  "mcpServers": {
    "rag-simplified": {
      "command": "python",
      "args": ["-m", "engine.mcp_server"],
      "cwd": "/absolute/path/to/rag-simplified"
    }
  }
}
```

Tools exposed: `rag_query`, `rag_answer`, `rag_stats`, `rag_corpus`,
`rag_add_website`.

## Repo tour — your learning path

Read the repo in this order; each file teaches one idea.

```
engine/
  chunking.py      text  → chunks           (why chunk? size, overlap)
  embeddings.py    chunks/questions → vectors (meaning as distance)
  store.py         save/load the index      (the row-order contract)
  retriever.py     vectors → best chunks    (dense + BM25 + RRF + rerank)
  answer.py        chunks → grounded answer  (the "G" in RAG + prompt)
  visualize.py     384-dim → 2D             (PCA, so you can see it)
  ingest.py        the offline half, wired end to end
  api.py           HTTP server for the UI
  cli.py           ingest / query / serve / stats
  sources/         where documents come from (pluggable)
scripts/
  fetch_learn_anything.py   mirror the learn-anything.xyz markdown link graph
web/               the animated, interactive front-end
data/corpus/       the bundled learning chapters (also a retrievable corpus)
mcp/               MCP setup notes
tests/             chunking units + an end-to-end ingest→query test
```

The `data/corpus/*.md` files are both the **course** and the **corpus** — the app
retrieves from the same chapters you read. Ask it "why is reranking useful?" and
the answer comes back from chapter 6.

## Configuration

Everything tunable lives in `config.json`:

```json
{
  "embed_model": "sentence-transformers/all-MiniLM-L6-v2",
  "rerank_model": "cross-encoder/ms-marco-MiniLM-L6-v2",
  "chunk_size": 900,
  "chunk_overlap": 120,
  "top_k": 5,
  "rerank": true
}
```

Change `chunk_size`, re-run `ingest`, and watch retrieval move — chunk size is
the dial people most underestimate.

## Roadmap

- [ ] Swap brute-force search for an **ANN index** (HNSW / FAISS) at scale.
- [ ] **Evaluation harness**: a golden set, Recall@k, MRR, answer faithfulness.
- [ ] More sources: PDF, GitHub repos, Notion, an RSS watcher.
- [ ] Async ingest with live progress in the UI.
- [ ] More embedding/reranker models selectable from the UI.

## Contributing

Issues and PRs welcome — especially new **sources**, more **course chapters**, and
**evaluation** tooling. See `CONTRIBUTING.md`.

## License

MIT © 2026 Nilabh Yadav — see `LICENSE`.

<div align="center"><sub>Built with numpy, sentence-transformers, rank_bm25, FastAPI, and the MCP SDK.</sub></div>