# Collibra Atlas

*Local-first AI knowledge layer for Collibra documentation via MCP.*

Atlas is a local-first knowledge layer for Collibra development. Two
[Model Context Protocol](https://modelcontextprotocol.io) servers expose
the entire [Collibra developer documentation](https://developer.collibra.com/)
to any AI agent:

- **collibra-fs** — deterministic filesystem server backed by `ripgrep`. Tools:
  list publications, list files, read file, full-text search, get release info.
  No model, no embeddings, no state. ~zero startup.

- **collibra-rag** — semantic search over precomputed embeddings plus
  fielded BM25 keyword search with title/heading/file-path weighting. Tools:
  search_docs, search_code, get_chunk, get_bundle_info. Three modes:
  `vector` (dense), `keyword` (fielded BM25), `hybrid` (score-level fusion).
  Loads a portable bundle once at startup (~1 GB); answers
  queries in ~1-2 ms (MLX on Apple Silicon) or ~50-200 ms (ONNX+CPU).

The RAG bundle is **built once** by the maintainer locally, then distributed
as a single download via `atlas-download`. End users never embed, never chunk,
never run a vector database, never pull a model.

---

## Table of contents

1. [Why two servers and not one?](#1-why-two-servers-and-not-one)
2. [Tech stack & rationale](#2-tech-stack--rationale)
3. [Project structure](#3-project-structure)
4. [For users: install and use](#4-for-users-install-and-use)
5. [For maintainers: building and releasing](#5-for-maintainers-building-and-releasing)
6. [Operating](#6-operating)
7. [Roadmap](#7-roadmap)
8. [Platform support & caveats](#8-platform-support--caveats)
9. [Troubleshooting](#9-troubleshooting)
10. [Development](#10-development)
11. [Validate RAG quality](#11-validate-rag-quality)
12. [Contributing](#contributing)
13. [License](#license)

---

## 1. Why two servers and not one?

A single RAG pipeline forces every consumer into a fixed retrieval
strategy and a fixed embedding model. Splitting into a filesystem
server and a RAG server buys:

- **Different strengths.** The filesystem server is unbeatable for
  "list the publications," "read this specific file," and "grep for
  the exact symbol." The RAG server is unbeatable for "find docs
  that talk about X" where the wording is fuzzy.
- **Different trust levels.** The filesystem server returns the
  verbatim markdown — no embedding can ever lie about it. The RAG
  server returns ranked candidates; the model should still verify
  with the filesystem server for anything load-bearing.
- **Different cost profiles.** The filesystem server is ~zero startup.
  The RAG server loads ~1 GB of vectors at startup but answers in
  ~100 ms. A smart client uses both: RAG to discover, FS to verify.
- **Different model fits.** A weak local model struggles with
  multi-hop FS navigation but does fine with RAG top-k. A
  strong external model can do either, and benefits from using both.

---

## 2. Tech stack & rationale

| Component | Choice | Why |
|-----------|--------|-----|
| **MCP transport** | Official Python `mcp` SDK (stdio) | The standard. Works in Zed, opencode, Claude Desktop, Continue, any MCP client. |
| **Filesystem search** | `ripgrep` subprocess | 10-100x faster than Python `re` over thousands of files. Single binary, well-maintained. |
| **Embedding model** | `Xenova/bge-base-en-v1.5` (ONNX) / `BAAI/bge-base-en-v1.5` (PyTorch) | 110M params, 768-dim, MTEB top-30. Pre-exported ONNX graph ships from HF. |
| **Inference runtime** | Layered: MLX -> ONNX+CUDA -> ONNX+CPU | Apple Silicon gets MLX (1-2 ms/query). Linux + NVIDIA gets ONNX+CUDA. Everything else gets ONNX+CPU. |
| **Vector store** | `numpy` `.npy` arrays | At 100k x 768 dims, a single matrix multiply is ~10-20 ms. FAISS is overkill. |
| **Chunk metadata** | `pyarrow` Parquet (snappy compressed) | Columnar, compressed, zero-copy reads. No pandas import overhead. |
| **Embedding dtype** | `float16` vectors, `float32` norms | Cosine similarity is rank-preserving under half precision. Halves bundle size. |
| **Tokenizer** | `transformers` `AutoTokenizer` | First-class support for BGE fast tokenizer. |
| **Docs source** | `https://developer.collibra.com/llms.txt` | Official LLM-friendly markdown index. Single flat file listing all 673 pages. |
| **Chunking** | H2-boundary sections per markdown file + 150-char overlap between adjacent sections | Respects the docs' deliberate structure. Overlap catches boundary-spanning queries. |
| **Chunk metadata** | Includes `cluster_tags` — space-joined sibling stems in the same directory | Enables path-aware relevance boost at query time without re-embedding. |
| **Distribution** | GitHub Releases (per-tag) | Simple, free, CLI-friendly API. End users download with one command. |
| **Doc crawler** | aiohttp + GitBook syntax cleaner | Async, rate-limited, with jitter and rotating User-Agent pool. Strips GitBook extensions before chunking. |
| **Keyword search** (build-time) | `rank-bm25` BM25Okapi (fielded) + request-side query expansion (morphological variant generation) | Indexed at bundle-build time from chunk texts + titles + headings + file paths with per-field weights (text=1.0, title=3.0, heading=2.0, file_path=2.5). Pickled state (~a few MB for 250k docs). Query expansion bridges singular/plural and verb-form gaps without stemming the corpus. Enables `keyword` and `hybrid` search modes. |
| **Re-ranking** (opt-in) | `BAAI/bge-reranker-v2-base` (MLX) or `cross-encoder/ms-marco-MiniLM-L6-v2` (ONNX) | 110M params (MLX), 22.7M params (ONNX). Auto-selects MLX on Apple Silicon, falls back to ONNX. Enabled with `--rerank`. |
| **Package management** | `uv` | Fast resolver, lockfile, virtualenv, build system. |

---

## 3. Project structure

```
collibra-atlas/
├── README.md                                 This file
├── pyproject.toml                            uv-managed deps, console-script entry points
├── .gitignore
│
├── atlas/                                    Core Atlas package (importable as `atlas`)
│   ├── __init__.py                           Version + package docstring
│   ├── chunk.py                              H2-boundary markdown chunker (with --clean-gitbook)
│   ├── log.py                                Structured logging via structlog
│   ├── fs_server.py                          Filesystem MCP server (ripgrep-backed)
│   ├── rag_server.py                         RAG MCP server (auto-selects backend)
│   ├── make_bundle.py                        Bundle builder (chunk -> embed -> write)
│   ├── download.py                           Bundle downloader + SHA256 verification
│   ├── backup.py                             Snapshot current bundle
│   ├── restore.py                            Roll back to a previous snapshot
│   ├── doctor.py                             Installation diagnosis + backend probe
│   ├── bm25_search.py                        BM25 keyword index (build-time, optional)
│   ├── smoke_test.py                         E2E validation (build + search)
│   ├── evaluate.py                           RAG quality (Precision@10, MRR)
│   ├── rerank.py                             Cross-encoder re-ranker (MiniLM-L6-v2 ONNX, portable)
│   ├── rerank_mlx.py                         Cross-encoder re-ranker (bge-reranker-v2-base MLX, Apple Silicon)
│   ├── embed/                                Embedding backends (factory pattern)
│   │   ├── __init__.py
│   │   ├── base.py                           ABC + factory + resolve_backend
│   │   ├── onnx.py                           OnnxEmbedder (portable, CPU/CUDA)
│   │   └── mlx.py                            MlxEmbedder (Apple Silicon)
│   └── sources/                              Source adapters for markdown input
│       ├── __init__.py
│       ├── base.py                           MarkdownSource ABC
│       ├── git.py                            Git repo source
│       ├── web_crawl.py                      Web crawl mirror source
│       └── local.py                          Local directory source
│
├── collibra_docs_nav/                        Collibra-specific doc crawler
│   ├── __init__.py
│   └── crawler.py                            Async llms.txt crawler with GitBook syntax cleaning
│
├── scripts/
│   └── publish-bundle.sh                     Local build + gh release create
│
├── tools/
│   ├── convert_bge_to_mlx.py                 Convert BGE embedder weights to MLX .npy
│   └── convert_reranker_to_mlx.py            Convert BGE reranker weights to MLX .npy
│
├── tests/
│   ├── __init__.py
│   ├── test_chunk.py                         Chunker tests
│   ├── test_embed.py                         Embedding backend tests
│   ├── test_bm25_search.py                   BM25 keyword index tests
│   ├── test_fs_server.py                     FS MCP server tests
│   ├── test_rag_server.py                    RAG MCP server tests
│   ├── test_sources.py                       Source adapter tests
│   ├── test_backup_restore.py                Backup/restore tests
│   ├── test_doctor.py                        Doctor diagnosis tests
│   ├── test_download.py                      Bundle download tests
│   └── test_rerank_mlx.py                    MLX reranker tests (skipped on non-MLX)
│
├── data/                                     Runtime data (gitignored, created at runtime)
```

---

## 4. For users: install and use

### 4.1 Prerequisites

- **Python 3.11+** (3.12 or 3.13 recommended).
- **[`uv`](https://docs.astral.sh/uv/)** for Python environment management.
- **[`ripgrep`](https://github.com/BurntSushi/ripgrep)** (`brew install ripgrep` or `apt install ripgrep`).
- ~1.5 GB of free disk for the bundle.

> **Platforms.** Atlas runs on three classes of host, each picking a
> different embedding backend by default:
>
> - **Apple Silicon (M1/M2/M3/M4).** Default backend is MLX, which talks
>   to the Apple Neural Engine directly. ~1-2 ms per query. Install with
>   `uv sync --extra mlx`.
> - **Linux x86_64 with an NVIDIA GPU.** Default backend is ONNX Runtime + CUDA.
>   ~1-2 ms per query. Install with `uv sync --extra gpu`.
> - **Anything else** falls back to ONNX+CPU. ~50-200 ms per query.

### 4.2 Install

```bash
git clone <this-repo-url> collibra-atlas
cd collibra-atlas
uv sync
```

**Optional extras** (pick what your machine can use):

```bash
# Apple Silicon: MLX embedder
uv sync --extra mlx

# Linux with NVIDIA GPU: ONNX Runtime + CUDA
uv sync --extra gpu

# Both
uv sync --extra mlx --extra gpu
```

> **Re-applying extras on subsequent `uv sync`.** `uv sync` synchronizes
> the venv to match the lockfile plus whichever extras you specify on the
> command line. If you later run plain `uv sync`, the MLX/GPU packages
> will be silently removed. Keep them by re-running with the same flags:
> `uv sync --extra mlx`. After a sync, `atlas-doctor` will tell you
> immediately if a previously-installed backend is now MISS.

### 4.3 Download the pre-built bundle

```bash
uv run atlas-download \
  --repo <owner>/collibra-atlas-bundles \
  --output ./data/collibra-rag-bundle
```

This downloads, verifies SHA256, and extracts the bundle. If a previous
bundle exists, it's snapshotted into `./data/collibra-rag-bundle/.backups/`
first (most recent 5 kept by default).

Pin to a specific release:

```bash
uv run atlas-download \
  --repo <owner>/collibra-atlas-bundles \
  --tag v2026.07 \
  --output ./data/collibra-rag-bundle
```

### 4.4 Get the docs mirror (for filesystem server)

Download the pre-crawled mirror from GitHub Releases:

```bash
uv run atlas-download \
  --repo <owner>/collibra-docs-mirror \
  --output ./data/collibra-docs
```

Or crawl it yourself (see [5.1 Crawl Collibra documentation](#51-crawl-collibra-documentation)).

### 4.5 Configure your IDE

Both servers speak MCP over stdio, so any client that supports
[the standard](https://modelcontextprotocol.io) works: Zed, opencode,
Claude Desktop, Continue, and others.

#### Zed (`~/.config/zed/settings.json`)

```json
"context_servers": {
  "collibra-fs": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
    ],
    "timeout": 60
  },
  "collibra-rag": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
    ],
    "timeout": 120
  }
}
```

> **Search mode** is configured per-query via the `mode` parameter
> (`vector`, `keyword`, or `hybrid`) on each `search_docs`/`search_code`
> call. See [Search Modes](#search-modes).

#### opencode (`~/.config/opencode/profiles/default.json`)

```json
"mcpServers": {
  "collibra-fs": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
    ]
  },
  "collibra-rag": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
    ]
  }
}
```

#### Claude Desktop (`~/.config/Claude/claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "collibra-fs": {
      "command": "uv",
      "args": [
        "run", "--directory", "/absolute/path/to/collibra-atlas",
        "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
      ]
    },
    "collibra-rag": {
      "command": "uv",
      "args": [
        "run", "--directory", "/absolute/path/to/collibra-atlas",
        "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
      ]
    }
  }
}
```

### 4.6 Quick CLI sanity check

```bash
# Confirm both servers start
uv run collibra-fs --help
uv run collibra-rag --help

# Run diagnosis
uv run atlas-doctor

# Run smoke test
uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs
```

---

## 5. For maintainers: building and releasing

### 5.0 First-time build walkthrough

A guided path for building the bundle on your own machine, from a
fresh `uv sync` to a working bundle.

**Preflight.** Confirm the embedding backend you expect is available:

```bash
uv sync --extra mlx        # or --extra gpu on Linux + NVIDIA
uv run atlas-doctor        # should report: MLX OK, selected: mlx
```

**Convert MLX weights (2 minutes, one-time).** The MLX backend needs
pre-converted weight files. This uses PyTorch temporarily to read the
Hugging Face checkpoint and write 197 `.npy` files to the MLX cache.

```bash
uv pip install torch
uv run python tools/convert_bge_to_mlx.py
uv pip uninstall torch --yes
```

The tokenizer (`transformers.AutoTokenizer`) works without PyTorch —
the warning about "PyTorch was not found" is harmless. PyTorch is only
needed for this one-time weight conversion.

**Smoke crawl (2 minutes).** Test the crawler with a subset:

```bash
uv run python -m collibra_docs_nav.crawler \
  --output ./data/collibra-docs \
  --max-pages 20
```

**Smoke build (2 minutes).** Test the pipeline with a tiny bundle:

```bash
uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output /tmp/atlas-smoke \
  --limit 100 \
  --prefer apple \
  --clean-gitbook
```

**Sanity-check the smoke bundle:**

```bash
.venv/bin/python -c "
from atlas.rag_server import Bundle
b = Bundle('/tmp/atlas-smoke', prefer='apple')
hits = b.search('Collibra community', top_k=3)
for h in hits:
    print(f'  {h[\"score\"]:.3f}  {h[\"file\"]} :: {h[\"heading\"]}')
"
```

Using `.venv/bin/python` directly avoids the ~300 ms `uv run` overhead.
For pure JSON inspection of the manifest:

```bash
jq . /tmp/atlas-smoke/manifest.json | head -20
```

**Full crawl (~10 minutes).** Collibra publishes a single flat `llms.txt`
listing all 673 pages — no sub-indexes to discover:

```bash
uv run python -m collibra_docs_nav.crawler \
  --output ./data/collibra-docs
```

**Full build (2-4 hours depending on hardware):**

```bash
uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output ./data/collibra-rag-bundle \
  --prefer auto \
  --clean-gitbook
```

### 5.1 Crawl Collibra documentation

The crawler fetches every `.md` page listed in the flat `llms.txt` index
at `https://developer.collibra.com/llms.txt`. Collibra publishes a **single
flat index** listing all 673 pages.

```bash
# Full crawl (~673 pages, ~10 min)
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs

# Test with subset
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --max-pages 50

# Crawl specific sections (available: api, cli, misc, tutorials, workflows)
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --sections api,tutorials
```

The crawler saves files to `markdown/<section>/<path>.md` and writes
a `crawl_meta.json` with source URL, timestamp, and per-category counts
(pages found, fetched, skipped, errors, cleaned, total bytes).

**GitBook syntax cleaning** is enabled by default. Collibra docs use
GitBook extensions (tabs, columns, hints, icons, HTML tables, `<mark>`
tags) that the crawler strips before saving. To skip cleaning:

```bash
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --no-clean
```

Useful flags:
- `--sections api,tutorials` — crawl only specific sections (available: api, cli, misc, tutorials, workflows)
- `--max-pages 50` — limit total pages fetched
- `--delay-min 0.5 --delay-max 2.0` — control jitter range (default: 0.3-1.5s)
- `--max-concurrent 5` — max concurrent requests (default: 5)
- `--no-clean` — skip GitBook syntax cleaning

After crawling, run the validation script to check for residual syntax:

```bash
uv run python .agents/skills/collibra-docs-nav/scripts/validate_crawl.py ./data/collibra-docs
```

This checks that headers are stripped, frontmatter fields are present, no
GitBook syntax remains, no uncleaned HTML is in regular markdown content
(`<code>` is excluded — it's valid inline formatting), and files are in
the correct directory structure. It intentionally skips HTML inside fenced
code blocks (e.g. `<code>` in OpenAPI JSON specs) to avoid false
positives — see the `collibra-docs-nav` skill for details.

### 5.1.5 MLX weight conversion (Apple Silicon)

The MLX backend requires pre-converted weights from the PyTorch
checkpoint. This is a **one-time step per machine** — once the
`.npy` files are in the cache, every build and query reuses them.

```bash
# Install torch temporarily for the conversion
uv pip install torch

# Run the conversion (writes to ~/.cache/atlas/models/bge-base-en-v1.5-mlx/)
uv run python tools/convert_bge_to_mlx.py

# Uninstall torch — not needed at runtime
uv pip uninstall torch --yes
```

The conversion reads `BAAI/bge-base-en-v1.5` (the PyTorch model)
and writes 197 `.npy` weight files. The MLX embedder then loads
them at `~/.cache/atlas/models/bge-base-en-v1.5-mlx/` and never
touches PyTorch again.

After conversion, `atlas-doctor` reports `MLX OK, selected: mlx`
and `atlas-build --prefer apple` will use the ANE/GPU accelerator.

> **Note:** If the MLX cache is empty, `atlas-build --prefer apple`
> falls back gracefully to ONNX+CPU with a reminder to run the
> conversion script. ONNX+CPU is ~3-5x slower but works without any
> setup.

### 5.2 Build RAG bundle

The build pipeline runs: crawl -> chunk (with overlap + cluster_tags) ->
embed -> fielded BM25 index (text + title + heading + file_path) ->
write artifacts -> stage model -> manifest.

```bash
# Full build
uv run atlas-build \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --output ./data/collibra-rag-bundle \
    --prefer auto \
    --clean-gitbook

# Test build (skip embeddings, limit files)
uv run atlas-build \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --output ./data/test-bundle \
    --limit 100 \
    --skip-embed \
    --clean-gitbook
```

The `--clean-gitbook` flag strips GitBook-specific syntax (tabs,
columns, hint blocks, icon references, HTML tables, `<mark>` tags)
from each markdown file before chunking. This is **required** for
Collibra docs to produce clean chunks.

The build writes a `manifest.json` with the source SHA, chunk count,
BM25 index size, model ID, and SHA256 of each artifact. A **fielded
BM25 index** is built automatically from chunk texts, titles, headings,
and file paths with per-field weights (text=1.0, title=3.0, heading=2.0,
file_path=2.5) via `rank-bm25::BM25Okapi` and saved as `bm25.pkl`.

### 5.3 Validate

```bash
# Run smoke tests
uv run atlas-smoke \
    --bundle ./data/collibra-rag-bundle \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --verbose

# Diagnose installation
uv run atlas-doctor --bundle ./data/collibra-rag-bundle
```

### 5.4 Publishing a release

The bundle is built locally — embedding ~100k chunks on a CI runner is
impractical. On Apple Silicon with MLX the build takes ~30 min; on a
CPU-only machine it takes longer but still finishes.

```bash
# Prerequisites: gh CLI installed and authenticated
#   brew install gh && gh auth login

# Publish the existing bundle (fast, no rebuild)
./scripts/publish-bundle.sh <owner>/collibra-atlas-bundles

# Or rebuild first (takes ~30 min on Apple Silicon)
./scripts/publish-bundle.sh --rebuild <owner>/collibra-atlas-bundles
```

This tars the bundle, creates a GitHub Release tagged `vYYYY.MM`, and
uploads it. Users can then install it with `atlas-download`.

The script lives at `scripts/publish-bundle.sh` — read it to see exactly
what it does before running.

### 5.4.5 MLX reranker weight conversion (Apple Silicon)

The MLX cross-encoder reranker (`atlas/rerank_mlx.py`) requires
pre-converted weights from the Hugging Face checkpoint. This is a
**one-time step per machine**, like the embedder MLX conversion.

```bash
# Install torch temporarily for the conversion
uv pip install torch

# Run the conversion (writes to ~/.cache/atlas/models/bge-reranker-v2-base-mlx/)
uv run python tools/convert_reranker_to_mlx.py

# Uninstall torch — not needed at runtime
uv pip uninstall torch --yes
```

The conversion reads `BAAI/bge-reranker-v2-base` (a BERT-base cross-encoder)
and writes 199 `.npy` weight files (197 encoder weights + classifier
head weight/bias). The MLX reranker loads them at
`~/.cache/atlas/models/bge-reranker-v2-base-mlx/`.

After conversion, the RAG server auto-selects MLX for reranking when
`--rerank` is passed, falling back to ONNX if MLX weights aren't found.

### 5.5 Smoke-test the bundle

After building or downloading a bundle, validate it end-to-end:

```bash
uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs
```

The smoke test confirms the bundle loads, embeddings are queryable, and
the filesystem server can serve files from the mirror.

---

## Architecture

```mermaid
flowchart LR
    subgraph Source["Docs Source"]
        URL[("developer.collibra.com<br/>llms.txt (flat index)")]
    end

    subgraph Crawl["Crawler"]
        CRAWLER["collibra-crawl<br/>aiohttp"]
        CLEAN["GitBook syntax<br/>cleaner"]
        JITTER["jitter + rotating UAs<br/>+ polite headers"]
    end

    subgraph Mirror["Local Mirror"]
        MD["markdown/<br/>section/*.md"]
        META["crawl_meta.json"]
    end

    subgraph Build["Bundle Builder"]
        CHUNK["atlas-build<br/>chunk --clean-gitbook<br/>embed -- write -- bm25"]
        MODEL[("bge-base-en-v1.5<br/>embedding model")]
        BM25_BUILD[("rank-bm25<br/>Fielded BM25<br/>text+title+heading+file_path")]
    end

    subgraph Bundle["RAG Bundle"]
        CHUNKS["chunks.parquet"]
        VECTORS["embeddings.f16.npy"]
        NORMS["norms.f32.npy"]
        BM25_PKL["bm25.pkl"]
        MANIFEST["manifest.json"]
    end

    subgraph Publish["Distribution"]
        RELEASE["GitHub Release<br/>tar.zst"]
        DOWNLOAD["atlas-download<br/>SHA256 verify"]
    end

    subgraph MCP["MCP Servers"]
        FS["collibra-fs<br/>ripgrep-backed"]
        RAG["collibra-rag<br/>vector / keyword / hybrid"]
    end

    subgraph Clients["MCP Clients"]
        ZED["Zed"]
        OPENCODE["opencode"]
        CLAUDE["Claude Desktop"]
        CONTINUE["Continue"]
    end

    URL -->|"flat llms.txt<br/>673 pages"| CRAWLER
    CRAWLER --> CLEAN
    CLEAN --> JITTER
    JITTER -->|"polite fetch"| URL
    CRAWLER -->|"write"| MD
    CRAWLER --> META

    MD -->|"read"| CHUNK
    MODEL -->|"embed"| CHUNK
    BM25_BUILD -->|"index"| CHUNK
    CHUNK -->|"write"| CHUNKS
    CHUNK --> VECTORS
    CHUNK --> NORMS
    CHUNK --> BM25_PKL
    CHUNK --> MANIFEST

    Bundle -->|"tar + gh release"| RELEASE
    RELEASE -->|"download + verify"| DOWNLOAD
    DOWNLOAD -->|"extract to"| Bundle

    MD -->|"read live"| FS
    Bundle -->|"load at startup"| RAG

    Clients -->|"MCP stdio"| FS
    Clients -->|"MCP stdio"| RAG
```

---

## 6. Operating

### 6.1 Update to the latest bundle

```bash
uv run atlas-download --repo <owner>/collibra-atlas-bundles --output ./data/collibra-rag-bundle
```

The current bundle is auto-snapshotted before being replaced. If the new
bundle is broken, the IDE still works against the old directory while
you sort it out.

### 6.2 List and roll back

```bash
uv run atlas-restore --snapshot ./data/collibra-rag-bundle/.backups/snapshot-20260701T120000Z.tar.gz --target ./data/collibra-rag-bundle
```

The backup directory preserves recent snapshots for rollback.

### 6.3 Manual snapshot

```bash
uv run atlas-backup --bundle ./data/collibra-rag-bundle --backup-dir ./data/collibra-rag-bundle/.backups
```

### 6.4 Diagnose with `atlas-doctor`

```bash
uv run atlas-doctor
uv run atlas-doctor --bundle ./data/collibra-rag-bundle
uv run atlas-doctor --refresh    # re-probe backends instead of using cache
uv run atlas-doctor --json       # print raw JSON instead of formatted report
```

Output (short form):

```
Platform       : Darwin 24.5.0 (arm64)
Python         : 3.13.5
ONNX Runtime   : 1.26.0  providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider']

Backend probe:
  ONNX+CPU       OK  always available
  Apple MLX      OK   weights cached
  NVIDIA CUDA    MISS nvidia-smi not on PATH

Selected       : mlx
                Apple Silicon detected and MLX is importable

Bundle         : /Users/me/data/collibra-rag-bundle
                manifest=OK chunks=OK embeddings=OK
                52 chunks, model=Xenova/bge-base-en-v1.5 (SHA ok)
```

`atlas-doctor` is non-destructive and never blocks startup. Run it first
when something is wrong.

---

## Commands Reference

| Command | Description |
|---------|-------------|
| `collibra-fs` | Filesystem MCP server (ripgrep-based) |
| `collibra-rag` | RAG MCP server (vector, fielded BM25 keyword, hybrid search) |
| `collibra-crawl` | Crawl Collibra docs to local mirror |
| `atlas-build` | Build RAG bundle from source |
| `atlas-download` | Download bundle from GitHub Releases |
| `atlas-backup` | Snapshot current bundle |
| `atlas-restore` | Roll back to snapshot |
| `atlas-smoke` | Run E2E smoke tests |
| `atlas-doctor` | Diagnose installation + backend |
| `atlas-evaluate` | Evaluate RAG quality (P@10, MRR) |

---

## Backend Selection

| Platform | Default Backend | Override |
|----------|----------------|----------|
| Apple Silicon (M1/M2/M3/M4) | MLX (~1-2 ms/query) | `--prefer cpu` |
| Linux + NVIDIA GPU | ONNX+CUDA (~1-2 ms/query) | `--prefer cpu` |
| Everything else | ONNX+CPU (~50-200 ms/query) | N/A |

The backend is resolved at **run time** (not build time). The same bundle
works everywhere. Use `atlas-doctor` to see which backend will be selected.

Override examples:

```bash
# Always MLX, even on a machine that would default to CUDA
uv run collibra-rag --prefer apple

# Force the portable floor (useful for debugging)
ATLAS_EMBED_BACKEND=cpu uv run collibra-rag
```

---

## Search Modes

The `collibra-rag` server supports three search modes, selected via the
`mode` parameter on each `search_docs`/`search_code` call:

| Mode | How it works | Best for |
|------|-------------|----------|
| `hybrid` (default) | Score-level fusion (0.6 x vector + 0.4 x normalised BM25) with interleaved candidate pool and AdaGReS heading-aware diversity ranking | Maximum recall with file-diverse top-k — no single file dominates the results |
| `vector` | Cosine similarity on dense embeddings only | General-purpose semantic search |
| `keyword` | Fielded BM25Okapi (body + title + heading + file-path scoring with per-field weights) | Exact identifier lookups (API endpoints, error codes, config options) |

**The mode is per-query**, so a client can use `keyword` for an API
endpoint lookup (`output-module`), `vector` for fuzzy concept search
(`"how do I set up asset mapping"`), and `hybrid` (the default) when
coverage matters most.

The BM25 index is built at bundle-build time. If a bundle lacks
`bm25.pkl` (older bundles), `keyword` and `hybrid` fall back to
a simple title-boost heuristic automatically — no crash, no error.

**Hybrid mode** additionally applies **heading-aware AdaGReS diversity ranking**
to prevent a single file from dominating the top results. Chunks from different
files are preferred; multiple chunks from the same file but different sections
are partially penalised; truly redundant chunks (same file + same heading) are
blocked. This is query-time only — no rebuild needed.

**BM25** also benefits from **request-side query expansion**: morphological
variants ("asset" <-> "assets", "create" <-> "creating") are generated
automatically at query time without stemming the corpus. Identifiers pass
through unchanged.

**Try the difference.** Point any of these at your built bundle:

```bash
# Hybrid — score-level fusion of vector + BM25 (best overall recall)
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('asset mapping configuration', top_k=3, mode='hybrid'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"

# Vector — pure dense embedding similarity
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('asset mapping configuration', top_k=3, mode='vector'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"

# Keyword — fielded BM25 (title/heading/file-path boosted, best for exact names)
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('output-module', top_k=3, mode='keyword'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"
```

---

## Retrieval Enhancements

### Chunk Overlap

Adjacent H2 sections share a small text overlap (150 characters from the
tail of the previous section) so that queries straddling a section boundary
still match. The overlap is prepended without a special marker — the
embedding model treats it as natural context.

- Controlled via `_OVERLAP_CHARS` in `atlas/chunk.py` (default 150).
- Zero runtime cost — embedded at build time.
- Can cause adjacent sections to inherit code-fence flags (`is_code`), which
  is correct: a section that begins with a code snippet from overlap _is_
  relevant for `search_code`.

### Hierarchical File-System Chunking (Path-Aware Boost)

At build time, each chunk is tagged with `cluster_tags` — the space-joined
stems of sibling `.md` files in the same directory. At query time, the RAG
server applies a small relevance boost (`0.03` weight) to chunks whose
`cluster_tags` match query tokens.

This is a lightweight form of path-aware retrieval that surfaces related
docs from the same file-system "neighbourhood" without re-embedding.

- **Build-time:** sibling map computed in `make_bundle.py::build_chunk_table()`.
- **Query-time:** `pc.match_substring` on `cluster_tags` in `Bundle.search()`.
- **Graceful fallback:** old bundles without the column skip the boost.

### Cross-Encoder Reranker (opt-in)

The `--rerank` flag enables a cross-encoder re-ranker that re-scores the
top-100 candidates from hybrid search using a joint query-passage model.

- **MLX backend** (Apple Silicon): `BAAI/bge-reranker-v2-base` — 110M params,
  BERT-base encoder + classification head. Weights converted via
  `tools/convert_reranker_to_mlx.py`.
- **ONNX backend** (portable): `cross-encoder/ms-marco-MiniLM-L6-v2` —
  22.7M params, auto-downloaded from Hugging Face.
- Auto-selects MLX if available, falls back to ONNX.
- Adds ~5 ms/query (MLX) or ~50-200 ms (ONNX+CPU).

### Fielded BM25 with File-Path Signals

The keyword search index is built as a **fielded BM25** that scores matches
in the body text, title, heading, and file path independently, then combines
them as a weighted sum:

| Field | Weight | Why |
|-------|--------|-----|
| `text` (body) | 1.0 | Baseline — the main content |
| `heading` | 2.0 | Section headings summarise the chunk's topic |
| `file_path` | 2.5 | Filename stems like `output-module` are expert-curated relevance tags |
| `title` | 3.0 | Document titles are concise human-written summaries |

BM25 parameters are tuned for long reference docs: `k1=2.0` (higher TF
saturation ceiling) and `b=0.7` (reduced length normalisation penalty).

The file-path field works because Collibra's `.md` filenames are
descriptive identifiers — a query for "output module" directly matches
the path tokens of `api/guides/output-module.md`. The `.md` extension
is stripped before tokenization so only meaningful tokens remain.

- **Build-time:** `atlas/make_bundle.py` extracts the `file` column and
  passes it as the `file_path` field to `build_fielded_index()`.
- **Query-time:** `atlas/bm25_search.py::FieldedBM25Index` computes per-field
  scores and combines them — `rag_server.py` doesn't need to know about fields.
- **Persisted:** Fields and weights are baked into `bm25.pkl` (version 2
  format). Version 1 pickles still load via backward compatibility.
- **Cost:** Zero query-time LLM cost. Build-time overhead ~1-2 s for 250k
  chunks. Pickle increases ~100 KB for 50k chunks with 4 fields.

**Request-side query expansion (zero build cost):** BM25 automatically generates
morphological variants at query time — `"asset"` also matches
`"assets"` and `"assetting"`; `"create"` also matches
`"creating"` and `"created"`. The corpus is never modified or stemmed.
Non-matching generated variants (e.g. `"assett"`) contribute zero score.
~0.01ms per query.

---

## Configuration

Create `~/.config/atlas.toml` to set default backend:

```toml
[backend]
prefer = "auto"  # or "apple", "nvidia", "cpu"
```

Or use environment variable:

```bash
export ATLAS_EMBED_BACKEND=apple
```

---

## 7. Roadmap

### Reasoning agent (`atlas/agent.py`, planned)

What it should eventually be:

- A thin local agent that wraps the two MCP servers for command-line
  use (`atlas-agent "find docs about Collibra community"`)
  for users who don't have an IDE with MCP support.
- Pre-built tool-use prompt templates tuned for Collibra tasks
  (community setup, asset mapping, governance configuration).
- A planner that fans a question out to `collibra-fs` + `collibra-rag` in
  parallel and merges the results.

### Fine-tuning pipeline (`atlas/training.py`, planned)

- Dataset curation from the same RAG bundle used for retrieval.
- QLoRA adapters for local models, with evaluation against MCP tool-calling contracts.

---

## 8. Platform support & caveats

### Supported

- **Apple Silicon (M1/M2/M3/M4), all macOS versions with current
  security updates.** MLX is the default backend on this hardware
  (`uv sync --extra mlx` to install).
- **Linux x86_64 + CPU.** The portable ONNX+CPU floor works everywhere.
- **Linux x86_64 + NVIDIA GPU.** ONNX+CUDA via `uv sync --extra gpu`.

### Not supported

- **Intel Macs.** Falls back to ONNX+CPU. No MLX acceleration.
- **Windows.** `mcp` and `onnxruntime` work on Windows, but `rg` and
  `tar` paths are untested.

### Known limitations

- The embedder's `max_seq_length` is 512 tokens. Chunks larger than
  that are silently truncated. The H2 chunker rarely produces such
  chunks, but the `chunk._hard_split` fallback handles them.
- Chunk overlap can cause adjacent sections to inherit code-fence flags
  (`is_code = True`) when the overlap text carries a code fence from the
  previous section. This is correct behaviour: sections near code are
  relevant for `search_code`.
- Cosine scores are not calibrated. A score of 0.7 is only meaningful
  relative to other scores from the same query. Use `min_score` loosely.
- The bundle targets a single snapshot of Collibra docs. Re-build
  periodically to stay current.
- Collibra docs use **slug URLs** (`/pages/tg5mBbcaVwKS4wowOuGp`) that
  map to flat `markdown/pages/*.md` files. The crawler and chunker handle
  these, but the file paths in search results may look opaque compared to
  the human-readable paths of other documentation sites.

---

## 9. Troubleshooting

### `No 'markdown/' directory at ...`

The filesystem server was started with `--mirror-path` pointing at
the wrong place. The expected layout is
`<mirror>/markdown/<section>/*.md`. If you haven't crawled yet:

```bash
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs
```

### `Bundle manifest missing`

Either `--bundle` points at the wrong directory, or the bundle wasn't
extracted cleanly. Re-run `atlas-download` and let it overwrite.

### `ripgrep (rg) not installed`

```bash
brew install ripgrep        # macOS
apt install ripgrep         # Debian/Ubuntu
```

### `Context server requires timeout` in Zed/opencode

The first RAG query after server startup pays a one-time warmup cost:
~3-5 s for ONNX+CPU, ~1 s for MLX. Raise the MCP `timeout` to 120 or
300 seconds to absorb the warmup.

### `atlas-doctor` says MLX is MISS but I have an M-series

You didn't install the optional MLX extra. Run:

```bash
uv sync --extra mlx
```

After that, `atlas-doctor` will find MLX and it becomes the default.

### Search returns nothing useful

1. Run `get_bundle_info` to confirm the bundle is loaded.
2. Try `search_docs` with a broader query.
3. Use `collibra-fs` to grep for exact terms.
4. Drop `min_score` to 0.0 to see all candidates.

### Crawler gets rate-limited

The crawler uses polite defaults (rotating User-Agents, jittered delays).
If you're still getting rate-limited, increase the jitter range:

```bash
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --delay-min 1.0 --delay-max 3.0
```

### GitBook syntax leaking through in search results

If you see raw GitBook syntax (tabs, columns, icon references) in
search results, the bundle was built without `--clean-gitbook`. Rebuild:

```bash
uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output ./data/collibra-rag-bundle \
  --clean-gitbook
```

### HTML tags visible in validation output

The `validate_crawl.py` script may report HTML tags (like `<code>` or
`<table>`) in API reference files. These are **false positives** when
they appear inside fenced code blocks — the validator intentionally
skips code fence content. Common cases:

- OpenAPI JSON specs contain `<code>null</code>` in description fields.
  These are formatting markers inside JSON, not markdown HTML.
- SQL templates reference `"<table>"` as a table name placeholder.
  These are not HTML table tags.

The validator's HTML check only flags tags in regular markdown content,
not inside ``` fences. If you see unexpected warnings, check whether
the flagged content is inside a code block.

### Console scripts not found after `uv sync`

If `uv run collibra-fs --help` returns "command not found," the package
isn't installed in the venv. Run `uv sync` (no flags) to install the
project itself, not just its dependencies.

---

## 10. Development

### Useful shell recipes

```bash
# Pretty-print the bundle manifest (no `cat`, no `uv run` overhead)
jq . data/collibra-rag-bundle/manifest.json

# If you don't have jq, fall back to stdlib
.venv/bin/python -m json.tool data/collibra-rag-bundle/manifest.json | head -20

# Load a bundle in Python without going through `uv run`
.venv/bin/python -c "
from atlas.rag_server import Bundle
b = Bundle('data/collibra-rag-bundle', prefer='apple')
for h in b.search('Collibra community', top_k=3):
    print(f'{h[\"score\"]:.3f}  {h[\"file\"]}')
"
```

Install `jq` once (`brew install jq` on macOS, `apt install jq` on
Linux) and the first command is the one you'll reach for every time.

### Lint

ruff enforces style *and* docstring conventions (pydocstyle rules are
enabled via `"D"` in `pyproject.toml`):

```bash
uv run ruff check .
# Auto-fix trivial issues (blank lines, punctuation):
uv run ruff check --fix .
```

### Run unit tests

```bash
uv run pytest tests/ -v
```

Runs the full test suite (chunker, embedder, MCP servers, bundle build,
BM25 keyword index).

### Run the smoke test

```bash
uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs
```

### Add a new tool

1. Add a `Tool(...)` entry in `atlas/fs_server.py` or `atlas/rag_server.py`.
2. Add a handler branch in `call_tool`.
3. Keep the handler synchronous (it's already running in the async
   event loop; `asyncio.to_thread` is fine for blocking I/O).

### Change the embedder

Edit `embed.DEFAULT_MODEL_ID` and `embed.EMBEDDING_DIM`. The bundle build
and the runtime server read both. Changing embedding models invalidates
all existing bundles.

### Change the chunker

Edit `atlas/chunk.py`. The schema is documented in the module docstring.
Run `atlas-smoke` after changes. Key parameters:
- `_OVERLAP_CHARS` (150) — tail chars carried between adjacent H2 sections
- `_MAX_CHUNK_CHARS` (8000) — hard-split threshold for oversized sections

### Add a reranker

Two paths exist:
- **MLX (Apple Silicon):** Add a model class in `atlas/rerank_mlx.py` and a
  converter in `tools/convert_reranker_to_mlx.py`. The BERT-base architecture
  is already supported via `atlas/embed/mlx.BgeModel`.
- **ONNX (portable):** Export a Hugging Face cross-encoder to ONNX and add a
  `CrossEncoderReranker` subclass in `atlas/rerank.py`. Registration is
  automatic via the try/except in `rag_server.py`.

---

## 11. Validate RAG quality

The project ships a golden set of queries covering the major Collibra
documentation areas (community setup, asset mapping, governance, API guides,
etc.). To evaluate retrieval quality:

```bash
# Vector search (cosine similarity on dense embeddings)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode vector

# Keyword search (fielded BM25 with query expansion)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode keyword

# Hybrid search (vector + BM25 fusion with AdaGReS diversity)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode hybrid

# With cross-encoder re-ranker (more accurate, slower)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode hybrid \
  --rerank

# Compare all three modes on a specific query:
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle', prefer='apple')
for mode in ('keyword', 'vector', 'hybrid'):
    print(f'--- {mode} ---')
    for r in b.search('asset mapping configuration', top_k=5, mode=mode):
        print(f'  {r[\"score\"]:.3f}  {r[\"file\"]}')
"
```

This measures **Precision@k** (fraction of top-k results matching expected
files) and **Mean Reciprocal Rank** (inverse rank of first relevant result)
over the full golden set.

Add `--rerank` to use the cross-encoder re-ranker (slower but more
accurate — re-scores top-100 candidates with a joint query-passage model).
Uses MLX on Apple Silicon, ONNX elsewhere. Add `--output results.json`
to save results to a file. The `--mode` flag lets you benchmark each
retrieval strategy independently against the golden set.

The golden set is to be placed at `data/golden_set.jsonl`. Example:

```jsonl
{"query": "Collibra community setup and configuration", "expected_files": ["community/setup.md", "community/configuration.md"], "expected_publications": ["markdown"]}
{"query": "asset mapping and metadata management", "expected_files": ["api/guides/asset-mapping.md", "api/guides/metadata.md"], "expected_publications": ["markdown"]}
{"query": "governance roles and access control", "expected_files": ["governance/roles.md", "governance/access-control.md"], "expected_publications": ["markdown"]}
{"query": "API authentication and tokens", "expected_files": ["api/authentication.md", "api/tokens.md"], "expected_publications": ["markdown"]}
{"query": "data lineage and impact analysis", "expected_files": ["governance/lineage.md", "governance/impact-analysis.md"], "expected_publications": ["markdown"]}
```

To add or customise queries, edit the file and re-run.

The evaluator source is at `atlas/evaluate.py`.

---

## Contributing

See the [development section](#10-development) for setup instructions. PRs
and issues welcome. The test suite is `uv run pytest tests/ -v`.

## License

Apache License 2.0 — see [LICENSE](LICENSE) file. The Collibra documentation
content is governed by the Collibra terms of service.