Collibra Atlas

Local-first AI knowledge layer for Collibra documentation via MCP.

Atlas is a local-first knowledge layer for Collibra development. Two Model Context Protocol servers expose the entire Collibra developer documentation to any AI agent:

• collibra-fs — deterministic filesystem server backed by ripgrep. Tools: list publications, list files, read file, full-text search, get release info. No model, no embeddings, no state. ~zero startup.

• collibra-rag — semantic search over precomputed embeddings plus fielded BM25 keyword search with title/heading/file-path weighting. Tools: search_docs, search_code, get_chunk, get_bundle_info. Three modes: vector (dense), keyword (fielded BM25), hybrid (score-level fusion). Loads a portable bundle once at startup (~1 GB); answers queries in ~1-2 ms (MLX on Apple Silicon) or ~50-200 ms (ONNX+CPU).

The RAG bundle is built once by the maintainer locally, then distributed as a single download via atlas-download. End users never embed, never chunk, never run a vector database, never pull a model.

Table of contents

1. Why two servers and not one?

2. Tech stack & rationale

3. Project structure

4. For users: install and use

5. For maintainers: building and releasing

6. Operating

7. Roadmap

8. Platform support & caveats

9. Troubleshooting

10. Development

11. Validate RAG quality

12. Contributing

13. License

Related MCP server: okfy

1. Why two servers and not one?

A single RAG pipeline forces every consumer into a fixed retrieval strategy and a fixed embedding model. Splitting into a filesystem server and a RAG server buys:

• Different strengths. The filesystem server is unbeatable for "list the publications," "read this specific file," and "grep for the exact symbol." The RAG server is unbeatable for "find docs that talk about X" where the wording is fuzzy.

• Different trust levels. The filesystem server returns the verbatim markdown — no embedding can ever lie about it. The RAG server returns ranked candidates; the model should still verify with the filesystem server for anything load-bearing.

• Different cost profiles. The filesystem server is ~zero startup. The RAG server loads ~1 GB of vectors at startup but answers in ~100 ms. A smart client uses both: RAG to discover, FS to verify.

• Different model fits. A weak local model struggles with multi-hop FS navigation but does fine with RAG top-k. A strong external model can do either, and benefits from using both.

2. Tech stack & rationale

3. Project structure

collibra-atlas/
├── README.md                                 This file
├── pyproject.toml                            uv-managed deps, console-script entry points
├── .gitignore
│
├── atlas/                                    Core Atlas package (importable as `atlas`)
│   ├── __init__.py                           Version + package docstring
│   ├── chunk.py                              H2-boundary markdown chunker (with --clean-gitbook)
│   ├── log.py                                Structured logging via structlog
│   ├── fs_server.py                          Filesystem MCP server (ripgrep-backed)
│   ├── rag_server.py                         RAG MCP server (auto-selects backend)
│   ├── make_bundle.py                        Bundle builder (chunk -> embed -> write)
│   ├── download.py                           Bundle downloader + SHA256 verification
│   ├── backup.py                             Snapshot current bundle
│   ├── restore.py                            Roll back to a previous snapshot
│   ├── doctor.py                             Installation diagnosis + backend probe
│   ├── bm25_search.py                        BM25 keyword index (build-time, optional)
│   ├── smoke_test.py                         E2E validation (build + search)
│   ├── evaluate.py                           RAG quality (Precision@10, MRR)
│   ├── rerank.py                             Cross-encoder re-ranker (MiniLM-L6-v2 ONNX, portable)
│   ├── rerank_mlx.py                         Cross-encoder re-ranker (bge-reranker-v2-base MLX, Apple Silicon)
│   ├── embed/                                Embedding backends (factory pattern)
│   │   ├── __init__.py
│   │   ├── base.py                           ABC + factory + resolve_backend
│   │   ├── onnx.py                           OnnxEmbedder (portable, CPU/CUDA)
│   │   └── mlx.py                            MlxEmbedder (Apple Silicon)
│   └── sources/                              Source adapters for markdown input
│       ├── __init__.py
│       ├── base.py                           MarkdownSource ABC
│       ├── git.py                            Git repo source
│       ├── web_crawl.py                      Web crawl mirror source
│       └── local.py                          Local directory source
│
├── collibra_docs_nav/                        Collibra-specific doc crawler
│   ├── __init__.py
│   └── crawler.py                            Async llms.txt crawler with GitBook syntax cleaning
│
├── scripts/
│   └── publish-bundle.sh                     Local build + gh release create
│
├── tools/
│   ├── convert_bge_to_mlx.py                 Convert BGE embedder weights to MLX .npy
│   └── convert_reranker_to_mlx.py            Convert BGE reranker weights to MLX .npy
│
├── tests/
│   ├── __init__.py
│   ├── test_chunk.py                         Chunker tests
│   ├── test_embed.py                         Embedding backend tests
│   ├── test_bm25_search.py                   BM25 keyword index tests
│   ├── test_fs_server.py                     FS MCP server tests
│   ├── test_rag_server.py                    RAG MCP server tests
│   ├── test_sources.py                       Source adapter tests
│   ├── test_backup_restore.py                Backup/restore tests
│   ├── test_doctor.py                        Doctor diagnosis tests
│   ├── test_download.py                      Bundle download tests
│   └── test_rerank_mlx.py                    MLX reranker tests (skipped on non-MLX)
│
├── data/                                     Runtime data (gitignored, created at runtime)

4. For users: install and use

4.1 Prerequisites

• Python 3.11+ (3.12 or 3.13 recommended).

• uv for Python environment management.

• ripgrep (brew install ripgrep or apt install ripgrep).

• ~1.5 GB of free disk for the bundle.

Platforms. Atlas runs on three classes of host, each picking a different embedding backend by default:

• Apple Silicon (M1/M2/M3/M4). Default backend is MLX, which talks to the Apple Neural Engine directly. ~1-2 ms per query. Install with uv sync --extra mlx.

• Linux x86_64 with an NVIDIA GPU. Default backend is ONNX Runtime + CUDA. ~1-2 ms per query. Install with uv sync --extra gpu.

• Anything else falls back to ONNX+CPU. ~50-200 ms per query.

4.2 Install

git clone <this-repo-url> collibra-atlas
cd collibra-atlas
uv sync

Optional extras (pick what your machine can use):

# Apple Silicon: MLX embedder
uv sync --extra mlx

# Linux with NVIDIA GPU: ONNX Runtime + CUDA
uv sync --extra gpu

# Both
uv sync --extra mlx --extra gpu

Re-applying extras on subsequent uv sync. uv sync synchronizes the venv to match the lockfile plus whichever extras you specify on the command line. If you later run plain uv sync, the MLX/GPU packages will be silently removed. Keep them by re-running with the same flags: uv sync --extra mlx. After a sync, atlas-doctor will tell you immediately if a previously-installed backend is now MISS.

4.3 Download the pre-built bundle

uv run atlas-download \
  --repo <owner>/collibra-atlas-bundles \
  --output ./data/collibra-rag-bundle

This downloads, verifies SHA256, and extracts the bundle. If a previous bundle exists, it's snapshotted into ./data/collibra-rag-bundle/.backups/ first (most recent 5 kept by default).

Pin to a specific release:

uv run atlas-download \
  --repo <owner>/collibra-atlas-bundles \
  --tag v2026.07 \
  --output ./data/collibra-rag-bundle

4.4 Get the docs mirror (for filesystem server)

Download the pre-crawled mirror from GitHub Releases:

uv run atlas-download \
  --repo <owner>/collibra-docs-mirror \
  --output ./data/collibra-docs

Or crawl it yourself (see 5.1 Crawl Collibra documentation).

4.5 Configure your IDE

Both servers speak MCP over stdio, so any client that supports the standard works: Zed, opencode, Claude Desktop, Continue, and others.

Zed (~/.config/zed/settings.json)

"context_servers": {
  "collibra-fs": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
    ],
    "timeout": 60
  },
  "collibra-rag": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
    ],
    "timeout": 120
  }
}

Search mode is configured per-query via the mode parameter (vector, keyword, or hybrid) on each search_docs/search_code call. See Search Modes.

opencode (~/.config/opencode/profiles/default.json)

"mcpServers": {
  "collibra-fs": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
    ]
  },
  "collibra-rag": {
    "command": "uv",
    "args": [
      "run", "--directory", "/absolute/path/to/collibra-atlas",
      "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
    ]
  }
}

Claude Desktop (~/.config/Claude/claude_desktop_config.json)

{
  "mcpServers": {
    "collibra-fs": {
      "command": "uv",
      "args": [
        "run", "--directory", "/absolute/path/to/collibra-atlas",
        "collibra-fs", "--source-type", "web-crawl", "--mirror-path", "/absolute/path/to/data/collibra-docs"
      ]
    },
    "collibra-rag": {
      "command": "uv",
      "args": [
        "run", "--directory", "/absolute/path/to/collibra-atlas",
        "collibra-rag", "--bundle", "/absolute/path/to/data/collibra-rag-bundle", "--prefer", "auto"
      ]
    }
  }
}

4.6 Quick CLI sanity check

# Confirm both servers start
uv run collibra-fs --help
uv run collibra-rag --help

# Run diagnosis
uv run atlas-doctor

# Run smoke test
uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs

5. For maintainers: building and releasing

5.0 First-time build walkthrough

A guided path for building the bundle on your own machine, from a fresh uv sync to a working bundle.

Preflight. Confirm the embedding backend you expect is available:

uv sync --extra mlx        # or --extra gpu on Linux + NVIDIA
uv run atlas-doctor        # should report: MLX OK, selected: mlx

Convert MLX weights (2 minutes, one-time). The MLX backend needs pre-converted weight files. This uses PyTorch temporarily to read the Hugging Face checkpoint and write 197 .npy files to the MLX cache.

uv pip install torch
uv run python tools/convert_bge_to_mlx.py
uv pip uninstall torch --yes

The tokenizer (transformers.AutoTokenizer) works without PyTorch — the warning about "PyTorch was not found" is harmless. PyTorch is only needed for this one-time weight conversion.

Smoke crawl (2 minutes). Test the crawler with a subset:

uv run python -m collibra_docs_nav.crawler \
  --output ./data/collibra-docs \
  --max-pages 20

Smoke build (2 minutes). Test the pipeline with a tiny bundle:

uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output /tmp/atlas-smoke \
  --limit 100 \
  --prefer apple \
  --clean-gitbook

Sanity-check the smoke bundle:

.venv/bin/python -c "
from atlas.rag_server import Bundle
b = Bundle('/tmp/atlas-smoke', prefer='apple')
hits = b.search('Collibra community', top_k=3)
for h in hits:
    print(f'  {h[\"score\"]:.3f}  {h[\"file\"]} :: {h[\"heading\"]}')
"

Using .venv/bin/python directly avoids the ~300 ms uv run overhead. For pure JSON inspection of the manifest:

jq . /tmp/atlas-smoke/manifest.json | head -20

Full crawl (~10 minutes). Collibra publishes a single flat llms.txt listing all 673 pages — no sub-indexes to discover:

uv run python -m collibra_docs_nav.crawler \
  --output ./data/collibra-docs

Full build (2-4 hours depending on hardware):

uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output ./data/collibra-rag-bundle \
  --prefer auto \
  --clean-gitbook

5.1 Crawl Collibra documentation

The crawler fetches every .md page listed in the flat llms.txt index at https://developer.collibra.com/llms.txt. Collibra publishes a single flat index listing all 673 pages.

# Full crawl (~673 pages, ~10 min)
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs

# Test with subset
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --max-pages 50

# Crawl specific sections (available: api, cli, misc, tutorials, workflows)
uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --sections api,tutorials

The crawler saves files to markdown/<section>/<path>.md and writes a crawl_meta.json with source URL, timestamp, and per-category counts (pages found, fetched, skipped, errors, cleaned, total bytes).

GitBook syntax cleaning is enabled by default. Collibra docs use GitBook extensions (tabs, columns, hints, icons, HTML tables, <mark> tags) that the crawler strips before saving. To skip cleaning:

uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --no-clean

Useful flags:

• --sections api,tutorials — crawl only specific sections (available: api, cli, misc, tutorials, workflows)

• --max-pages 50 — limit total pages fetched

• --delay-min 0.5 --delay-max 2.0 — control jitter range (default: 0.3-1.5s)

• --max-concurrent 5 — max concurrent requests (default: 5)

• --no-clean — skip GitBook syntax cleaning

After crawling, run the validation script to check for residual syntax:

uv run python .agents/skills/collibra-docs-nav/scripts/validate_crawl.py ./data/collibra-docs

This checks that headers are stripped, frontmatter fields are present, no GitBook syntax remains, no uncleaned HTML is in regular markdown content (<code> is excluded — it's valid inline formatting), and files are in the correct directory structure. It intentionally skips HTML inside fenced code blocks (e.g. <code> in OpenAPI JSON specs) to avoid false positives — see the collibra-docs-nav skill for details.

5.1.5 MLX weight conversion (Apple Silicon)

The MLX backend requires pre-converted weights from the PyTorch checkpoint. This is a one-time step per machine — once the .npy files are in the cache, every build and query reuses them.

# Install torch temporarily for the conversion
uv pip install torch

# Run the conversion (writes to ~/.cache/atlas/models/bge-base-en-v1.5-mlx/)
uv run python tools/convert_bge_to_mlx.py

# Uninstall torch — not needed at runtime
uv pip uninstall torch --yes

The conversion reads BAAI/bge-base-en-v1.5 (the PyTorch model) and writes 197 .npy weight files. The MLX embedder then loads them at ~/.cache/atlas/models/bge-base-en-v1.5-mlx/ and never touches PyTorch again.

After conversion, atlas-doctor reports MLX OK, selected: mlx and atlas-build --prefer apple will use the ANE/GPU accelerator.

Note: If the MLX cache is empty, atlas-build --prefer apple falls back gracefully to ONNX+CPU with a reminder to run the conversion script. ONNX+CPU is ~3-5x slower but works without any setup.

5.2 Build RAG bundle

The build pipeline runs: crawl -> chunk (with overlap + cluster_tags) -> embed -> fielded BM25 index (text + title + heading + file_path) -> write artifacts -> stage model -> manifest.

# Full build
uv run atlas-build \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --output ./data/collibra-rag-bundle \
    --prefer auto \
    --clean-gitbook

# Test build (skip embeddings, limit files)
uv run atlas-build \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --output ./data/test-bundle \
    --limit 100 \
    --skip-embed \
    --clean-gitbook

The --clean-gitbook flag strips GitBook-specific syntax (tabs, columns, hint blocks, icon references, HTML tables, <mark> tags) from each markdown file before chunking. This is required for Collibra docs to produce clean chunks.

The build writes a manifest.json with the source SHA, chunk count, BM25 index size, model ID, and SHA256 of each artifact. A fielded BM25 index is built automatically from chunk texts, titles, headings, and file paths with per-field weights (text=1.0, title=3.0, heading=2.0, file_path=2.5) via rank-bm25::BM25Okapi and saved as bm25.pkl.

5.3 Validate

# Run smoke tests
uv run atlas-smoke \
    --bundle ./data/collibra-rag-bundle \
    --source-type web-crawl \
    --mirror-path ./data/collibra-docs \
    --verbose

# Diagnose installation
uv run atlas-doctor --bundle ./data/collibra-rag-bundle

5.4 Publishing a release

The bundle is built locally — embedding ~100k chunks on a CI runner is impractical. On Apple Silicon with MLX the build takes ~30 min; on a CPU-only machine it takes longer but still finishes.

# Prerequisites: gh CLI installed and authenticated
#   brew install gh && gh auth login

# Publish the existing bundle (fast, no rebuild)
./scripts/publish-bundle.sh <owner>/collibra-atlas-bundles

# Or rebuild first (takes ~30 min on Apple Silicon)
./scripts/publish-bundle.sh --rebuild <owner>/collibra-atlas-bundles

This tars the bundle, creates a GitHub Release tagged vYYYY.MM, and uploads it. Users can then install it with atlas-download.

The script lives at scripts/publish-bundle.sh — read it to see exactly what it does before running.

5.4.5 MLX reranker weight conversion (Apple Silicon)

The MLX cross-encoder reranker (atlas/rerank_mlx.py) requires pre-converted weights from the Hugging Face checkpoint. This is a one-time step per machine, like the embedder MLX conversion.

# Install torch temporarily for the conversion
uv pip install torch

# Run the conversion (writes to ~/.cache/atlas/models/bge-reranker-v2-base-mlx/)
uv run python tools/convert_reranker_to_mlx.py

# Uninstall torch — not needed at runtime
uv pip uninstall torch --yes

The conversion reads BAAI/bge-reranker-v2-base (a BERT-base cross-encoder) and writes 199 .npy weight files (197 encoder weights + classifier head weight/bias). The MLX reranker loads them at ~/.cache/atlas/models/bge-reranker-v2-base-mlx/.

After conversion, the RAG server auto-selects MLX for reranking when --rerank is passed, falling back to ONNX if MLX weights aren't found.

5.5 Smoke-test the bundle

After building or downloading a bundle, validate it end-to-end:

uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs

The smoke test confirms the bundle loads, embeddings are queryable, and the filesystem server can serve files from the mirror.

Architecture

flowchart LR
    subgraph Source["Docs Source"]
        URL[("developer.collibra.com<br/>llms.txt (flat index)")]
    end

    subgraph Crawl["Crawler"]
        CRAWLER["collibra-crawl<br/>aiohttp"]
        CLEAN["GitBook syntax<br/>cleaner"]
        JITTER["jitter + rotating UAs<br/>+ polite headers"]
    end

    subgraph Mirror["Local Mirror"]
        MD["markdown/<br/>section/*.md"]
        META["crawl_meta.json"]
    end

    subgraph Build["Bundle Builder"]
        CHUNK["atlas-build<br/>chunk --clean-gitbook<br/>embed -- write -- bm25"]
        MODEL[("bge-base-en-v1.5<br/>embedding model")]
        BM25_BUILD[("rank-bm25<br/>Fielded BM25<br/>text+title+heading+file_path")]
    end

    subgraph Bundle["RAG Bundle"]
        CHUNKS["chunks.parquet"]
        VECTORS["embeddings.f16.npy"]
        NORMS["norms.f32.npy"]
        BM25_PKL["bm25.pkl"]
        MANIFEST["manifest.json"]
    end

    subgraph Publish["Distribution"]
        RELEASE["GitHub Release<br/>tar.zst"]
        DOWNLOAD["atlas-download<br/>SHA256 verify"]
    end

    subgraph MCP["MCP Servers"]
        FS["collibra-fs<br/>ripgrep-backed"]
        RAG["collibra-rag<br/>vector / keyword / hybrid"]
    end

    subgraph Clients["MCP Clients"]
        ZED["Zed"]
        OPENCODE["opencode"]
        CLAUDE["Claude Desktop"]
        CONTINUE["Continue"]
    end

    URL -->|"flat llms.txt<br/>673 pages"| CRAWLER
    CRAWLER --> CLEAN
    CLEAN --> JITTER
    JITTER -->|"polite fetch"| URL
    CRAWLER -->|"write"| MD
    CRAWLER --> META

    MD -->|"read"| CHUNK
    MODEL -->|"embed"| CHUNK
    BM25_BUILD -->|"index"| CHUNK
    CHUNK -->|"write"| CHUNKS
    CHUNK --> VECTORS
    CHUNK --> NORMS
    CHUNK --> BM25_PKL
    CHUNK --> MANIFEST

    Bundle -->|"tar + gh release"| RELEASE
    RELEASE -->|"download + verify"| DOWNLOAD
    DOWNLOAD -->|"extract to"| Bundle

    MD -->|"read live"| FS
    Bundle -->|"load at startup"| RAG

    Clients -->|"MCP stdio"| FS
    Clients -->|"MCP stdio"| RAG

6. Operating

6.1 Update to the latest bundle

uv run atlas-download --repo <owner>/collibra-atlas-bundles --output ./data/collibra-rag-bundle

The current bundle is auto-snapshotted before being replaced. If the new bundle is broken, the IDE still works against the old directory while you sort it out.

6.2 List and roll back

uv run atlas-restore --snapshot ./data/collibra-rag-bundle/.backups/snapshot-20260701T120000Z.tar.gz --target ./data/collibra-rag-bundle

The backup directory preserves recent snapshots for rollback.

6.3 Manual snapshot

uv run atlas-backup --bundle ./data/collibra-rag-bundle --backup-dir ./data/collibra-rag-bundle/.backups

6.4 Diagnose with atlas-doctor

uv run atlas-doctor
uv run atlas-doctor --bundle ./data/collibra-rag-bundle
uv run atlas-doctor --refresh    # re-probe backends instead of using cache
uv run atlas-doctor --json       # print raw JSON instead of formatted report

Output (short form):

Platform       : Darwin 24.5.0 (arm64)
Python         : 3.13.5
ONNX Runtime   : 1.26.0  providers: ['CoreMLExecutionProvider', 'CPUExecutionProvider']

Backend probe:
  ONNX+CPU       OK  always available
  Apple MLX      OK   weights cached
  NVIDIA CUDA    MISS nvidia-smi not on PATH

Selected       : mlx
                Apple Silicon detected and MLX is importable

Bundle         : /Users/me/data/collibra-rag-bundle
                manifest=OK chunks=OK embeddings=OK
                52 chunks, model=Xenova/bge-base-en-v1.5 (SHA ok)

atlas-doctor is non-destructive and never blocks startup. Run it first when something is wrong.

Commands Reference

Backend Selection

The backend is resolved at run time (not build time). The same bundle works everywhere. Use atlas-doctor to see which backend will be selected.

Override examples:

# Always MLX, even on a machine that would default to CUDA
uv run collibra-rag --prefer apple

# Force the portable floor (useful for debugging)
ATLAS_EMBED_BACKEND=cpu uv run collibra-rag

Search Modes

The collibra-rag server supports three search modes, selected via the mode parameter on each search_docs/search_code call:

The mode is per-query, so a client can use keyword for an API endpoint lookup (output-module), vector for fuzzy concept search ("how do I set up asset mapping"), and hybrid (the default) when coverage matters most.

The BM25 index is built at bundle-build time. If a bundle lacks bm25.pkl (older bundles), keyword and hybrid fall back to a simple title-boost heuristic automatically — no crash, no error.

Hybrid mode additionally applies heading-aware AdaGReS diversity ranking to prevent a single file from dominating the top results. Chunks from different files are preferred; multiple chunks from the same file but different sections are partially penalised; truly redundant chunks (same file + same heading) are blocked. This is query-time only — no rebuild needed.

BM25 also benefits from request-side query expansion: morphological variants ("asset" <-> "assets", "create" <-> "creating") are generated automatically at query time without stemming the corpus. Identifiers pass through unchanged.

Try the difference. Point any of these at your built bundle:

# Hybrid — score-level fusion of vector + BM25 (best overall recall)
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('asset mapping configuration', top_k=3, mode='hybrid'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"

# Vector — pure dense embedding similarity
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('asset mapping configuration', top_k=3, mode='vector'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"

# Keyword — fielded BM25 (title/heading/file-path boosted, best for exact names)
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle')
for r in b.search('output-module', top_k=3, mode='keyword'):
    print(f'  {r[\"score\"]:.3f}  {r[\"file\"]} :: {r[\"heading\"]}')
"

Retrieval Enhancements

Chunk Overlap

Adjacent H2 sections share a small text overlap (150 characters from the tail of the previous section) so that queries straddling a section boundary still match. The overlap is prepended without a special marker — the embedding model treats it as natural context.

• Controlled via _OVERLAP_CHARS in atlas/chunk.py (default 150).

• Zero runtime cost — embedded at build time.

• Can cause adjacent sections to inherit code-fence flags (is_code), which is correct: a section that begins with a code snippet from overlap is relevant for search_code.

Hierarchical File-System Chunking (Path-Aware Boost)

At build time, each chunk is tagged with cluster_tags — the space-joined stems of sibling .md files in the same directory. At query time, the RAG server applies a small relevance boost (0.03 weight) to chunks whose cluster_tags match query tokens.

This is a lightweight form of path-aware retrieval that surfaces related docs from the same file-system "neighbourhood" without re-embedding.

• Build-time: sibling map computed in make_bundle.py::build_chunk_table().

• Query-time: pc.match_substring on cluster_tags in Bundle.search().

• Graceful fallback: old bundles without the column skip the boost.

Cross-Encoder Reranker (opt-in)

The --rerank flag enables a cross-encoder re-ranker that re-scores the top-100 candidates from hybrid search using a joint query-passage model.

• MLX backend (Apple Silicon): BAAI/bge-reranker-v2-base — 110M params, BERT-base encoder + classification head. Weights converted via tools/convert_reranker_to_mlx.py.

• ONNX backend (portable): cross-encoder/ms-marco-MiniLM-L6-v2 — 22.7M params, auto-downloaded from Hugging Face.

• Auto-selects MLX if available, falls back to ONNX.

• Adds ~5 ms/query (MLX) or ~50-200 ms (ONNX+CPU).

Fielded BM25 with File-Path Signals

The keyword search index is built as a fielded BM25 that scores matches in the body text, title, heading, and file path independently, then combines them as a weighted sum:

BM25 parameters are tuned for long reference docs: k1=2.0 (higher TF saturation ceiling) and b=0.7 (reduced length normalisation penalty).

The file-path field works because Collibra's .md filenames are descriptive identifiers — a query for "output module" directly matches the path tokens of api/guides/output-module.md. The .md extension is stripped before tokenization so only meaningful tokens remain.

• Build-time: atlas/make_bundle.py extracts the file column and passes it as the file_path field to build_fielded_index().

• Query-time: atlas/bm25_search.py::FieldedBM25Index computes per-field scores and combines them — rag_server.py doesn't need to know about fields.

• Persisted: Fields and weights are baked into bm25.pkl (version 2 format). Version 1 pickles still load via backward compatibility.

• Cost: Zero query-time LLM cost. Build-time overhead ~1-2 s for 250k chunks. Pickle increases ~100 KB for 50k chunks with 4 fields.

Request-side query expansion (zero build cost): BM25 automatically generates morphological variants at query time — "asset" also matches "assets" and "assetting"; "create" also matches "creating" and "created". The corpus is never modified or stemmed. Non-matching generated variants (e.g. "assett") contribute zero score. ~0.01ms per query.

Configuration

Create ~/.config/atlas.toml to set default backend:

[backend]
prefer = "auto"  # or "apple", "nvidia", "cpu"

Or use environment variable:

export ATLAS_EMBED_BACKEND=apple

7. Roadmap

Reasoning agent (atlas/agent.py, planned)

What it should eventually be:

• A thin local agent that wraps the two MCP servers for command-line use (atlas-agent "find docs about Collibra community") for users who don't have an IDE with MCP support.

• Pre-built tool-use prompt templates tuned for Collibra tasks (community setup, asset mapping, governance configuration).

• A planner that fans a question out to collibra-fs + collibra-rag in parallel and merges the results.

Fine-tuning pipeline (atlas/training.py, planned)

• Dataset curation from the same RAG bundle used for retrieval.

• QLoRA adapters for local models, with evaluation against MCP tool-calling contracts.

8. Platform support & caveats

Supported

• Apple Silicon (M1/M2/M3/M4), all macOS versions with current security updates. MLX is the default backend on this hardware (uv sync --extra mlx to install).

• Linux x86_64 + CPU. The portable ONNX+CPU floor works everywhere.

• Linux x86_64 + NVIDIA GPU. ONNX+CUDA via uv sync --extra gpu.

Not supported

• Intel Macs. Falls back to ONNX+CPU. No MLX acceleration.

• Windows. mcp and onnxruntime work on Windows, but rg and tar paths are untested.

Known limitations

• The embedder's max_seq_length is 512 tokens. Chunks larger than that are silently truncated. The H2 chunker rarely produces such chunks, but the chunk._hard_split fallback handles them.

• Chunk overlap can cause adjacent sections to inherit code-fence flags (is_code = True) when the overlap text carries a code fence from the previous section. This is correct behaviour: sections near code are relevant for search_code.

• Cosine scores are not calibrated. A score of 0.7 is only meaningful relative to other scores from the same query. Use min_score loosely.

• The bundle targets a single snapshot of Collibra docs. Re-build periodically to stay current.

• Collibra docs use slug URLs (/pages/tg5mBbcaVwKS4wowOuGp) that map to flat markdown/pages/*.md files. The crawler and chunker handle these, but the file paths in search results may look opaque compared to the human-readable paths of other documentation sites.

9. Troubleshooting

No 'markdown/' directory at ...

The filesystem server was started with --mirror-path pointing at the wrong place. The expected layout is <mirror>/markdown/<section>/*.md. If you haven't crawled yet:

uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs

Bundle manifest missing

Either --bundle points at the wrong directory, or the bundle wasn't extracted cleanly. Re-run atlas-download and let it overwrite.

ripgrep (rg) not installed

brew install ripgrep        # macOS
apt install ripgrep         # Debian/Ubuntu

Context server requires timeout in Zed/opencode

The first RAG query after server startup pays a one-time warmup cost: ~3-5 s for ONNX+CPU, ~1 s for MLX. Raise the MCP timeout to 120 or 300 seconds to absorb the warmup.

atlas-doctor says MLX is MISS but I have an M-series

You didn't install the optional MLX extra. Run:

uv sync --extra mlx

After that, atlas-doctor will find MLX and it becomes the default.

Search returns nothing useful

1. Run get_bundle_info to confirm the bundle is loaded.

2. Try search_docs with a broader query.

3. Use collibra-fs to grep for exact terms.

4. Drop min_score to 0.0 to see all candidates.

Crawler gets rate-limited

The crawler uses polite defaults (rotating User-Agents, jittered delays). If you're still getting rate-limited, increase the jitter range:

uv run python -m collibra_docs_nav.crawler --output ./data/collibra-docs --delay-min 1.0 --delay-max 3.0

GitBook syntax leaking through in search results

If you see raw GitBook syntax (tabs, columns, icon references) in search results, the bundle was built without --clean-gitbook. Rebuild:

uv run atlas-build \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs \
  --output ./data/collibra-rag-bundle \
  --clean-gitbook

HTML tags visible in validation output

The validate_crawl.py script may report HTML tags (like <code> or <table>) in API reference files. These are false positives when they appear inside fenced code blocks — the validator intentionally skips code fence content. Common cases:

• OpenAPI JSON specs contain <code>null</code> in description fields. These are formatting markers inside JSON, not markdown HTML.

• SQL templates reference "<table>" as a table name placeholder. These are not HTML table tags.

The validator's HTML check only flags tags in regular markdown content, not inside ``` fences. If you see unexpected warnings, check whether the flagged content is inside a code block.

Console scripts not found after uv sync

If uv run collibra-fs --help returns "command not found," the package isn't installed in the venv. Run uv sync (no flags) to install the project itself, not just its dependencies.

10. Development

Useful shell recipes

# Pretty-print the bundle manifest (no `cat`, no `uv run` overhead)
jq . data/collibra-rag-bundle/manifest.json

# If you don't have jq, fall back to stdlib
.venv/bin/python -m json.tool data/collibra-rag-bundle/manifest.json | head -20

# Load a bundle in Python without going through `uv run`
.venv/bin/python -c "
from atlas.rag_server import Bundle
b = Bundle('data/collibra-rag-bundle', prefer='apple')
for h in b.search('Collibra community', top_k=3):
    print(f'{h[\"score\"]:.3f}  {h[\"file\"]}')
"

Install jq once (brew install jq on macOS, apt install jq on Linux) and the first command is the one you'll reach for every time.

Lint

ruff enforces style and docstring conventions (pydocstyle rules are enabled via "D" in pyproject.toml):

uv run ruff check .
# Auto-fix trivial issues (blank lines, punctuation):
uv run ruff check --fix .

Run unit tests

uv run pytest tests/ -v

Runs the full test suite (chunker, embedder, MCP servers, bundle build, BM25 keyword index).

Run the smoke test

uv run atlas-smoke \
  --bundle ./data/collibra-rag-bundle \
  --source-type web-crawl \
  --mirror-path ./data/collibra-docs

Add a new tool

1. Add a Tool(...) entry in atlas/fs_server.py or atlas/rag_server.py.

2. Add a handler branch in call_tool.

3. Keep the handler synchronous (it's already running in the async event loop; asyncio.to_thread is fine for blocking I/O).

Change the embedder

Edit embed.DEFAULT_MODEL_ID and embed.EMBEDDING_DIM. The bundle build and the runtime server read both. Changing embedding models invalidates all existing bundles.

Change the chunker

Edit atlas/chunk.py. The schema is documented in the module docstring. Run atlas-smoke after changes. Key parameters:

• _OVERLAP_CHARS (150) — tail chars carried between adjacent H2 sections

• _MAX_CHUNK_CHARS (8000) — hard-split threshold for oversized sections

Add a reranker

Two paths exist:

• MLX (Apple Silicon): Add a model class in atlas/rerank_mlx.py and a converter in tools/convert_reranker_to_mlx.py. The BERT-base architecture is already supported via atlas/embed/mlx.BgeModel.

• ONNX (portable): Export a Hugging Face cross-encoder to ONNX and add a CrossEncoderReranker subclass in atlas/rerank.py. Registration is automatic via the try/except in rag_server.py.

11. Validate RAG quality

The project ships a golden set of queries covering the major Collibra documentation areas (community setup, asset mapping, governance, API guides, etc.). To evaluate retrieval quality:

# Vector search (cosine similarity on dense embeddings)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode vector

# Keyword search (fielded BM25 with query expansion)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode keyword

# Hybrid search (vector + BM25 fusion with AdaGReS diversity)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode hybrid

# With cross-encoder re-ranker (more accurate, slower)
uv run atlas-evaluate \
  --bundle ./data/collibra-rag-bundle \
  --golden ./data/golden_set.jsonl \
  --prefer apple \
  --top-k 5 \
  --mode hybrid \
  --rerank

# Compare all three modes on a specific query:
uv run python -c "
from atlas.rag_server import Bundle
b = Bundle('./data/collibra-rag-bundle', prefer='apple')
for mode in ('keyword', 'vector', 'hybrid'):
    print(f'--- {mode} ---')
    for r in b.search('asset mapping configuration', top_k=5, mode=mode):
        print(f'  {r[\"score\"]:.3f}  {r[\"file\"]}')
"

This measures Precision@k (fraction of top-k results matching expected files) and Mean Reciprocal Rank (inverse rank of first relevant result) over the full golden set.

Add --rerank to use the cross-encoder re-ranker (slower but more accurate — re-scores top-100 candidates with a joint query-passage model). Uses MLX on Apple Silicon, ONNX elsewhere. Add --output results.json to save results to a file. The --mode flag lets you benchmark each retrieval strategy independently against the golden set.

The golden set is to be placed at data/golden_set.jsonl. Example:

{"query": "Collibra community setup and configuration", "expected_files": ["community/setup.md", "community/configuration.md"], "expected_publications": ["markdown"]}
{"query": "asset mapping and metadata management", "expected_files": ["api/guides/asset-mapping.md", "api/guides/metadata.md"], "expected_publications": ["markdown"]}
{"query": "governance roles and access control", "expected_files": ["governance/roles.md", "governance/access-control.md"], "expected_publications": ["markdown"]}
{"query": "API authentication and tokens", "expected_files": ["api/authentication.md", "api/tokens.md"], "expected_publications": ["markdown"]}
{"query": "data lineage and impact analysis", "expected_files": ["governance/lineage.md", "governance/impact-analysis.md"], "expected_publications": ["markdown"]}

To add or customise queries, edit the file and re-run.

The evaluator source is at atlas/evaluate.py.

Contributing

See the development section for setup instructions. PRs and issues welcome. The test suite is uv run pytest tests/ -v.

License

Apache License 2.0 — see LICENSE file. The Collibra documentation content is governed by the Collibra terms of service.