# Mosaic

[![CI](https://github.com/alexxb2mg-svg/Mosaic/actions/workflows/ci.yml/badge.svg)](https://github.com/alexxb2mg-svg/Mosaic/actions/workflows/ci.yml)

**Semantic search over your own documents, built in the open from textbook parts —
local, deterministic, no LLM in the loop. As much a working lesson as a tool.**

```bash
pip install -e ".[dev,ingest]"
mosaic build ./my_documents -o ./index_docs
mosaic search "how do I wire the differential breaker" ./index_docs --top 5
# [{"id": "tableau_garage.md", "score": 0.62}, ...]   JSON out, ~50 ms warm
```

Python ≥ 3.12, numpy as the only core dependency, no GPU, no network at runtime.
[Full quickstart](#quickstart) · [measured performance](#measured-performance) ·
[architecture](docs/ARCHITECTURE.md) · [campaign log](docs/MEASURES.md).

Mosaic is one practitioner's answer to one practitioner's constraints: a small
business's documents made searchable by *meaning*, with no cloud, no GPU, no per-query
bill, and results that can be trusted because they can be reproduced. It does not
claim to reinvent retrieval. The mechanisms are textbook — co-occurrence statistics in
the LSA lineage, hyperdimensional computing, BM25, self-organizing maps, rank fusion —
and where a standard tool wins, the benches below say so in print.

What the name promises is the only originality claimed: an *assembly*. Not one clever
algorithm but several complementary retrieval processes — a semantic grid learned from
your corpus, exact lexical postings, static embeddings, a learned semantic atlas, a
deterministic grammatical channel — each one measured, kept only where it wins, fused
only where fusion wins, and
calibrated on *your* corpus rather than on doctrine. The same corpus produces the same
index and the same ranking, on any machine, and no data ever leaves yours.

Take it accordingly: first as a **didactic experiment** — an engine assembled from
measurable parts, every design decision settled by a bench you can rerun, the dead
ends documented next to the wins. It does run in production here, and if your
constraints resemble the author's — local-first, deterministic, French-heavy corpora,
agents in the loop — it may serve you as a tool too. Contributions from professionals
of the field are warmly welcome, and understood accordingly: corrections, measures,
stronger baselines, proofs that a design choice is wrong. They feed an educational
resource more than a product roadmap — beat a number in this README on its own bench,
and the result belongs here, whichever way it points.

It is built **agent-first** — compact JSON everywhere, an MCP server, usage guidance
embedded in tool descriptions: your agent sees five lines of JSON instead of thousands
of tokens of raw files — and **human-usable**: a plain CLI, explanations on demand
(`explain`, `--explique`), nothing you cannot run and read yourself.

## The idea

The engine does everything **mechanical**: index, retrieve, remember, and *measure its
own confidence* — deterministic, free, reproducible. An LLM (if you use one at all)
only enters at the **point of judgment**: never in the storage or search loop, only
when something must be *decided* — settling an ambiguity the engine itself has
flagged. The deterministic part carries the weight; intelligence only pays for the
irreducible.

And because no single retrieval method wins everywhere (we measured — see
[the campaign log](docs/MEASURES.md)), the engine does not pretend one does: you pick
the tiles of the mosaic per corpus, guided by measured evidence, and `mosaic calibrer`
tunes the weights on *your* ground truth.

## What it does

- **Semantic search** (`mosaic search`) — paraphrase-friendly retrieval over any
  folder (`.md`, `.txt`, `.pdf`, `.docx`, `.xlsx`, `.html`, `.pptx`, images via
  optional OCR). Query algebra opt-in (`--connecteurs`): *"A sans B"* pushes concept B
  down the ranking.
- **Typed grids** (`--grilles-typees`) — route meaning words, identifiers and path
  tokens to separate grids; identifier lookup stops drowning in prose (+7.5 pts on
  drowned references, measured on real product records).
- **Four-channel fusion** (`--hybride` + `--fusion`, `--atlas`) — grid + BM25 +
  embeddings + semantic atlas, fused by rank (RRF). Wins on lexical terrain; refused
  in degraded subsets, never a silent default.
- **Multi-channel identity card** — exact facets per document (*type*, *date*,
  *reference codes*): a code in your query triggers an exact-match boost; `--recence`
  blends rank with freshness; `--type` filters exactly.
- **Temporal truth** (`mosaic actuel`) — versions of the same aspect are grouped, the
  newest is canonical, older ones are flagged **stale**: an agent can no longer
  mistake outdated data for truth.
- **Deterministic grammatical channel** (`--grammatical`) — rule-based French role
  analysis (negation scope, upstream/downstream order, agent/patient) bound to token
  signatures: separates clauses with *identical words and opposite meaning* that
  bag-of-words provably confuses (25/34 planted pairs at cosine exactly 1.0 bare;
  33/34 separated with the channel; 2.1% role error with clean abstention — silence
  over guessing). Separate vector, zero impact unless asked.
- **Semantic corpus diff** (`mosaic diff`) — what changed in *meaning* between two
  states of a corpus: vocabulary born and dead, words whose context drifted, usage
  declines, and untouched documents whose meaning moved because the world around them
  did. Identical corpora yield a **strictly empty** diff — determinism makes that
  guarantee contractual, something re-embedding pipelines cannot offer.
- **Cross-index meta-search** (`mosaic meta`) — several indexes at once, rank-fused,
  provenance kept.
- **Graph traversal without a graph database** (`mosaic chemin`) — two vector-space
  hops answer *"the other documents of the same project / the same year"*.
- **Belief memory** (`mosaic croyance`) — agents assert facts; the newest wins,
  history is kept, a hypervector layer provides a confidence margin, and the
  "contested" threshold can be conformally calibrated on your own store (error rate of
  confident answers guaranteed ≤ α).
- **Declarative per-index profile** (`mosaic profil`) — your world in ten lines of
  JSON: folder-tree roles, custom document types, your trade's reference-code shape.
  `--suggere` proposes one from a corpus scan.
- **Measured calibration** (`mosaic calibrer`) — encoding weights picked by benchmark
  on ground-truth queries (or deterministically generated ones, `--verite-auto`),
  recommending a change only when the gain is clear.
- **MCP server** (`infra_mcp/`) — all of the above as 11 MCP tools, dynamic domain
  discovery, zero dependencies beyond the engine.
- **Alternative document converter** (opt-in, `MOSAIC_CONVERTISSEUR=anydoc`) — the
  default reader is markitdown; `firecrawl-anydoc` (pure Rust, no transitive
  dependency, no model) can replace it. Measured on 15 real supplier quotes: far more
  regular structure — one article per table row, and product names that markitdown cut
  in half across the layout come back **whole** — which cut unattachable fragments
  downstream by **7×**, at 31× the conversion speed. Deterministic (8 documents × 3
  conversions, identical output). Strictly opt-in and **recorded in the index
  metadata**: a different converter means different text, therefore different grids —
  two indexes read by different converters are not comparable, and `add()` refuses to
  mix them. Trade-off: anydoc *refuses* PDFs with no text layer (`OCR is required`)
  where markitdown returned partial text; the existing OCR hook takes over.

## Choose your setup — the measured terrain map

There is no universal winner — every result below was measured both ways, defeats
included (the full stories live in [docs/MEASURES.md](docs/MEASURES.md)). The same
fusion that wins outright on lexical terrain *loses* on a paraphrase-heavy corpus,
where it divides the semantic-trap MRR by three: a rank fusion averages opinions,
which converges on lexical terrain and dilutes the grid's semantic wins elsewhere.
Pick by terrain — and know the bill before you flip a flag:

```mermaid
flowchart TD
    Q{"What does your corpus look like?"}
    Q -->|"prose, paraphrase-heavy queries"| A["default grid + embeddings + rerank, no fusion — disk 48 KB/word + 12 KB/doc, RAM 370 MB @ 18k docs, 27-63 ms/query, build minutes (nightly)"]
    Q -->|"lexical Q&A: queries reuse the docs' vocabulary"| B["--hybride --atlas at build, --fusion at search — SERVING: same RAM as above, + a few MB postings + 4.5 KB/doc on disk, 29 ms/query. BUILD ONLY: the SOM adds minutes and a few GB of RAM at 70k+ words, then releases everything"]
    Q -->|"dense in identifiers (catalogs, part numbers)"| C["--grilles-typees + --rerank-vectors — often CHEAPER: meaning grid up to 4x smaller, identifier grids 3 KB/word, same latency"]
    Q -->|"code"| U["not measured yet: run the benches, tell us"]
    A --> T{"Folder evolves over time?"}
    B --> T
    C --> T
    T -->|"yes"| V["add mosaic actuel — free, reads stored facets"] --> M
    T -->|"no"| M{"Machine budget tight?"}
    M -->|"yes"| S["--grid 32x32: everything /4 (3 KB/doc, 12 KB/word) — --smoothing-rank 0: skip the SVD, faster build, lower recall — skip extras: pure numpy"] --> DONE
    M -->|"no"| DONE["build, then let measurement tune it: mosaic calibrer + your ground truth"]
```

| Your corpus looks like | Measured best setup | Evidence | What it costs |
|---|---|---|---|
| Prose documents, paraphrase-heavy queries (knowledge bases, notes, procedures) | default grid + embeddings + `--rerank` — **no fusion** | private real-corpus bench: solo beats every fusion on MRR-semantic ×3; bundled bench 11/12 top-1 | disk ≈ vocab × 48 KB + 12 KB/doc; serving 18k docs ≈ 370 MB RAM, 27–63 ms warm; build is nightly-batch, its RAM peak is the SVD smoothing's working set — measured 8.6 GB at 50k words on grid 64 (`research/ram_build.py`, was 13.3 GB before the buffer-discipline pass), floor ≈ 12 bytes × vocab × grid dims; one shared 84 MB embedding table |
| Lexical Q&A — queries reuse the documents' own vocabulary (FAQ, homework, tickets) | four-channel fusion (`--hybride --atlas` + `--fusion`) | full Alloprof through the engine: quartet **0.5461 R@10** vs 0.5035 trio, 29 ms/query | the row above + BM25 postings (a few MB — an explicit word inventory of your docs: the privacy trade-off is yours) + 0.5 KB/doc rerank vectors + 4 KB/doc atlas maps + a build-time SOM (minutes, a few GB RAM at large vocabularies) |
| Catalogs and records dense in identifiers (products, part numbers, case files) | `--grilles-typees` (+ `--rerank-vectors`) | product bench: drowned ref 0.90 vs 0.825, bare ref 0.9917 with rerank | often *cheaper* than default: meaning grid up to 4× smaller when the vocabulary allows, identifier grids are tiny (768 dims ≈ 3 KB/word) |
| Folders that evolve over time, where stale versions are a trap | any of the above + `mosaic actuel` | temporal-truth bench (stale version ranked first by flat search, flagged by `actuel`) | free — reads the facets the index already stores |
| Code repositories | **unknown — not measured yet** | open question: for symbol lookup, grep/LSP are native and exact — the honest hypothesis to test is intent-to-code on mixed repos | — |

Two rules fall out of this table. First: **measure on your own corpus** — 20–40
ground-truth queries and the bundled benches (`bench/run_bench.py`,
`bench/fusion_bench.py`) settle in minutes what no doctrine can. Second: the roadmap
follows the same logic — `mosaic calibrer` already picks encoding weights from your
ground truth; teaching it to pick the *architecture* flags the same way is the natural
next step.

## Quickstart

```bash
pip install -e ".[dev,ingest]"        # core + document conversion
# optional extras: .[ocr] (scanned documents), .[rerank] (model2vec channels),
#                  .[ingest-rapide] (alternative converter, see below)

mosaic build ./my_documents -o ./index_docs
mosaic search "how do I wire the differential breaker" ./index_docs --top 5
mosaic explain <doc_id> ./index_docs --query "..."   # why did it match?
```

Everything is JSON on stdout — built for agents first, readable by humans. For agent
frameworks, run the MCP server (`infra_mcp/mosaic_mcp.py`): indexes stay open in
memory, warm answers in tens of milliseconds. Real output from the bundled benchmark
(paraphrased queries, no keyword overlap with the documents):

```bash
$ mosaic search "melanger de l'huile et un jaune pour une sauce onctueuse" ./index_bench --rerank
[{"id": "04_mayonnaise.md", "score": 0.4988, "score_rerank": 0.5507}, ...]
```

**Requirements:** Python ≥ 3.12, any OS (CI runs Linux; developed on Windows). No GPU,
no network access at runtime. Core dependency: numpy only.

## Measured performance

Headline numbers — the methods, the defeats and the full campaign stories are in
[docs/MEASURES.md](docs/MEASURES.md), and every figure is replayable from `bench/` or
`research/`.

| Benchmark | Result |
|---|---|
| Bundled paraphrase bench (40 docs, 12 traps) | 11/12 top-1 default, **12/12** with `--grilles-typees` |
| Alloprof (2,556 docs, 2,316 real queries) — single systems | Mosaic calibrated 0.385 R@10 > model2vec 0.379; BM25 0.482 |
| Alloprof — fusions | standard hybrid 0.498 < three-channel 0.517 < **engine quartet with `--atlas` 0.5461** (29 ms/query) |
| Real product records (private corpus, 500 refs) | drowned reference 0.90 vs 0.825 standard; bare ref 0.9917 with rerank |

Footprint, measured on production indexes (default 64×64×3 grid, plain CPU):

| Metric | Small index (579 docs) | Large index (18,070 docs) |
|---|---|---|
| Disk total | 1.55 GB | 3.79 GB |
| — document grids (int8) | **12.1 KB/doc** | 12.1 KB/doc |
| — co-occurrence profiles | 48.4 KB/word × 31k words | 48.2 KB/word × 72k words |
| Warm search latency (MCP server) | **27 ms** | **63 ms** |
| Process RAM with index open | — | **373 MB** (lazy memmaps) |
| Full build, all channels (40 docs) | 14.5 s | scales ~linearly, nightly-batch |

Levers if you need smaller/faster: `--grid 32x32` (÷4 every vector cost),
`--grilles-typees` on structured corpora (meaning grid up to 4× smaller, measured),
`--smoothing-rank 0` (skip the SVD: faster build, lower recall), skip optional extras.

## How it works — the tiles of the mosaic

Each document is encoded into a color grid — **12,288 dimensions (64×64×3) by
default**; typed grids size each grid to its own vocabulary in the (c,c,3) family, so
every grid still renders as a mosaic. The grid superposes a deterministic SHA-seeded
**signature** per token, a **co-occurrence profile** learned from *your* corpus
(PPMI + truncated SVD), and optionally a static **embedding**. Search is a cosine
against int8-quantized vectors — one reading per grid on a typed index, synthesized by
the query's idf mass.

<p align="center"><img src="docs/grid_example.png" width="256" alt="The same document as a standard hash grid: uniform confetti, the layout is arbitrary"> <img src="docs/atlas_example.png" width="256" alt="The same document as an atlas heatmap: a few semantic neighborhoods light up"><br><em>The same real document (a recipe from the bundled bench), two ways. Left: the standard
grid — the founding metaphor, and an honest confession: a permutation test proved this
layout is inert (shuffle every cell, not one score moves). Right: the learned atlas
(<code>--atlas</code>) — neighboring cells now hold related concepts, the document lives in a
few semantic neighborhoods, and the picture finally means what it shows. The metaphor
had to earn its truth, and the measures of that journey ship in <code>research/</code>.</em></p>

Around that core, the other tiles: **BM25 postings** over the same token stream
(`--hybride`); a **semantic atlas** (`--atlas`) — a SOM learned from the co-occurrence
profiles, so neighboring cells hold related tokens, whose document heatmaps form a
4th fusion channel with errors decorrelated from the grid's; a **grammatical channel**
(`--grammatical`) — rule-produced roles bound to token signatures by the same circular
permutation as relations, a few structural traces per document with abstention over
guessing; **relations** (hyperdimensional binding by circular permutation) for graph
hops; the **belief memory** (bipolar MAP vectors) with conformal calibration.

Determinism, stated precisely: on a given machine, the same corpus produces the same
index bit for bit; across machines, the int8-quantized search matrix is provably
identical (BLAS float variance is absorbed by quantization — measured: 0 changed
values out of 24.5M) while float artifacts may differ in inert decimals — the
*ranking* is identical everywhere.

The research behind every design decision ships with the code (`research/`):
superposition capacity limits, multi-hop viability, conformal abstention, the atlas
track including its measured dead end — every mechanism was measured before it was
built, and the burials are documented next to the graduations.

## Language support (read this before installing)

Mosaic is currently **French-first**: tokenizer, stopwords, bundled lexicons and the
recommended embedding table target French corpora.

- **French** — native, fully supported; all benchmarks above are French.
- **English queries over French documents** — supported through a deterministic
  ~11.8k-term lexicon bridge.
- **Other Latin languages** (ES/IT/PT/DE) — partially usable, no bundled lexicon yet.
- **Non-Latin scripts (Arabic, CJK…)** — **not supported yet**: a query returns an
  empty list, not bad results.

The CLI is French-first too (`calibrer`, `chemin`, `actuel`, `--explique`…);
human-mode explanations are bilingual (`--langue en`).

## Limitations — the honest list

- **Bag-of-words semantics at the core.** General word order is not encoded; a large
  transformer will beat it on subtle nuance — that is the price of sovereignty. Two
  named exceptions narrow specific failures without changing the nature of the engine:
  learned collocations, and the opt-in grammatical channel, which deposits a few
  high-value structural *traces* (negation scope, upstream/downstream, agent/patient)
  — traces, not sentence reconstruction.
- **Linear scan latency.** Search is a full-corpus cosine: excellent up to tens of
  thousands of documents (63 ms @ 18k docs warm), not designed for millions — and the
  measured sub-linear shortcut (pyramid prefilter) failed its bench, so we did not
  ship it.
- **Disk is vocabulary-driven** (~48 KB per distinct word on the default grid). Large
  vocabularies mean multi-GB indexes; RAM stays low (lazy memmaps), budget the disk.
- **The engine recalls; it does not read.** It returns the right documents and why —
  your agent (or you) still reads them.

## Project status

**v0.1 — early.** Extracted from a private codebase where it was built and benchmarked
against real business corpora (thousands of real documents, ~590 tests, measured
research notes in `research/`). Day-to-day production usage is just beginning: expect
rough edges, and expect honest fixes.

## Why not just…

- **…BM25?** It wins alone on lexical terrain (we publish that defeat) and loses on
  paraphrase; the measured winner on lexical terrain is the fusion *with* the grid's
  decorrelated errors, not either system alone. The baseline ships in `bench/` —
  measure it on your own corpus.
- **…an embeddings API?** Every indexed document is a paid API call, re-paid on every
  rebuild, and your data leaves the machine. Mosaic rebuilds nightly for free, offline.
- **…a static embedding model alone (model2vec)?** Measured twice: behind the
  corpus-learned channels on the bundled paraphrase bench (MRR 0.830 vs 0.958) and
  behind calibrated Mosaic on Alloprof (0.379 vs 0.385). The corpus-learned channel is
  not decoration.
- **…a vector database?** That is infrastructure to run and secure. Mosaic is files on
  disk and one Python process.

## Scientific background

The mechanisms are standard, measured, and referenced in the code: PPMI + truncated
SVD (LSA lineage), hyperdimensional computing / MAP-VSA binding (Kanerva; Gayler),
self-organizing maps (Kohonen) for the atlas, reciprocal rank fusion (Cormack et al.),
conformal prediction for calibrated abstention (Vovk et al.), static embeddings via
model2vec distillation.

## Design principles

1. **Deterministic or explicit.** Same input, same ranking, any machine — bit-for-bit
   reproducible on the same machine. What cannot be guaranteed is stated, never
   silently degraded.
2. **The engine recalls, the caller judges.** Scores, margins, provenance and
   explanations are always exposed; ambiguity raises a flag instead of a silent guess.
3. **Parameters describe *your world*, never the geometry.** Profiles configure roles,
   types and codes; weights are calibrated by measurement; the core invariants are not
   knobs.
4. **Measured, defeats included.** No feature ships without a bench; no bench is
   quoted only when it flatters. The dead ends stay in `research/` next to the
   graduations.

## Development

```bash
pip install -e ".[dev]"
pytest -q            # ~590 tests, two CI regimes (with and without optional extras)
ruff check && ruff format
```

Architecture is enforced by import-linter contracts (core bricks never depend on the
orchestrator). CI runs both a lean profile (no extras) and a full one.

## License

Apache License 2.0 — see [LICENSE](LICENSE).

The bundled French↔English lexicon derived from [WikDict](https://www.wikdict.com/) is
redistributed under its own terms (CC BY-SA); see `src/mosaic/data/LICENSE_wikdict.txt`.