# NOEMA-AI

An end-to-end Retrieval-Augmented Generation pipeline that ingests documents and
answers queries grounded in their content. It supports **three retrieval paths**
side by side — **vector-based** (embeddings + vector store), **vector-less**
(keyword/BM25 structured retrieval), and **hybrid** (Reciprocal Rank Fusion of
both) — orchestrated with **LangChain** and **LangGraph**, traced with
**LangSmith**, and exposed to external clients via **MCP** (Model Context
Protocol). It also supports **multi-turn chat with session memory** and
**streaming answers** over Server-Sent Events.

```
Document  ─▶  Ingestion  ─▶  Chunking  ─▶  ┬─▶  Embeddings ─▶ Vector Store  ─┐
                                            └─▶  BM25 Index (vector-less)    ─┤─▶  LangGraph  ─▶  LLM  ─▶  Answer
                                                                              │      (retrieve node)
Query     ───────────────────────────────────────────────────────────────────┘
```

## Screenshots

<!--
  Drop your screenshot images into docs/screenshots/ using these exact
  filenames and they'll render automatically on GitHub — no other changes
  needed. Recommended: 1280px+ wide PNGs, dark mode (matches the UI theme).
-->

**Home page — landing view of the NOEMA UI**
![Home page](docs/screenshots/homepage.png)

**Console — ask a question, watch retrieval + generation trace live**
![Console view](docs/screenshots/console.png)

**Hybrid retrieval mode — fused vector + BM25 results with match source**
![Hybrid retrieval](docs/screenshots/hybrid-retrieval.png)

**Chat — multi-turn conversation with session memory**
![Chat with memory](docs/screenshots/chat.png)

**API reference — full endpoint interface view**
![API interface](docs/screenshots/api-interface.png)

**Streaming answer — tokens arriving live over SSE**
![Streaming answer](docs/screenshots/streaming.png)

## Features

- **Vector RAG** — chunking → embeddings → persistent vector store (Chroma) → similarity search. Ships with a zero-dependency offline hash embedding by default, and a real local semantic embedding option (`EMBEDDING_PROVIDER=sentence-transformers`) for actual semantic search
- **Vector-less RAG** — BM25 keyword ranking over raw chunks, no embedding model required
- **Hybrid RAG** — fuses the vector and BM25 result lists with Reciprocal Rank Fusion (RRF), so a chunk that ranks well in *either* retriever surfaces, and one that ranks well in *both* surfaces first
- **Multi-turn chat with memory** — `POST /chat` keeps a rolling window of prior turns per session, so follow-up questions ("what about the second one?") resolve against earlier turns instead of being answered in a vacuum
- **Streaming answers** — `GET /query/stream` (Server-Sent Events) streams the answer token-by-token instead of waiting for the full response
- **Document management** — `GET /documents` lists every indexed source; `DELETE /documents/{source}` removes it from every retrieval path at once
- **Query result caching** — a short-lived in-memory TTL cache serves repeated identical questions without re-running retrieval + the LLM; automatically invalidated on ingest/delete
- **Persistence across restarts** — the BM25 corpus and the ingested-document manifest are saved to `PERSIST_DIR` and reloaded on startup, so a restart no longer silently wipes what was ingested
- **LangGraph pipeline** — ingestion, retrieval (vector / vectorless / hybrid), and generation modeled as composable graph nodes, with automatic fallback from vector to vector-less
- **LangSmith tracing** — every LLM call and retrieval step is traceable/evaluatable when configured
- **LLM Gateway / Middleware** — a single layer that routes requests, logs, and rate-limits before calls reach the LLM
- **Guardrails** — input/output checks with normalized (whitespace/zero-width-character-resistant) prompt-injection pattern matching; still pattern-based, not a full classifier — see the note below
- **API key auth** — optional `X-API-Key` header enforcement on `/ingest`, `/query/*`, `/documents/*`, and `/chat/*`, off by default so the app still runs with zero setup
- **MCP server** — exposes `ingest_document`, `delete_document`, `list_documents`, `query_vector_rag`, `query_vectorless_rag`, `query_hybrid_rag`, and `chat_with_memory` as MCP tools for external LLM clients
- **Docker support** — a `Dockerfile` and `docker-compose.yml` for one-command containerized deployment with a persistent volume
- **52 unit/API tests** covering the chunker, guardrails (including evasion attempts), the BM25 store, hybrid RRF fusion, the query cache, chat memory, and the FastAPI endpoints (including auth enforcement, document deletion, and chat sessions)

## Project layout

```
noema/
├── app/
│   ├── config.py              # environment / settings
│   ├── main.py                 # FastAPI entrypoint (REST API)
│   ├── middleware.py           # LLM gateway / logging / rate-limit middleware
│   ├── guardrails.py           # input & output guardrails
│   ├── cache.py                 # TTL query-result cache
│   ├── chat.py                  # session-based conversational memory
│   ├── ingestion/
│   │   ├── loader.py           # load .txt / .pdf / .docx into raw text
│   │   └── chunker.py          # sliding-window chunking with overlap
│   ├── retrieval/
│   │   ├── embeddings.py       # embedding provider (API or local fallback)
│   │   ├── vector_store.py     # Chroma-backed Vector RAG
│   │   ├── vectorless_store.py # BM25-backed Vector-less RAG (+ persistence)
│   │   └── hybrid.py            # Reciprocal Rank Fusion of vector + BM25
│   ├── llm/
│   │   └── client.py           # LLM client (Groq / LLaMA 3.3 70B, mockable, streaming + history-aware)
│   ├── graph/
│   │   └── pipeline.py         # LangGraph state machine wiring it all together
│   ├── mcp/
│   │   └── server.py           # MCP server exposing the pipeline as tools
│   └── static/                 # web UI (served at "/" by app/main.py)
│       ├── index.html
│       └── assets/
│           ├── style.css
│           └── app.js
├── docs/screenshots/            # drop screenshot images here (see Screenshots above)
├── data/sample_docs/sample.txt # example document to ingest
├── scripts/ingest.py            # CLI: ingest a document from the command line
├── tests/                       # 52 tests: chunker, guardrails, stores, hybrid, cache, chat, API
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .env.example
```

## Setup

```bash
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env                # then fill in your keys
```

### Environment variables (`.env`)

| Variable | Purpose |
|---|---|
| `GROQ_API_KEY` | Key for Groq-hosted LLaMA 3.3 70B calls (leave blank to use the built-in mock LLM for local testing) |
| `LANGCHAIN_TRACING_V2` | `true` to enable LangSmith tracing |
| `LANGCHAIN_API_KEY` | LangSmith API key |
| `LANGCHAIN_PROJECT` | LangSmith project name, e.g. `noema` |
| `VECTOR_DB_PATH` | Local path for the persistent Chroma store, default `./chroma_db` |
| `EMBEDDING_PROVIDER` | `hash` (default, deterministic offline bag-of-words embedding, zero deps) · `sentence-transformers` (real local semantic embeddings — `pip install sentence-transformers`, downloads a small model on first use) · `remote` (placeholder for a hosted embedding API you wire in) |
| `EMBEDDING_MODEL_NAME` | Model name used when `EMBEDDING_PROVIDER=sentence-transformers`, default `all-MiniLM-L6-v2` |
| `API_KEY` | Leave blank to run the API in open dev mode. Set a value to require a matching `X-API-Key` header on `/ingest` and `/query/*` |
| `PERSIST_DIR` | Where the BM25 corpus and ingested-document manifest are saved, default `./storage`. Restoring on startup means a restart no longer wipes what was ingested |
| `QUERY_CACHE_TTL_SECONDS` / `QUERY_CACHE_MAX_ENTRIES` | How long identical `(mode, query, top_k)` requests are served from cache, and how many entries the cache holds before evicting |
| `CHAT_HISTORY_MAX_TURNS` / `CHAT_SESSION_TTL_SECONDS` | How many turns of conversation `/chat` keeps per session, and how long an idle session is kept before being swept |
| `HYBRID_RRF_K` | Reciprocal Rank Fusion constant for `/query/hybrid`; higher values reduce how much rank position 1 vs 2 matters |

## Running it

**1. Ingest a document**

```bash
python scripts/ingest.py data/sample_docs/sample.txt
```

**2. Start the API (and the UI)**

```bash
uvicorn app.main:app --reload
```

Open **http://127.0.0.1:8000** — this now serves the NOEMA web UI directly
from the FastAPI app (`app/static/`). Drop a file on the ingest panel, pick a
retrieval mode, and ask a question; the pipeline trace strip lights up as the
request moves through guardrails → retrieval → generation, and results show
which lane (vector / vector-less) answered along with the retrieved chunks
and their scores.

Endpoints:
- `GET /health` — liveness + uptime
- `GET /stats` — indexed document list, BM25 chunk count, vector store availability, active chat sessions, query cache stats
- `GET /documents` — list every indexed source with its chunk count
- `DELETE /documents/{source}` — remove a document from every retrieval path
- `POST /ingest` — upload/ingest a document
- `POST /query/vector` — answer a query using Vector RAG
- `POST /query/vectorless` — answer a query using Vector-less (BM25) RAG
- `POST /query/hybrid` — answer a query using Reciprocal Rank Fusion of both
- `POST /query/graph` — run the full LangGraph pipeline (vector / vectorless / hybrid, with fallback)
- `GET /query/stream` — Server-Sent Events: streams the answer token-by-token
- `POST /chat` — multi-turn conversational endpoint with session memory
- `GET /chat/{session_id}/history` — read back a session's turns
- `DELETE /chat/{session_id}` — clear a session

The UI is plain HTML/CSS/JS (no build step) so it runs from the same
`uvicorn` process with no extra tooling. If you'd rather host it separately,
set `window.__DOCUMIND_API_BASE__` in `app/static/index.html` before
`app.js` loads, and point it at wherever the API is running — CORS is
already open on the API side.

**3. Run the MCP server** (so an MCP-compatible client can call this pipeline as tools)

```bash
python -m app.mcp.server
```

**4. Or run it in Docker**

```bash
docker compose up --build
```

This builds the image, mounts a named volume at `/data` for the BM25
corpus/Chroma store to persist across container restarts, and serves the
API + UI on `http://localhost:8000`. Configuration still comes from `.env`
(referenced via `env_file` in `docker-compose.yml`).

## Notes on the current state

This is a working scaffold, not a black-box demo: the chunker, guardrails,
BM25 retrieval, hybrid fusion, query cache, chat memory, and the FastAPI
endpoints (52 tests total) run fully offline and are unit-tested. The
vector store, LLM client, and MCP server are wired up with real
integrations but need a `GROQ_API_KEY` (and, optionally,
`chromadb`/LangSmith credentials) to hit real APIs. Without keys, the LLM
client falls back to a mock responder so the pipeline still runs end-to-end
for development and demos.

Known, honestly-stated limitations:
- **Embeddings**: the default `EMBEDDING_PROVIDER=hash` is a bag-of-words
  hash, not real semantic search — good for wiring/testing the pipeline
  offline, not for retrieval quality. Set `EMBEDDING_PROVIDER=sentence-transformers`
  for real semantic embeddings (needs an install + a one-time model download).
- **Guardrails** are regex/pattern-based prompt-injection checks, not a
  trained classifier — they catch common phrasings and simple spacing/unicode
  evasions, not a determined adversary. Swap in a managed guardrails service
  for anything beyond a demo.
- **Persistence is file-based, not a real database**: the BM25 corpus and
  ingested-document manifest are now saved to `PERSIST_DIR` and reloaded on
  startup (previously they were in-memory only and a restart silently wiped
  them), but this is still a JSON file on local disk, not a proper store —
  it won't behave correctly behind multiple worker processes or hosts.
  Chat sessions and the rate limiter are still in-memory-only by design
  (short-lived, session-scoped state that doesn't need to survive a
  restart).
- **API auth** is a single shared `X-API-Key`, not per-user auth — fine for a
  demo or an internal tool, not a substitute for real auth in a multi-tenant
  deployment.
- **The query cache is a simple in-memory TTL dict**, not a distributed
  cache — fine for a single-process deployment, and it's invalidated on
  every ingest/delete so it can't serve a stale answer past a document
  change, but it won't be shared across multiple worker processes.

A dependency-free web UI sits in front of all of this (`app/static/`, served
at `/`), and several functional bugs were fixed while wiring it up so the
demo actually works out of the box: `python-multipart` was missing from
`requirements.txt` (FastAPI needs it for file uploads — `/ingest` would 500
on boot without it), the BM25 vector-less path was silently dropping
every hit on small corpora (with only one or two chunks indexed, BM25's IDF
term can go negative for common words, so the old `score <= 0` filter
discarded true matches — it now ranks by score but only discards chunks that
share no terms with the query), and the frontend was reading
`stats.vectorless_chunk_count` / `stats.ingested_documents` while `/stats`
returned differently-named fields, so the document list and chunk count in
the UI always silently showed zero regardless of what had actually been
ingested.