SAGA

SAGA — Self-organizing Archive for Generative Agents — is a self-organizing, AI-native document archive for RAG agents: ingest any text-convertible document, enrich it with LLM-extracted metadata, keep it in Postgres (system of record), project it into OpenSearch (keyword + vector), and let agents query it via fused hybrid search and organise it over MCP.

Status: 1.0.0 — the full ingestion-to-search pipeline is implemented (storage, REST API, conversion, LLM analysis, chunking/embeddings/indexing, hybrid search + MCP, and backup/export). See the implementation plan.

What it does

• Ingest documents in many formats (PDF, Office, HTML, images, scanned docs, …) via the REST API.

• Convert to Markdown using containerised Docling (PDF) and Kreuzberg (everything else, incl. OCR) — routing is configurable.

• Enrich with an LLM: doc-type classification (first-class types), extraction of identifiers/numbers (invoice/contract numbers, phone numbers, IBANs, dates, amounts, …), a short summary, and placement into folders (hierarchical, n:m) using document-similarity voting.

• Store the authoritative record in Postgres (documents, folders, doc-types, notes, memberships) and the original binary in MinIO; project text + summary + filter fields into an OpenSearch document index and chunk vectors into a separate vector index.

• Search via an MCP server offering performant fused hybrid (keyword + semantic, RRF) retrieval, metadata filtering, and folder-tree browsing — plus write tools to reorganise documents, folders, doc-types, and notes.

• Back up everything to a directory tree via a paginated export API + script.

See the architecture for details.

Related MCP server: Librarian

Architecture at a glance

Client ──REST(Bearer)──► API ──┬─► Postgres (system of record)
                               ├─► MinIO (originals)
                               ├─► OpenSearch (doc projection + vector index)
                               └─► Redis ──► Worker (ARQ)
                                              ├─► Docling / Kreuzberg (convert)
                                              ├─► LLM (classify type / extract / summarise / place)
                                              ├─► Embeddings
                                              ├─► Postgres (persist)
                                              └─► OpenSearch (project + index)
Agent ──MCP(HTTP, Bearer)──► MCP server ──► Postgres + OpenSearch (fused hybrid search) + Embeddings

Providers (LLM + embeddings) are pluggable: Ollama (default), OpenAI, or Azure OpenAI — all configurable.

Quick start (Docker)

cp .env.example .env          # then edit the secrets
docker compose up -d          # starts the full stack

# Pull the default Ollama models (first run only)
docker compose exec ollama ollama pull llama3.1:8b
docker compose exec ollama ollama pull nomic-embed-text

• REST API + Swagger UI: http://localhost:8000/docs

• MCP endpoint: http://localhost:8100

• OpenSearch Dashboards: http://localhost:5601

• MinIO console: http://localhost:9001

Local development

# Install uv: https://docs.astral.sh/uv/
uv sync                       # create venv + install deps (app + dev)
uv run ruff check .           # lint
uv run ruff format --check .  # format check
uv run mypy                   # strict type check
uv run pytest                 # unit tests

Configuration

All behaviour is driven by YAML in config/ with ${ENV} overrides for secrets:

Prompts and MCP tool descriptions live in prompts/.

Documentation

Start at the documentation index.

• Operations & deployment · Configuration reference

• REST API · MCP tools · Backup

• Functional requirements · Non-functional requirements · Architecture

• Implementation plan

• Contributing · Guide for AI agents

License

Apache-2.0 — see LICENSE.