# AgentOS Knowledge Platform

**A production-minded hybrid RAG system for grounded, cited answers over unstructured pharma documents.**

Part of a larger 5-project Agentic AI platform ([see roadmap](#platform-roadmap)). This project is the foundational knowledge service — every other agent in the platform reaches it through [MCP](#mcp-server).

---

## Why this project

This is a from-scratch reference implementation of the same category of system I built professionally: an unstructured-data pipeline making pharma documents (SOWs, clinical protocols, safety reports, regulatory correspondence) queryable in natural language, with citation-grounded answers a business or clinical user can trust. No proprietary data is used — the corpus is synthetic, mixed with real, public FDA guidance documents for genuine content diversity.

The goal wasn't a tutorial-grade demo. Every major design decision, bug, and fix documented here happened during real development — including a citation-grounding bug that was root-caused and fixed with before/after evidence (see the [case study](#case-study-a-real-citation-grounding-bug)).

## What it does

Ask a natural-language question, get a grounded answer with source citations and a confidence score:

```json
POST /ask
{
  "question": "What is the primary endpoint for the NVX-204 Phase II study?",
  "session_id": "demo",
  "top_k": 5
}
```

```json
{
  "answer": "The primary endpoint for the NVX-204 Phase II study is the change from baseline in DAS28-CRP score at Week 24.",
  "citations": [
    { "chunk_id": "protocol_summary_nvx204.md_recursive_1", "source_file": "protocol_summary_nvx204.md" }
  ],
  "confidence": 1.0,
  "context_used": true
}
```

## Features

- **Hybrid retrieval** — dense (pgvector) + sparse (BM25) fusion via Reciprocal Rank Fusion, with cross-encoder reranking
- **Grounded generation** — structured-output answers with per-claim citations and a confidence score, explicitly instructed to decline when context is insufficient
- **Multi-format ingestion** — Markdown, plain text, PDF (with OCR fallback for scanned pages), DOCX, HTML, and CSV/XLSX, all through one dispatcher
- **Deduplication** — cosine-similarity dedup at ingest, idempotent across repeated ingestion runs (deterministic chunk IDs)
- **Conversational memory** — LangGraph-orchestrated, Redis-checkpointed short-term memory (session-scoped) plus Postgres-backed long-term conversation summaries, with query rewriting so follow-up questions resolve correctly
- **MCP server** — exposes `search_documents`, `retrieve_context`, and `index_documents` as MCP tools for other agents in the platform
- **Live ingestion API** — `POST /ingest` with automatic index refresh, no restart required
- **Tested** — unit + integration tests (isolated test database, mocked external calls), including regression tests tied to real bugs found during development
- **Evaluated** — RAGAS-based quality scoring (faithfulness, context precision/recall, answer relevancy) against a golden dataset
- **Visualized** — UMAP projection of the embedding space, confirming real semantic clustering by document category

## Architecture

```mermaid
flowchart TB
    Client(["Client / Swagger UI / MCP client"])

    Client -->|"POST /ingest"| Ingest["Ingestion"]
    Ingest --> Loader["Multi-format loader<br/>md, txt, pdf, docx, html, csv"]
    Loader --> Chunker["Chunker<br/>fixed-size / recursive (heading-aware)"]
    Chunker --> Store["pgvector store<br/>+ dedup + embeddings"]
    Store --> BM25["In-memory BM25 index"]

    Client -->|"POST /ask"| Graph["LangGraph pipeline"]
    Graph --> Rewrite["Query rewriting<br/>(uses chat history)"]
    Rewrite --> Hybrid["Hybrid search<br/>dense + sparse + RRF + rerank"]
    Hybrid --> Store
    Hybrid --> BM25
    Hybrid --> Gen["Grounded generation<br/>citations + confidence"]
    Gen --> Client

    Graph <-->|"checkpointed state"| Redis[("Redis Stack<br/>short-term memory")]
    Graph -->|"on-demand summarize"| PG[("Postgres<br/>long-term memory")]

    MCP(["MCP clients<br/>(future: Project 2 agents)"]) --> MCPServer["MCP server<br/>search_documents / retrieve_context / index_documents"]
    MCPServer --> Hybrid
    MCPServer --> Ingest
```



## Tech stack

FastAPI · Pydantic v2 · SQLAlchemy (async) · PostgreSQL + pgvector · Redis Stack (RediSearch) · LangGraph · OpenAI (embeddings + generation) · rank-bm25 · sentence-transformers (cross-encoder reranking) · FastMCP · Docker / Docker Compose · pytest · RAGAS · UMAP

## Quickstart

```bash
git clone <repo-url>
cd AgentOS-Knowledge
cp .env.example .env   # add your OPENAI_API_KEY

docker compose up -d
python scratch_test.py   # ingests data/raw_docs/ and runs a sanity-check query

uvicorn app.main:app --reload
```

Open `http://127.0.0.1:8000/docs` for the interactive API.

## API


| Endpoint                                     | Purpose                                                     |
| -------------------------------------------- | ----------------------------------------------------------- |
| `GET /health`                                | Liveness check                                              |
| `POST /ask`                                  | Ask a question, get a grounded, cited, session-aware answer |
| `POST /ingest`                               | Upload a document, index it, immediately searchable         |
| `POST /conversations/{session_id}/summarize` | Persist a long-term summary of a conversation               |
| `GET /conversations/recent-summaries`        | Retrieve recent cross-session summaries                     |


## MCP server

```bash
mcp dev app/mcp_server.py
```

Exposes `search_documents`, `retrieve_context`, and `index_documents` for any MCP-compatible agent client.

## Testing & evaluation

```bash
pytest -v                    # unit + integration tests
python -m eval.run_eval      # RAGAS evaluation against the golden dataset
python -m eval.visualize_embeddings   # UMAP embedding visualization
```

**Results**: faithfulness 1.0, context recall 1.0, context precision 0.93, answer relevancy 0.82. Full breakdown, embedding visualizations, and a worked debugging case study: [`eval/evaluation_report.md`](eval/evaluation_report.md).

## Engineering challenges & fixes

Real production-level issues found and resolved during development — kept
here in detail because they're a stronger signal of engineering depth than
a feature list.

### 1. Citation-grounding bug (retrieval quality)

`/ask` returned a factually correct answer but cited the wrong source
document — a regulatory correspondence log, instead of the actual protocol
document where the fact came from.

**Root cause**: the markdown chunking library split section headings into
metadata only, invisible to embeddings, BM25, and the reranker. The
protocol's "Primary Endpoint" chunk was stored as just *"Change from
baseline in DAS28-CRP score at Week 24"* — the words "primary endpoint"
never appeared in its searchable text, while a competing chunk from another
document happened to contain that exact phrase in a fluent sentence.

**Fix**: contextual chunk enrichment — prepending document title and
section heading directly into each chunk's indexed content. **Verified
with before/after reranker scores**: the correct chunk went from absent in
the top 20 results to a confident second place, and the citation corrected
accordingly.

**Residual limitation**: fluency bias is reduced, not eliminated — a
longer, more fluently-written passage can still occasionally outrank a
short, factually authoritative chunk. Would need a stronger reranker or
explicit source-type weighting to close further.

### 2. Idempotency bug (data integrity)

Re-running ingestion on unchanged documents created duplicate rows on every
run, instead of recognizing the content already existed.

**Root cause**: `chunk_id` was generated with `uuid4()` — random on every
run — defeating the `ON CONFLICT DO NOTHING` upsert logic, which relies on
a stable, repeatable ID to detect "this row already exists."

**Fix**: deterministic chunk IDs derived from stable properties
(`source_file` + chunking strategy + `chunk_index`). **Verified** by
running ingestion twice in a row and confirming stored chunk count didn't
grow.

### 3. Stale in-memory index after live ingestion (operational correctness)

Documents uploaded via `POST /ingest` weren't searchable until the server
was manually restarted.

**Root cause**: BM25 is an in-memory index, rebuilt once per process at
startup. Any new data written to Postgres wasn't reflected in the BM25
index until it was explicitly rebuilt — a gap that showed up repeatedly
across different entry points (the API server, a RAGAS eval script, and
the MCP server each needed their own fix).

**Fix**: `/ingest` (and the MCP `index_documents` tool) now call
`rebuild_bm25_index()` immediately after storing new chunks. **Verified**
by uploading a new document and successfully querying it with no restart
in between.

**Known limitation, by design**: the in-memory approach is a deliberate
dev-scale tradeoff — production deployment would move to OpenSearch
(natively persistent, incrementally indexable), rather than solving
persistence for the current BM25 library.

### 4. Missing infrastructure capability (Redis + LangGraph memory)

Wiring LangGraph's checkpointer to Redis failed with
`unknown command 'FT._LIST'`.

**Root cause**: `langgraph-checkpoint-redis` requires RediSearch (a Redis
module for indexed lookups), which isn't present in vanilla open-source
Redis.

**Fix**: switched the Redis image to `redis/redis-stack-server`, which
bundles RediSearch.

**Deployment implication, documented ahead of time**: AWS ElastiCache's
standard managed Redis does not include RediSearch — production deployment
will need a self-managed Redis Stack instance or an alternative managed
service.

### 5. Dependency-ecosystem instability (RAGAS and MCP)

Two separate multi-step incidents, both root-caused via direct evidence
(inspecting installed package versions and declared dependencies) rather
than trial-and-error:

- **RAGAS** eagerly imports every optional LLM-provider integration at
module load time; an installed `langchain_community` version had removed
a submodule RAGAS's code depended on. Worked around with a stub module;
flagged as fragile and a candidate to revisit (e.g. `deepeval`).
- **MCP/FastMCP**: the official SDK's API had been substantially
restructured between versions, and the companion `fastmcp` package's
loose dependency constraint (`mcp>=1.24.0`, no upper bound) allowed
installing an incompatible newer major version. Fixed with an explicit
compatible pin (`mcp>=1.24.0,<2.0`).

Both are documented as a real characteristic of building on current-
generation LLM/agent tooling — young, fast-moving ecosystems where
companion-library version compatibility isn't always guaranteed by default
resolution.

### 6. CSV parsing — a deliberate strictness tradeoff

A hand-authored CSV with an unquoted comma inside a text field caused
`pandas` to reject the entire file rather than misparse it.

**Design decision, not just a fix**: rather than switching to lenient
row-skipping parsing, strict parsing was kept deliberately — in a
compliance-adjacent domain, silently dropping a row of batch-release data
is worse than failing loudly and requiring the source file to be
corrected.

## Known limitations

- Fluency bias in reranking (see #1 above) — reduced, not eliminated
- No image/diagram captioning — embedded figures in PDFs/DOCX aren't
extracted or described; a page with a diagram and no surrounding text
yields no searchable content for that diagram
- Tabular (CSV/XLSX) chunking isn't row-boundary-aware — rows are
converted to paragraphs but chunked with the same generic logic as prose
- BM25 index is in-memory/per-process (see #3 above)
- Corpus has real class imbalance — 2 real FDA documents account for 92%
of stored chunks, visible directly in the embedding visualization
- CSV parsing fails the whole file on a malformed row, by design (see #6)

## Platform roadmap

This is Project 1 of 5 in a larger interconnected Agentic AI platform:


| #   | Project                                | Status      |
| --- | -------------------------------------- | ----------- |
| 1   | AgentOS Knowledge Platform (this repo) | ✅ Complete  |
| 2   | Multi-Agent Enterprise Assistant       | Not started |
| 3   | AgentOps Platform                      | Not started |
| 4   | AI Safety & Guardrails Platform        | Not started |
| 5   | Continuous Learning Platform           | Not started |


AWS deployment (ECS/EKS, OpenSearch, Terraform, CI/CD) is deliberately deferred until all 5 projects have working local versions.