*English | [日本語](README.ja.md)*

# Hybrid RAG — Agent Long-Term Memory System

A hybrid RAG system combining dense and sparse retrieval, with a **tag-based memory mechanism** (importance, staleness rate per `knowledge_type`, and access frequency) built into its reranking. For design rationale, see [hybrid_rag_agent_spec.en.md](hybrid_rag_agent_spec.en.md).

Run it as an MCP server and agents such as Claude Code can use it directly as "long-term memory."

## How this mechanism works

Following the spec, processing is split into two kinds.

| Class | Content | Implementation |
|---|---|---|
| **① Model-dependent (reasoning)** | Importance tagging, query expansion / sufficiency judgment, orchestration | Agent side (LLM judgment) |
| **② Structure-dependent (deterministic processing)** | Chunking, embedding generation, hybrid search, staged reranking, forgetting/archival | RAG side (this library / MCP server) |

"Importance" and "staleness rate per `knowledge_type`" are treated as separate axes; rather than a simple linear combination, they are applied in stages: **① cutoff by importance → ② time decay by `knowledge_type` → ③ boost by access frequency** (see spec section 2.3 for details).

```
principle  : no decay (MBSE design principles, math/algorithms)
paper      : re-evaluated roughly every half year (papers, technical articles)
news       : decays significantly over weeks to months (news, model-release info)
experiment : decays according to project duration (experiment logs, run records)
```

`knowledge_type` is designed to be determined deterministically from the ingestion source rather than judged by an LLM from chunk content (e.g., a design document explicitly registered by a human → `principle`; an arXiv paper/technical article → `paper`; news/web search results → `news`; an execution log → `experiment`).

> **Note**: `principle` (no decay) does **not** guarantee that a chunk will "never be forgotten by `run_forgetting_batch`." Because staged reranking applies the ① importance cutoff first, a `knowledge_type=principle` chunk can still become an archival target if its `importance` is set low and falls below `importance_threshold` (confirmed by `tests/test_archival.py`). "No decay" applies only to the ② time-decay stage — it is not a "never forgotten" guarantee across the full ①②③ pipeline.

> **Note**: The memory mechanism (knowledge_type/importance/staged reranking/forgetting batch/MCP server) is **implemented only for the FAISS backend (`HybridRAGSystem`)**. The Qdrant/Chroma/PostgreSQL versions are available only as a plain hybrid-search library.

## Installation

```bash
pip install -r requirements.txt
```

For development/testing:
```bash
pip install -r requirements-dev.txt
```

## Usage ① As an MCP server (recommended)

### Starting the server

```bash
python mcp_server/server.py
```

The storage locations can be set via environment variables (defaults: `hybrid_rag.db` / `indices`).

```bash
HYBRID_RAG_DB_PATH=my_memory.db HYBRID_RAG_INDEX_PATH=my_indices python mcp_server/server.py
```

### Registering with Claude Code

[.mcp.json](.mcp.json) at the project root is already set up as follows. Claude Code picks it up automatically when it opens this repository.

```json
{
  "mcpServers": {
    "hybrid-rag-memory": {
      "type": "stdio",
      "command": "python",
      "args": ["mcp_server/server.py"],
      "env": {
        "HYBRID_RAG_DB_PATH": "hybrid_rag.db",
        "HYBRID_RAG_INDEX_PATH": "indices"
      }
    }
  }
}
```

If you use a virtual environment, rewrite `command` to the absolute path of the Python interpreter inside your venv (e.g., `"command": "./.venv/Scripts/python.exe"`).

### Tools provided

In addition to the minimal 3 tools (①–③) called for in spec section 5, this server provides 10 more tools (④–⑬, spec extensions) for data ingestion, tagging, duplicate prevention, forgetting batches, and health checks.

| # | Tool | Description |
|---|---|---|
| ① | `embed(text)` | Vectorizes text with a fixed embedding model (deterministic processing) |
| ② | `hybrid_search(query, tags?, filters?, top_k?, include_stats?)` | Hybrid vector+BM25 search. Returns chunks that have already gone through relevance reranking (Cross-encoder). With `include_stats=True`, the return shape becomes `{"chunks": [...], "stats": {...}}`, adding `orphan_index_entries` (entries dropped as index debris) and `duplicate_contents` (entries collapsed for identical body text) alongside timing info **[`include_stats` is a spec extension, added 2026-08-08]** |
| ③ | `rerank(chunks, time_weight?, freq_weight?, importance_threshold?)` | Staged reranking: importance cutoff → per-`knowledge_type` time decay → access-frequency boost |
| ④ | `ingest(file_paths, metadata?, rebuild_index?)` | Ingests documents. `rebuild_index=True` (default) runs a lightweight incremental update (`update_index`) internally **[spec extension]** |
| ⑤ | `set_chunk_tags(doc_id, chunk_index, importance?, knowledge_type?, tags?)` | Assigns importance tags / re-tags `knowledge_type` (intended for a human gate) **[spec extension]** |
| ⑥ | `find_by_tag(tag)` | Exact-match tag lookup (bypasses semantic search). Used to check whether a document from the same source has already been ingested **[spec extension]** |
| ⑦ | `get_document_chunks(doc_id, chunk_index?, window?)` | Fetches neighboring chunks within the same document (a direct lookup that bypasses semantic search). Compensates for context lost at chunk boundaries **[spec extension]** |
| ⑧ | `delete_document(doc_id, rebuild_index?)` | Deletes a document and all of its chunks. Used in the "replace" flow when re-ingesting **[spec extension]** |
| ⑨ | `update_index()` | A lightweight index update that incrementally incorporates only the chunks added since the last index update **[spec extension, added 2026-07-30]** |
| ⑩ | `rebuild_index()` | Does a **full** rebuild of the FAISS/BM25 indexes from every chunk in the DB. Required after deletions (⑧ or `run_forgetting_batch`), since incremental updates cannot handle them **[spec extension]** |
| ⑪ | `run_forgetting_batch(time_weight?, freq_weight?, importance_threshold?, score_threshold?, dry_run?)` | The forgetting/archival batch job. Intended for infrequent execution only **[spec extension]** |
| ⑫ | `index_health()` | Checks consistency between the index and the DB and reports it (makes no changes). Detects "in the index but not in the DB" debris (from forgetting to call `rebuild_index` after a deletion) and "in the DB but not in the index" gaps (from forgetting to call `update_index` after `ingest(rebuild_index=False)`) **[spec extension, added 2026-08-08]** |
| ⑬ | `get_system_stats()` | Reports the DB's memory-mechanism tagging coverage (makes no changes). Where `index_health` looks at structural consistency between the index and the DB, this looks at "how well the memory axis can actually function" — counts per `knowledge_type`, the importance-set rate, the tag-coverage rate, etc. **[spec extension, added 2026-08]** |

Without ④–⑬, the ①–③ tools alone can neither ingest data, finalize importance tags, nor avoid duplicate registration from the same source — making the system impractical, which is why they were added.

### A note on ingesting many files back-to-back (important)

**Background (a past issue fixed on 2026-07-30)**: `ingest` used to default to "re-embed every chunk in the DB and rebuild the index on every call," so the cost of a single call grew linearly with corpus size, and ingesting files one at a time in a row would time out. `ingest(rebuild_index=True)` (the default) now calls `update_index()` internally — an incremental approach that embeds only the **newly added chunks** since the last update and `.add()`s them to the FAISS index — so it is now fast regardless of overall corpus size (the BM25 side still does a lightweight full rebuild every time, because its IDF statistics depend on the whole corpus, but this is cheap since it involves no neural embedding).

That said, running an incremental update on every single file is still wasted overhead, so when ingesting many files back-to-back it's better to pass `rebuild_index=False` to each `ingest` call and call `update_index()` once at the end of the batch to reconcile everything at once. [.claude/agents/doc-to-memory.md](.claude/agents/doc-to-memory.md) and [.claude/agents/session-to-memory.md](.claude/agents/session-to-memory.md) are already implemented with this pattern. Checking the DB via `find_by_tag` (for duplicate prevention / progress verification) queries SQLite directly, so it works without waiting for the index to catch up.

**When a full `rebuild_index()` is required**: whenever the batch includes even a single `delete_document` call or an archival pass from `run_forgetting_batch` (i.e., any vector deletion). Incremental addition (`update_index`) only supports adding to FAISS, not removing from it, so any batch that includes deletions must end with a full `rebuild_index()`. A batch consisting purely of new additions is fine with `update_index()`.

### Preventing duplicates when re-registering the same source

Because `ingest` derives `doc_id` from a hash of the file's contents, **re-ingesting byte-identical content is automatically skipped** (a diff-based update). However, in cases where the same source (e.g., the same session) is re-summarized by an LLM and re-ingested every time, slight variations in the summary text each time will cause it to be treated as a different document — creating duplicates.

To avoid this, ingest with a unique identifier tag (e.g., `session_id:xxx`) and an updated-at tag (e.g., `session_last_activity:2026-07-28T15:59:49Z`), and on subsequent runs:

1. Check whether the document already exists via `find_by_tag("session_id:xxx")`
2. If the existing updated-at tag matches the current value, **skip — do nothing**
3. Only if it differs (the source has changed), remove the old one with `delete_document(doc_id, rebuild_index=False)` before `ingest`-ing the new content

Implementing this "skip if unchanged, replace if changed" pattern is recommended. [.claude/agents/session-to-memory.md](.claude/agents/session-to-memory.md) is a reference implementation of this pattern.

### Usage example (conceptual)

```
1. ingest(["design_doc.md"], metadata={"knowledge_type": "principle", "tags": ["mbse"]})
2. hybrid_search("about consistency between requirements and architecture", top_k=5)
   -> [{"doc_id": ..., "chunk_index": ..., "content": ..., "knowledge_type": "principle",
        "importance": null, "access_count": 0, "score": 0.87}, ...]
3. set_chunk_tags(doc_id, chunk_index, importance=0.9)
4. rerank(chunks, time_weight=0.5, freq_weight=0.1, importance_threshold=0.3)
   -> chunks reordered along the memory axis (staleness, frequency, importance)
```

## Usage ② As a Claude Code agent

[.claude/agents/rag-memory.md](.claude/agents/rag-memory.md) provides a sub-agent definition responsible for the "agent side (class ①)" of this memory mechanism. Once `.mcp.json` is registered, you can invoke it from Claude Code like:

```
Use the rag-memory agent to look into past design decisions
```

The operating rules for actions that need human intent confirmation — importance tagging, `knowledge_type` re-tagging, deciding when to run the forgetting batch — are also written into this agent definition.

Additionally, [.claude/agents/session-to-memory.md](.claude/agents/session-to-memory.md) is a dedicated agent that summarizes past Claude Code sessions (chat transcripts) and ingests them into long-term memory as `knowledge_type="experiment"`. It runs on a Haiku model to keep costs down, and when reprocessing the same session, it compares against the existing entry via the `session_id`/updated-at tags — skipping if unchanged, replacing if changed (see the previous section). The caller must explicitly specify which sessions to target; it never targets all sessions without limit.

## Usage ③ Directly as a Python library

You can also call it directly from Python code without going through the MCP server.

```python
from hybrid_rag import HybridRAGSystem

rag = HybridRAGSystem(db_path="hybrid_rag.db", index_path="indices")

rag.ingest_documents(
    ["design_doc.md"],
    metadata={"knowledge_type": "principle", "importance": 0.9, "tags": ["mbse"]},
)

result = rag.query(
    "about consistency between requirements and architecture",
    top_k=5,
    enable_memory_rerank=True,   # enable the memory mechanism's staged reranking
    memory_time_weight=0.5,
    memory_freq_weight=0.1,
    memory_importance_threshold=0.3,
)
print(result["context"])

# assign an importance tag after the fact (no vector rebuild needed)
rag.set_chunk_tags(doc_id="design_doc_xxxx", chunk_index=0, importance=0.9)

# forgetting/archival batch (normally run infrequently)
report = rag.run_forgetting_batch(score_threshold=0.05, dry_run=True)
```

### Running the forgetting batch from the CLI

A script intended for infrequent batch execution — e.g., on a 3-month cycle or when a new model is released (it is never run automatically inside the server).

```bash
python scripts/run_forgetting_batch.py --dry-run
python scripts/run_forgetting_batch.py --score-threshold 0.1 --time-weight 0.8
```

Main options: `--db-path` `--index-path` `--archive-path` `--time-weight` `--freq-weight` `--importance-threshold` `--score-threshold` `--dry-run`

Archived chunks are evacuated to `archive/chunks_archive.jsonl` (raw text + metadata + score + deletion reason + deletion timestamp), and their vector representations are discarded.

### Automatically evaluating retrieval accuracy from the CLI

A script that measures retrieval accuracy against a golden query set (Precision@k/Recall@k/MRR/NDCG@k/Hit Rate@k, authority-document rank, noise rate) in a reproducible way, instead of relying on manual queries and eyeballing results via Cursor/Claude Code.

```bash
cp eval/golden_queries.example.yaml eval/golden_queries.yaml  # once, at first use — rewrite the doc_ids for your own corpus
python scripts/run_evaluation.py --db-path mcp_server/hybrid_rag.db --index-path mcp_server/hybrid_rag_indices
```

Main options: `--db-path` `--index-path` `--golden-set` (default `eval/golden_queries.yaml`) `--k-values` (default `1,3,5,10`) `--authority-window` (default `20`) `--output`

### Auditing near-duplicate ingests

`ingest`'s duplicate detection cannot catch cases where identical content enters via a different file (a different path/filename — see "Preventing duplicates when re-registering the same source" above). This script just lists near-duplicates that have already made it into an existing corpus. **It never deletes anything.**

```bash
python scripts/find_near_duplicates.py --db-path mcp_server/hybrid_rag.db
python scripts/find_near_duplicates.py --db-path mcp_server/hybrid_rag.db --output eval/duplicates_report.json
```

It groups documents whose normalized-content hash (`documents.content_hash`) matches. Deciding which one to keep — and whether to delete anything — is left to the user; call `delete_document(doc_id, rebuild_index=False)` manually (and be sure to call `rebuild_index` at the end of the batch).

`eval/golden_queries.yaml` is `.gitignore`d, since it is personal data containing doc_ids specific to your actual corpus. Reports are written to `eval/eval_report_<date>.md` (plus a same-named `.json`), which are likewise `.gitignore`d (they remain on your machine for ongoing tracking).

## Memory-mechanism fields

Fields carried in the `metadata` of `ingest`/the Python API, or per chunk:

| Field | Type | Description |
|---|---|---|
| `knowledge_type` | `str` | `principle` / `paper` / `news` / `experiment`. Determined deterministically from the ingestion source |
| `importance` | `float (0.0–1.0)` | Importance assigned after the fact by the agent. Unset (`None`) always passes the cutoff |
| `tags` | `list[str]` | Arbitrary tags. Used to narrow results via `hybrid_search`'s `tags` argument |
| `access_count` | `int` | Access frequency. Auto-incremented each time a chunk is actually returned by a query |
| `last_accessed_at` / `created_at` | `str` | Last-accessed / created timestamps. Serve as the basis for time decay |

## Tests

```bash
pytest tests/ -v
```

- `test_metadata_pipeline.py`: a regression test that `knowledge_type`/`importance`/`tags` survive the ingest → build_index → query pipeline
- `test_memory_scoring.py`: unit tests for staged reranking (cutoff, decay, frequency boost)
- `test_archival.py`: unit tests for the forgetting/archival batch
- `test_index_health.py`: unit tests for `index_health` (index/DB consistency checking)

See [File layout](#file-layout) for the full list and role of the other test files.

## Base-library functionality (common across backends)

The base RAG functionality — dense/sparse hybrid search, RRF, Cross-encoder reranking, MMR diversity selection, query expansion, caching, etc. — is common to every backend (FAISS/Qdrant/Chroma/PostgreSQL).

```python
from hybrid_rag import create_rag_system

rag = create_rag_system(backend="faiss")   # "qdrant" / "chroma" / "postgres" are also available
rag.ingest_documents(["document1.pdf", "document2.md"])
result = rag.query("What is machine learning?", top_k=5)
```

| Aspect | FAISS | Qdrant | ChromaDB | PostgreSQL |
|------|-------|--------|----------|------------|
| Filtered search | post-processing | fast (single stage) | post-processing | post-processing |
| Server required | no | no | no | yes |
| Scale | up to ~20M | up to ~50M | mid-size | large-scale |
| Memory mechanism (this README) | ✓ | ✗ | ✗ | ✗ |

Optional installs: this repository has no `pyproject.toml`/`setup.py`, so it is not distributed in the `pip install hybrid-rag[...]` form. To use the Qdrant/Chroma/PostgreSQL versions, install the corresponding client library directly (`pip install qdrant-client` / `pip install chromadb` / `pip install "psycopg[binary]" pgvector` — all of these are already listed in `requirements.txt`, so `pip install -r requirements.txt` alone covers them).

Key additional settings (a subset of the FAISS version's `HybridRAGSystem` constructor arguments):

```python
rag = HybridRAGSystem(
    dense_model="paraphrase-multilingual-MiniLM-L12-v2",
    rerank_model="BAAI/bge-reranker-v2-m3",
    max_chunk_size=512,
    index_type="hnsw",           # "flat" / "ivf" / "hnsw"
    enable_mmr=True, mmr_lambda=0.6,
    enable_cache=True, cache_ttl_seconds=3600,
    query_expander=None,          # pass a QueryExpander instance for LLM-based query expansion
    memory_half_life_overrides=None,  # override the half-life (days) per knowledge_type
    enable_guaranteed_candidates=True,  # always add principle/high-importance chunks to the candidate pool (default True)
    guaranteed_knowledge_types=None,    # defaults to ["principle"]
    guaranteed_importance_threshold=0.7,
    guaranteed_candidates_limit=50,
)
```

`enable_guaranteed_candidates` (default `True`) addresses a problem where `knowledge_type=principle` chunks (or chunks with `importance>=0.7`) never made it into the search candidate pool in the first place, and staged reranking could not rescue them (the "principle documents getting buried" issue reported in `RAG_EVALUATION_REPORT_2026-07-30.md`/`RAG_精度テスト_2026-07-31.md`). It works by always adding matching chunks to the candidate pool right after retrieval and letting the Cross-encoder score their relevance — it does not force them to the top. Calls to `query()`/`hybrid_search` that pass `metadata_filters` (`filters`) skip this merge.

## Documentation (Sphinx) / diagrams (PlantUML)

```bash
pip install sphinx sphinx-rtd-theme
python -m sphinx -b html docs/source docs/build
```

`docs/uml/` is intended to hold PlantUML sources for class diagrams, sequence diagrams, and state-machine diagrams (not yet populated as of this writing).

## File layout

```
hybrid_rag_agent_spec.md   # design spec for the memory mechanism
.mcp.json                  # MCP server registration for Claude Code
.claude/agents/rag-memory.md  # sub-agent definition for Claude Code

mcp_server/
└── server.py              # the MCP server itself (13 tools, see the table above)

scripts/
├── run_forgetting_batch.py       # CLI for the forgetting/archival batch
├── run_evaluation.py             # CLI that automatically evaluates retrieval accuracy against a golden query set
├── find_near_duplicates.py       # CLI that audits near-duplicate ingests in the existing corpus (report-only, never deletes)
├── backfill_source_date.py       # bulk-backfills source_date on existing chunks
├── list_md_files.py              # lists candidate Markdown files for ingestion
├── manage_ingest_status.py       # tracks ingest progress against list_md_files.py's listing
├── manage_conv_ingest_status.py  # tracks ingest progress against convert_conversations.py's output
└── convert_conversations.py      # converts a Claude.ai export (JSON) into Markdown

hybrid_rag/
├── __init__.py
├── ingestion.py            # document processing
├── chunking.py             # semantic chunking
├── indexing.py             # dense & sparse index (FAISS)
├── indexing_bm25.py        # BM25 index
├── indexing_sparse_tfidf.py  # TF-IDF sparse index (shared by the Chroma/Postgres/Qdrant backends)
├── indexing_qdrant.py / indexing_chroma.py / indexing_postgres.py
├── retrieval.py            # RRF search
├── reranking.py            # Cross-encoder reranking (relevance axis)
├── memory_scoring.py        # staged reranking (memory axis: importance/decay/frequency)
├── archival.py              # forgetting/archival batch processing
├── index_health.py          # index/DB consistency checking (backs the ⑫ index_health tool)
├── caching.py / embedding_cache.py
├── context.py / diversity.py / evaluation.py
├── storage.py               # SQLite database (including memory-mechanism fields)
├── query_expansion.py
├── rag_system.py            # main orchestrator (FAISS version, implements the memory mechanism)
├── _rag_system_indexing.py  # ^ ingest/build/incremental-update/load (mixin)
├── _rag_system_query.py     # ^ query pipeline (mixin)
├── _rag_system_memory.py    # ^ tags/neighboring chunks/forgetting batch (mixin)
├── _rag_system_stats.py     # ^ stats & cache management (mixin)
├── _rag_system_docops.py    # ^ embedding/delete/lightweight search (mixin)
├── rag_system_base.py       # base class shared by the Chroma/Postgres/Qdrant backends
├── rag_system_qdrant.py / rag_system_chroma.py / rag_system_postgres.py
└── rag_system_factory.py

tests/
├── test_metadata_pipeline.py   # metadata regression test across ingest → build_index → query
├── test_memory_scoring.py      # unit tests for staged reranking
├── test_archival.py            # unit tests for the forgetting/archival batch
├── test_incremental_index.py   # unit/integration tests for update_index (incremental updates)
├── test_index_health.py        # unit tests for index_health (index/DB consistency check)
├── test_result_dedup.py        # unit tests for RRF fusion-key stability and search-result dedup
├── test_diversity.py           # unit tests for MMR diversity selection
├── test_reranking.py           # unit tests for Cross-encoder reranking stats
├── test_retriever_shutdown.py  # tests for RRFRetriever resource cleanup (thread leaks)
├── test_indexing_bm25.py       # unit tests for the BM25 index
├── test_storage_concurrency.py # unit tests for concurrent SQLite writes
├── test_source_date.py         # unit tests for source_date derivation (time-decay reference point)
├── test_document_chunks.py     # unit tests for get_document_chunks (fetching neighboring chunks)
├── test_evaluation.py          # unit tests for RAGEvaluator (Precision@k, etc.)
├── test_database_stats.py      # unit tests for get_database_stats / duplicate-ingest detection
├── test_guaranteed_candidates.py  # unit tests for guaranteed candidate-pool merging (the fix for principle burial)
├── test_rag_system_factory.py  # unit tests for create_rag_system (backend switching)
└── conftest.py                 # shared pytest configuration
```

## License

MIT License