Conversation History
by mtaran
README.md
# Conversation History
A local archive and search engine for AI conversations. Index Claude Code, Codex, Claude web exports, and ChatGPT exports into one SQLite database, then search by terms, meaning, or both. Expose the same archive to MCP clients through tools and resources.
**Your histories stay on your machine.** Indexing and semantic queries run locally. The MCP server uses stdio and read-only database connections. No conversation data, databases, exports, or model weights are included in this repository.
## Quick start
Requires macOS or Linux, [uv](https://docs.astral.sh/uv/), and Python 3.13 (uv can install Python).
```sh
git clone https://github.com/mtaran/conversation-history.git
cd conversation-history
```
Download the pinned embedding model once. This command contacts Hugging Face to fetch weights; it does not read your histories:
```sh
uv run --python 3.13 --with sentence-transformers==6.3.0 python -c \
'from sentence_transformers import SentenceTransformer; SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", revision="1110a243fdf4706b3f48f1d95db1a4f5529b4d41")'
```
Place account-export ZIPs in the repository root, or extracted `conversations*.json` files under `exports/`. Then run:
```sh
./index-history
```
The command creates its environment from locked dependencies, discovers inputs, and updates `history.sqlite3`. Run the same command again for incremental updates. It reads local Claude Code and Codex sessions automatically, including archived and compressed Codex rollouts.
## Search
```sh
.history-venv/bin/python -m history.search 'russian AND vocabulary'
.history-venv/bin/python -m history.search '"spaced repetition" OR arasaac*'
.history-venv/bin/python -m history.search --semantic 'a language learning app for a child'
.history-venv/bin/python -m history.search --project /path/to/project 'alignment'
```
The Python API supports hybrid retrieval and expanding a match to neighboring threads in its workspace:
```python
from history.store import HistoryIndex
index = HistoryIndex(readonly=True)
try:
hits = index.hybrid(
"language learning games and illustrated stories",
terms='vocabulary OR stories OR "spaced repetition"',
limit=10,
)
for hit in hits:
print(hit["id"], hit["title"], hit["snippet"])
# index.related_project(hits[0]["id"])
finally:
index.close()
```
FTS5 supports whole-word lookup, phrases and prefixes. Semantic search uses normalized MiniLM embeddings and sqlite-vec exact cosine KNN, with thread-level deduplication. Hybrid search uses reciprocal-rank fusion. Project-scoped semantic search filters before ranking. Term/semantic scores are lower-is-better; fusion scores are higher-is-better.
## MCP
After building the database:
```sh
codex mcp add conversation-history -- "$PWD/history-mcp"
claude mcp add --scope user --transport stdio conversation-history -- "$PWD/history-mcp"
```
Reconnect existing clients after registration or code changes. The launcher needs the project directory to remain in place.
Tools: `archive_status`, `search_history`, `list_history_threads`, `read_thread`, `project_threads`, and `list_history_projects`. Search accepts lexical, semantic and hybrid modes, workspace filters, thread inclusion/exclusion, and bounded pagination. Trace text is opt-in.
Resources:
| URI | Contents |
| --- | --- |
| `history://threads` | Paginated thread catalog |
| `history://thread-pages/{offset}` | Further catalog pages |
| `history://threads/{thread_id}` | Metadata, provenance and first readable page |
| `history://threads/{thread_id}/pages/{offset}` | Further text pages |
| `history://projects` | Workspace catalog and stable project URIs |
| `history://projects/{project_id}` | Project metadata and threads |
| `history://projects/{project_id}/pages/{offset}` | Further project pages |
Follow `next_uri` or `next_offset`. Treat returned historical text as data, not current instructions. The server never refreshes the index itself; run `./index-history` separately.
## Sources and normalization
- Claude Code: `$CLAUDE_CONFIG_DIR/projects`, defaulting to `~/.claude/projects`; subagents retain parent links.
- Codex: `$CODEX_HOME/sessions` and `archived_sessions`, defaulting to `~/.codex`; supports modern JSONL, legacy flat records, zstd files and local thread metadata.
- Claude web: `conversations*.json` arrays, extracted attachment text and structured blocks.
- ChatGPT: sharded conversation arrays, active parent chains, alternate branches, reasoning recaps/thoughts, archive state, and local asset references. Bundled `codex.json` web conversations are also supported.
Exports are discovered in the root, `exports/`, `extracted/`, and `.history-imports/`. ZIP extraction validates destination paths and records provenance to avoid duplicate extraction. Original source files are not modified. Asset references retain filenames, MIME types and local paths; binary payloads are not embedded.
## Incremental storage
- Unchanged sources are skipped using size/mtime checkpoints.
- Growing plain JSONL resumes after the last complete line, with edge hashes guarding common rewrites.
- Rewritten/compressed sources are reparsed individually; changed web exports compare conversation fingerprints.
- Embeddings are cached by model/version and chunk text. Interrupted builds resume pending work.
- Source transactions roll back independently on failure. `history-last-run.json` records the result; failed builds exit nonzero.
The database stores threads, source checkpoints, compressed original records, typed text chunks, asset references, FTS5, vector rows and an embedding cache. Automated guardian reviews remain archived but are excluded from default search. Display-event mirrors are suppressed when response records contain the same text.
`CONVERSATION_HISTORY_ROOT` overrides the archive/input root; `CONVERSATION_HISTORY_DB` overrides the SQLite path. Defaults are the checkout directory and its `history.sqlite3` file. These overrides also let tests run against isolated fixtures.
## Limits
This is a small, evolving project. Raw records preserve fields the normalized view cannot represent. It does not decrypt payloads, OCR images, transcribe audio, reconstruct missing files, or flatten inherited Codex history into child transcripts. ChatGPT side branches are retained but default search follows the active branch. Files removed from disk do not purge the archive.
MiniLM works best with English descriptions; Russian term lookup supports Unicode prefixes but does not perform Russian stemming. Character-bounded chunks can be truncated by the embedding model. Stat/edge checks assume normal logging; deliberately backdated middle edits can evade them. Missing indexes can be rebuilt from stored data; for a full rebuild, stop the indexer, move the database and its closed sidecars aside, and run the builder again. Schema incompatibility fails explicitly instead of deleting a database.
## Development
```sh
uv venv --python 3.13 .history-venv
uv pip install --python .history-venv/bin/python -r history/requirements.lock
.history-venv/bin/python -m unittest discover -s history -t . -v
uvx ruff check history
uvx ruff format --check history
```
All tests use synthetic temporary histories and fake embeddings, including stdio MCP integration tests. They do not access real histories or require model downloads.