alaya
by luke-kucing
README.md
# alaya
> **ālaya** (ālayavijñāna) — storehouse consciousness. The layer of mind that holds all impressions, memories, and seeds of knowledge.
A FastMCP server that makes Claude Code the primary interface for a personal knowledge vault (zk or Obsidian). Full read/write/search/synthesis access — the AI is the interface, the vault is the source of truth.
## Quickstart
```bash
# 1. Register with Claude Code (one command, no clone needed)
claude mcp add alaya -e ALAYA_VAULT_DIR=$HOME/notes -- uvx alaya-mcp
# 2. Start Claude Code — alaya connects automatically
claude
```
That's it. Ask Claude to "search my notes for X" or "create a note about Y" and it works.
### Vault setup
**zk vault** (requires [zk](https://github.com/zk-org/zk) CLI):
```bash
brew install zk
mkdir -p ~/notes && cd ~/notes && zk init
```
**Obsidian vault** — just point `ALAYA_VAULT_DIR` at any directory with a `.obsidian/` folder. No extra tools needed.
### Running from source
```bash
git clone git@github.com:luke-kucing/alaya.git
cd alaya && uv sync
# Register with Claude Code from source
claude mcp add alaya \
-e ALAYA_VAULT_DIR=$HOME/notes \
-- uv run --directory $(pwd) alaya
```
### Running the server standalone
```bash
# Via make
make serve
# Or directly with env var
ALAYA_VAULT_DIR=~/notes uv run alaya
```
The server communicates over stdio (MCP protocol) — it's designed to be launched by Claude Code, not run in a browser.
### Running tests
```bash
make test # unit tests (607+ tests, no external deps)
make test-integration # integration tests (requires zk binary)
make lint # ruff check
```
## Philosophy
- **The AI is the interface.** Stay in one place — a Claude Code session — and converse. Claude reads, writes, searches, and reasons across the vault.
- **Frictionless capture, deliberate structure.** Inbox capture is a single sentence. Processing, linking, and filing are separate structured acts.
- **Propose, then act.** Destructive or structural changes follow a confirmation flow. Appends and captures are always safe.
- **Pluggable vault backend.** `ZkBackend` delegates to the `zk` CLI; `ObsidianBackend` is pure Python. The vault stays portable either way.
- **Semantic retrieval over keyword search.** LanceDB hybrid search finds notes by meaning, not just exact words.
## Architecture
```
┌──────────────────────────────────────────────────────┐
│ Claude Code │
│ (MCP client / AI layer) │
│ Synthesis, weekly review, gap analysis, │
│ confirmation flow — all Claude's responsibility │
└───────────────────────┬──────────────────────────────┘
│ MCP protocol (stdio)
┌───────────────────────▼──────────────────────────────┐
│ alaya (FastMCP server) │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────┐ │
│ │ tools/ │ │ tools/ │ │ tools/ │ │
│ │ read.py │ │ write.py │ │ search.py │ │
│ │ inbox.py │ │ edit.py │ │ structure.py │ │
│ │ tasks.py │ │ ingest.py │ │ external.py │ │
│ │ capture.py │ │ stats.py │ │ graph.py │ │
│ └──────┬──────┘ └──────┬──────┘ └───────┬────────┘ │
│ │ │ │ │
│ ┌──────▼───────────────▼────────────────▼─────────┐ │
│ │ backend/ (VaultBackend protocol) │ │
│ │ ┌──────────────────┐ ┌──────────────────────┐ │ │
│ │ │ ZkBackend │ │ ObsidianBackend │ │ │
│ │ │ (delegates to zk │ │ (pure Python, │ │ │
│ │ │ CLI subprocess) │ │ no external deps) │ │ │
│ │ └──────────────────┘ └──────────────────────┘ │ │
│ └──────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌──────────────────▼──────────────────────────────┐ │
│ │ Shared layer │ │
│ │ vault.py (path safety) zk.py (CLI wrapper) │ │
│ │ config.py (env vars) watcher.py (watchdog)│ │
│ │ events.py (pub/sub) audit.py (JSONL log) │ │
│ │ cache.py (metadata cache, O(1) lookups) │ │
│ └──────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌──────────────────▼──────────────────────────────┐ │
│ │ index/ providers/ │ │
│ │ embedder.py (nomic ONNX+LRU)gitlab.py (glab) │ │
│ │ store.py (LanceDB) github.py (gh) │ │
│ │ router.py (query class.) │ │
│ │ corrective.py (retry RAG) │ │
│ │ graph_rag.py (link expand) │ │
│ │ hyde.py (HyDE embed) │ │
│ │ contextual.py (chunk ctx) │ │
│ │ reindex.py (rebuild) │ │
│ └──────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────┘
│ │
┌────▼─────┐ ┌──────────────────▼──────────────┐
│ zk CLI │ │ External providers │
│ (notes, │ │ glab CLI · gh CLI · Outline API │
│ links, │ │ (pull issues in, push ideas out)│
│ tags) │ └──────────────────────────────────┘
└────┬─────┘
│
┌────▼──────────────────────────────┐
│ ~/notes (vault) │
│ ├── daily/ │
│ ├── inbox.md │
│ ├── projects/ │
│ ├── areas/ │
│ ├── people/ │
│ ├── ideas/ │
│ ├── learning/ │
│ ├── resources/ │
│ ├── raw/ │
│ ├── archives/ │
│ └── .zk/ or .obsidian/ │
│ ├── vectors/ (lance) │
│ └── audit.jsonl │
└───────────────────────────────────┘
```
## Search Pipeline
Alaya's search is a multi-stage RAG pipeline, not a simple keyword lookup:
```
Query ──▶ Router ──▶ Retrieval ──▶ Correction ──▶ Expansion ──▶ Results
│ │ │ │
│ │ │ └─ Graph RAG (opt-in)
│ │ │ wikilink traversal
│ │ │
│ │ └─ Corrective RAG
│ │ score check → reformulate → retry
│ │
│ └─ Hybrid Search (vector + BM25 + RRF)
│ or Keyword-only / Semantic-only
│
└─ Adaptive Router
KEYWORD | SEMANTIC | TEMPORAL | HYBRID
```
### Search features
| Feature | Description | Flag |
|---------|-------------|------|
| **Adaptive routing** | Classifies queries — short terms use BM25, questions use vectors, time refs auto-extract dates | Always on |
| **Hybrid search** | LanceDB native vector + FTS with Reciprocal Rank Fusion | Default |
| **Corrective RAG** | Retries with reformulated queries when results score below threshold | Always on |
| **Cross-encoder reranking** | Second-stage reranking with `ms-marco-MiniLM-L-6-v2` for higher precision | `rerank=True` |
| **Graph RAG** | Expands results via 1-hop wikilink traversal (outlinks + backlinks) | `graph_expand=True` |
| **HyDE** | Embeds a hypothetical answer document instead of the raw query | `hyde=True` |
| **Contextual retrieval** | Prepends metadata context (title, directory, tags, section) to chunks at index time | Always on |
### Design decisions
**Plain functions, not classes.** Tools are `async def get_note(path, vault) -> str` — no inheritance, no base classes. State is passed explicitly. A `Chunk` dataclass is the only data container.
**VaultBackend as the seam.** Tools call `backend.list_notes()`, `backend.search()`, etc. `ZkBackend` delegates to the `zk` CLI via `run_zk()`; `ObsidianBackend` is pure Python. Unit tests mock `alaya.zk.run_zk` for zk paths, or use real filesystem fixtures for Obsidian paths. Wiring tests prove each tool dispatches through the backend.
**LanceDB is additive.** Core tools work without any index. Search falls back to backend keyword search. The vector index is an optimization layer, not a requirement.
**Tools return Markdown strings.** Claude reads Markdown; it doesn't parse JSON. Tables, bullet lists, and metadata headers are formatted for Claude consumption.
**Confirmation lives in the server, with Claude on top.** Claude Code prompts before a destructive tool call, but an arbitrary MCP client — an agent loop, a custom harness — will not. Destructive tools are therefore two-phase server-side: the first call returns a preview and a token, the second executes. See [Destructive operations](#destructive-operations). Set `ALAYA_CONFIRM_MODE=off` to restore the older client-side-only contract.
**External systems are bridges, not mirrors.** The vault doesn't replicate GitLab/GitHub/Outline — it pulls context in and pushes ideas out.
## Tools
### Read & navigate
| Tool | Purpose |
|---|---|
| `get_note` | Read a note by path |
| `list_notes` | List notes filtered by directory, tag |
| `get_backlinks` | Notes that link to a given note |
| `get_links` | Outgoing links from a note |
| `get_tags` | All tags with counts |
| `vault_stats` | Vault overview: note counts, directories, top tags |
| `vault_graph` | Wikilink graph: orphans, hubs, topology |
| `vault_health` | Index health, embedding model status, migration progress |
### Write & capture
| Tool | Purpose |
|---|---|
| `create_note` | Create a new note with frontmatter and tags |
| `append_to_note` | Append text to an existing note (with optional section targeting) |
| `update_tags` | Add or remove inline #tags |
| `smart_capture` | Auto-route thoughts to the right note (person/daily/topic/inbox) |
| `capture_to_inbox` | Timestamped quick capture to inbox.md |
| `get_inbox` | Read inbox contents |
| `clear_inbox_item` | Remove a processed inbox item |
### Search
| Tool | Purpose |
|---|---|
| `search_notes` | Hybrid semantic + keyword search with adaptive routing, corrective RAG, and optional reranking/graph expansion/HyDE |
Notes carrying frontmatter `type: agent-memory` are excluded from `search_notes`
by default so agent scratch memory cannot crowd out curated notes. Pass
`include_types=["agent-memory"]` to search them explicitly.
### Agent memory
For agent harnesses rather than humans. A loop iteration needs budgeted
retrieval in one call, somewhere durable to put a summary when its context is
compacted, and a way to rebuild working memory from a fresh context.
| Tool | Purpose |
|---|---|
| `memory_recall` | Retrieve vault context for a query, trimmed to a token budget. Returns chunk text directly — no follow-up `get_note` per hit |
| `memory_checkpoint` | Persist a session summary, decisions, open items, and entities to `agents/<session>/checkpoint.md` as `type: agent-memory` |
| `memory_resume` | Return the latest checkpoint plus its 1-hop linked notes, budgeted — the handoff document a fresh iteration starts from |
Every response ends with a `<!-- token_count: N -->` comment so a caller can do
its own budget accounting. Entities passed to `memory_checkpoint` are written as
wikilinks, so checkpoints join the note graph and graph RAG can reach them.
Token counting uses tiktoken's `cl100k` encoding when the `tokenizer` extra is
installed, and a word-based approximation otherwise.
### Tool profiles
alaya registers ~35 tools. For smaller local models the JSON schemas are a real
fraction of the context window, and the size of the decision surface hurts
tool-call accuracy. A profile trims the exposed set at startup, so every client
benefits rather than each having to filter for itself.
| Profile | Tools |
|---|---|
| `librarian` | Everything. The default — existing clients see no change |
| `orchestrator` | Memory, search, and capture: `memory_*`, `search_notes`, `get_note`, `smart_capture`, `append_to_note`, `get_todos` |
| `readonly` | Read the vault, never change it: search, navigation, stats, health, `memory_recall` |
```bash
alaya --profile orchestrator
# or
ALAYA_TOOL_PROFILE=readonly alaya
```
Define your own under `[profiles]` in `alaya.toml`; a profile of the same name
overrides a built-in. A profile naming a tool that does not exist is a startup
error rather than a silently smaller tool list.
### Output budget
Every tool response ends with a `<!-- token_count: N -->` comment, so an agent
can account for what a call cost without guessing.
No response exceeds `ALAYA_MAX_TOOL_OUTPUT_TOKENS` (default 4000). A truncated
response says how much was dropped and how to get it:
```
…[+2140 tokens truncated; narrow the query, lower limit, or pass unbounded=True]
```
`get_note` pages instead of just truncating. `offset` is a 0-based body line and
`limit` is lines per page; the footer names the line to resume from, so
successive calls reconstruct the whole note:
```
get_note(path="projects/big.md")
-> **Lines:** 0-84 of 300
…
…[+216 line(s) truncated; get_note(path="projects/big.md", offset=84)]
```
`get_note`, `list_notes`, `search_notes`, `get_tags`, `get_inbox`, `get_todos`,
`find_references`, and `vault_graph` accept `unbounded=True` to bypass the cap.
### Destructive operations
`delete_note`, `rename_note`, `move_note`, `replace_section`, `reindex_vault`,
and `clear_inbox_item` are two-phase. The first call returns a preview of what
would change plus a `confirm_token`; the second call, with that token, executes.
```
delete_note(path="projects/old.md")
-> CONFIRM REQUIRED — delete_note_tool
Archive `projects/old.md` -> `archives/old.md` (soft delete, the file is moved not erased)
2 note(s) link to this one and will be left with broken links: …
confirm_token="1757193042.a1b2…"
delete_note(path="projects/old.md", confirm_token="1757193042.a1b2…")
-> archives/old.md
```
The token is an HMAC over the tool name, the arguments, and an expiry, keyed by a
secret generated per process. It cannot be replayed against different arguments,
expires after `ALAYA_CONFIRM_TTL` seconds, and does not survive a restart. The
propose and execute entries share a `confirm_id` in the audit log.
`rename_note`'s preview lists every note whose wikilinks would be rewritten.
| Mode | Behaviour |
|---|---|
| `required` (default) | Preview first, token required to execute |
| `optional` | Executes directly, but a supplied token is still validated |
| `off` | Previous behaviour — confirmation is the client's job |
### Edit & structure
| Tool | Purpose |
|---|---|
| `replace_section` | Replace content of a named `##` section |
| `extract_section` | Extract a section into a new note, leave wikilink |
| `move_note` | Move note to a different directory |
| `rename_note` | Rename note, update all wikilinks vault-wide |
| `delete_note` | Soft-delete to archives/ |
| `find_references` | Find wikilinks and text mentions of a title |
| `get_todos` | Find all `- [ ]` tasks across the vault |
| `complete_todo` | Mark a task complete (fuzzy line fallback) |
### Ingest & external
| Tool | Purpose |
|---|---|
| `ingest` | Ingest a URL, PDF, or markdown file into the vault + index |
| `batch_ingest` | Batch ingest multiple URLs/files |
| `reindex_vault` | Full LanceDB rebuild (requires confirm) |
| `pull_external` | Pull issues/docs from GitLab, GitHub, or Outline |
| `push_external` | Push a vault note to an external provider |
External issue CRUD (list, close, update) is handled by Claude Code natively via `glab`/`gh` CLI — the MCP server only bridges the vault to external systems.
## Index Pipeline
### Chunking strategies
| Strategy | Trigger | How it works |
|----------|---------|-------------|
| `DailyNoteChunker` | Notes in `daily/` | Splits on `###` sub-headers |
| `SectionChunker` | Notes with `##` headers | Splits on `##`, sub-splits long sections on paragraphs |
| `SemanticChunker` | Default for flat notes | Paragraph-aware splitting with code block preservation |
| `SlidingWindowChunker` | Legacy fallback | Fixed token window with overlap |
### Embedding
- **Model:** nomic-embed-text-v1.5 via fastembed (ONNX, no PyTorch required)
- **Dimensions:** 768
- **Prefixes:** `search_query:` for queries, `search_document:` for chunks
- **Quantized variant:** `nomic-v1.5-q4` available via `ALAYA_EMBEDDING_MODEL` env var
- **Hot-swap:** Changing the model triggers automatic background re-embedding
### Write-through indexing
All write operations (create, append, edit, move, rename, delete) emit events that trigger immediate index updates. The file watcher catches external edits with 2-second debounce. Double-indexing is prevented via coordination between the event system and watcher. A `VaultMetadataCache` keeps frontmatter, tags, and wikilinks in memory for O(1) lookups — automatically invalidated by both the event system and file watcher.
### Audit logging
Every tool call is logged to `audit.jsonl` in the vault data directory (`.zk/` or `.obsidian/`) with timestamp, tool name, arguments, status, duration, and result summary. The TUI dashboard tails this file for real-time activity monitoring.
## Vault structure
```
~/notes/
├── daily/ # daily notes — connective tissue
├── inbox.md # frictionless capture, process weekly
├── projects/ # active work with a finish line
├── areas/ # ongoing responsibilities
├── people/ # 1:1s, context, relationships
├── ideas/ # not ready to be a project yet
├── learning/ # active study
├── resources/ # settled reference, distilled knowledge
├── raw/ # drop zone for PDFs and URLs (auto-ingested)
└── archives/ # completed or dead things (soft-delete target)
```
Directory names are configurable via `alaya.toml` (see Configuration below). Notes use minimal YAML frontmatter (`title` + `date`) and inline `#tags`. Links are wikilinks (`[[title]]`).
## Stack
| Component | Choice |
|---|---|
| Language | Python 3.12+ |
| Package manager | uv |
| MCP framework | FastMCP |
| Vault backend | zk CLI or Obsidian (pure Python) |
| Vector store | LanceDB (local, Apache Arrow) |
| Embeddings | nomic-embed-text-v1.5 (ONNX, no PyTorch) |
| Search | Hybrid vector + BM25 via LanceDB with RRF reranking |
| Reranking | cross-encoder/ms-marco-MiniLM-L-6-v2 (optional) |
| PDF extraction | pymupdf4llm |
| Web extraction | trafilatura |
| File watching | watchdog |
| HTTP client | httpx |
| External bridge | glab CLI (GitLab) / gh CLI (GitHub) / Outline API |
## Configuration
### Prerequisites
- Python 3.12+
- [uv](https://docs.astral.sh/uv/) — `brew install uv`
- [zk](https://github.com/zk-org/zk) — `brew install zk` (optional, only needed for zk vaults)
- [glab](https://gitlab.com/gitlab-org/cli) — `brew install glab` (optional, for GitLab bridge)
- [gh](https://cli.github.com/) — `brew install gh` (optional, for GitHub bridge)
### Environment variables
| Variable | Required | Description |
|---|---|---|
| `ALAYA_VAULT_DIR` | Yes | Path to your vault (e.g. `~/notes`) |
| `ZK_NOTEBOOK_DIR` | Compat | Backward-compatible alias for `ALAYA_VAULT_DIR` |
| `ALAYA_EMBEDDING_MODEL` | No | Embedding model variant (`nomic-v1.5` or `nomic-v1.5-q4`, default: `nomic-v1.5`) |
| `ALAYA_TOOL_PROFILE` | No | Limit which tools are exposed (`librarian`, `orchestrator`, `readonly`, or a profile from `alaya.toml`; default: `librarian`) |
| `ALAYA_CONFIRM_MODE` | No | Server-side confirmation for destructive tools: `required` (default), `optional`, `off` |
| `ALAYA_CONFIRM_TTL` | No | Seconds a confirm token stays valid (default: `60`) |
| `ALAYA_MAX_TOOL_OUTPUT_TOKENS` | No | Cap on tool output in tokens; `0` disables capping (default: `4000`) |
| `ALAYA_TOKENIZER` | No | tiktoken encoding used for token counts (default: `cl100k_base`) |
| `GITLAB_PROJECT` | No | GitLab project path — enables GitLab provider |
| `GITLAB_DEFAULT_LABELS` | No | Comma-separated default labels for new issues |
| `GITHUB_REPO` | No | GitHub repo (e.g. `owner/repo`) — enables GitHub provider |
| `GITHUB_DEFAULT_LABELS` | No | Comma-separated default labels for new issues |
| `OUTLINE_URL` | No | Outline instance URL — enables Outline provider |
| `OUTLINE_API_KEY` | No | Outline API key |
### `alaya.toml` (optional)
Place an `alaya.toml` in your vault root to override defaults:
```toml
[vault]
type = "obsidian" # or "zk" — overrides auto-detection
[directories]
person = "people"
idea = "ideas"
project = "projects"
learning = "learning"
resource = "resources"
daily = "daily"
agent = "agents" # where memory_checkpoint writes
[settings]
archives_dir = "archives"
default_capture_dir = "inbox"
default_external_dir = "external"
[profiles]
# Custom tool profiles. "*" means every tool. Names may be written with or
# without the `_tool` suffix. A profile of the same name overrides a built-in.
planner = ["memory_recall", "memory_resume", "search_notes", "get_note"]
```
All fields are optional. Without this file, alaya auto-detects the backend from `.zk/` or `.obsidian/` and uses sensible defaults.
### Optional dependencies
```bash
# Install cross-encoder reranking support
uv sync --extra rerank
# Install tiktoken for exact token counting in the agent memory tools
uv sync --extra tokenizer
```
Pass additional env vars when registering with Claude Code using `-e`:
```bash
claude mcp add alaya \
-e ALAYA_VAULT_DIR=$HOME/notes \
-e GITHUB_REPO=owner/repo \
-- uvx alaya-mcp
```
Configure any combination of providers. `pull_external` and `push_external` auto-detect the provider from URLs or use the configured defaults.
## Development
```bash
make install # install dependencies
make test # run unit tests (607+ tests)
make test-unit # run unit tests verbose
make test-integration # run integration tests (requires zk binary)
make lint # ruff check
make serve # start the server
```
### Project structure
```
src/alaya/
├── server.py # FastMCP server, tool registration, audit wrapping
├── config.py # env var loading, vault validation
├── vault.py # path safety, frontmatter parsing
├── cache.py # VaultMetadataCache: in-memory metadata index, O(1) lookups
├── zk.py # zk CLI subprocess wrapper
├── events.py # pub/sub event bus for write-through indexing
├── watcher.py # watchdog file system monitor + cache invalidation
├── errors.py # structured error codes
├── audit.py # JSONL tool call logging
├── backend/
│ ├── protocol.py # VaultBackend protocol + VaultConfig
│ ├── config.py # vault type detection, alaya.toml loading
│ ├── zk.py # ZkBackend (delegates to zk CLI)
│ └── obsidian.py # ObsidianBackend (pure Python)
├── tools/
│ ├── read.py # get_note, list_notes, backlinks, links, tags, reindex
│ ├── write.py # create_note, append_to_note, update_tags
│ ├── inbox.py # capture, get_inbox, clear_inbox_item
│ ├── capture.py # smart_capture (auto-routing)
│ ├── search.py # adaptive search with corrective RAG pipeline
│ ├── structure.py # move, rename, delete, find_references
│ ├── edit.py # replace_section, extract_section
│ ├── tasks.py # get_todos, complete_todo
│ ├── external.py # pull_external, push_external
│ ├── ingest.py # URL/PDF/markdown ingestion
│ ├── stats.py # vault_stats
│ ├── graph.py # vault_graph (wikilink topology)
│ └── enrich.py # LLM-powered enrichment (person cache, suggestions)
├── providers/
│ ├── gitlab.py # glab CLI wrapper
│ └── github.py # gh CLI wrapper
└── index/
├── embedder.py # chunk notes, embed via nomic ONNX (LRU-cached queries)
├── store.py # LanceDB: upsert, delete, hybrid/keyword/vector search
├── router.py # adaptive query classification
├── corrective.py # retrieval quality check + query reformulation
├── graph_rag.py # wikilink-based search expansion (cache-aware)
├── hyde.py # hypothetical document embeddings
├── contextual.py # chunk context prepending
├── late_chunking.py # full-doc embedding with per-chunk pooling
├── chunking.py # pluggable chunking strategies
├── models.py # embedding model registry
├── reindex.py # full/incremental rebuild
└── health.py # index health tracking
```
### Testing approach
- **Unit tests** mock `alaya.zk.run_zk` (the subprocess boundary) and the embedding model. No subprocess calls, no model loading. Obsidian tests use real filesystem fixtures.
- **Wiring tests** prove each tool dispatches through the backend by patching `run_zk` with `AssertionError` side effects.
- **Integration tests** use the real `zk` binary and LanceDB against `vault_fixture/`.
- Tests use a copy of `vault_fixture/` in a temp directory — vault is never modified in place.
- TDD: failing tests written before implementation for every tool.
## Companion: alaya-tui
[alaya-tui](https://github.com/luke-kucing/alaya-tui) is a Go + Bubble Tea terminal dashboard for vault observability and agent-agnostic chat. It shows vault health, tails the audit log, browses notes, and spawns any configured LLM agent as a subprocess.
## Status
All 5 milestones implemented plus advanced RAG pipeline, pluggable vault backend (zk + Obsidian), and in-memory metadata cache. 607+ unit tests passing. See [open issues](https://github.com/luke-kucing/alaya/issues) for planned improvements.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessUnresponsive