Skip to main content
Glama
README.md
# Trove: Knowledge Engine for AI Development

A local-first knowledge engine with hybrid search, source connectors, plugin extensibility, and MCP integration for Claude Code. Trove discovers, indexes, and searches your documents, code, and web content so AI tools always have the right context.

## Features

### Search & Retrieval
- **Hybrid Search**: BM25 keyword + vector semantic search with cross-encoder re-ranking
- **RRF Merge Strategy**: Reciprocal Rank Fusion as alternative to weighted merging
- **MMR Diversity**: Maximal Marginal Relevance reduces redundant results
- **Page Diversity**: At most two chunks per page (a print copy counts as its page) come before other pages' results
- **Heading Match**: A result whose page title or section heading holds the query's words gets a small rerank bonus, so short official sections are found by their topic words
- **HyDE**: Hypothetical Document Embeddings for better vector search on vague queries
- **Multi-Query / RAG-Fusion**: Generate query variants and merge results via RRF
- **Search Quality Indicators**: Scores, timing, source diversity, and strategy metadata
- **Configurable Embeddings**: Choose embedding model per client (MiniLM, BGE, E5)
- **Stemming & Synonyms**: Porter/Snowball stemmer and domain-specific synonyms
- **Agentic/Recursive Search**: Automatically refines queries when results are insufficient
- **Reference Resolution**: Detects "see Appendix X" and fetches referenced content
- **Query Contextualization**: Understands follow-up questions in conversation context
- **Search Result Caching**: LRU + TTL cache with filter-aware keys
- **Score Normalization**: MinMax, Z-score, or raw normalization for merge strategies

### Source Connectors
- **GitHub Connector**: Discover repos from GitHub accounts/orgs, clone/fetch, track changed files. API client with rate limiting, pagination, and retries. Tokens via environment variables only.
- **Web Crawler**: BFS crawl with scope control, HTML-to-Markdown conversion (markdownify + trafilatura), incremental cache via ETag/content hash.
- **Auto-sync**: Sources sync before ingestion when `sync_on_ingest` is configured. Manual sync via CLI or MCP.
- **Incremental Updates**: Only changed files are re-crawled and reindexed.

### Plugin System
- **6 Extension Points**: Interfaces for embedding models, chunkers, parsers, rerankers, LLM providers, and merge strategies
- **Plugin Registry**: Register and look up plugins by name with duplicate detection
- **Plugin Loader**: Discovers `.py` files and packages in a `plugins/` folder and calls their `register()`. Trove does not call the loader at start-up, so plugins are not loaded automatically; see [Known limits](#known-limits)
- **Built-in Implementations**: The LLM providers and rerankers implement the plugin interfaces

### Ingestion & Indexing
- **Producer-Consumer Pipeline**: Parallel file parsing (ThreadPool) → GPU batch embedding → progressive ChromaDB indexing
- **Fast File Collection**: `git ls-files` for git repos (near-instant), `os.scandir` fallback, parallel change detection
- **Ctrl+C Safe**: Each GPU batch is indexed immediately; interrupt loses at most one batch, resume skips already-indexed files
- **Code Indexing**: Tree-sitter AST parsing for Python, TypeScript, JavaScript, PHP, HTML, CSS, YAML, JSON, Gherkin, Dart
- **Markdown Sections**: Markdown is split at its headings, with frontmatter and the heading path stored on each chunk (see [Using Trove with a scraper](#using-trove-with-a-scraper))
- **Chunking Strategies**: Title-based, semantic (split where embedding similarity drops) and recursive (priority-ordered separators) chunkers in `core/chunking.py`. Ingestion uses the heading splitter for Markdown, the title chunker for other parsed documents, its own email splitter for `.mbox` and the tree-sitter code chunker for code files; see [Known limits](#known-limits)
- **Contextual Embeddings**: Prepend document context to chunks before embedding
- **Chunk Quality Validation**: Min-length, alphanumeric, and content-hash deduplication

### Response Generation
- **Inline Citations**: LLM responses cite sources with `[1]`, `[2]` references
- **Configurable Reranker**: Local CrossEncoder or API-based (Cohere-compatible) with automatic fallback
- **Token Budgeting**: Separate context budgets for document, code, and reference chunks
- **Context Pre-Filtering**: Drop low-scoring chunks before LLM context

### Knowledge & Agents
- **Knowledge Graph**: Entity/relationship extraction, graph-augmented search (local/global/hybrid/mix modes)
- **Persistent Memory**: SQLite-backed store with vector search, TTL expiration, per-client or global
- **Workflow DAGs**: Multi-step task pipelines with dependency ordering and auto-status tracking
- **Agent Templates**: YAML-defined personas (code-reviewer, security-reviewer, etc.)
- **Local Agent Runner**: Search for context, inject system prompt, generate via LM Studio
- **Prompt Discovery** (optional): search a local prompt index and the live prompts.chat API; needs a prompts.chat client that is not included, see [Prompt library](#prompt-library-optional)

## Stack

| Layer | Library | License |
|-------|---------|---------|
| Document Processing | Unstructured.io | Apache 2.0 |
| Keyword Search | rank-bm25 | Apache 2.0 |
| Stemming/NLP | NLTK | Apache 2.0 |
| Vector Search | ChromaDB + sentence-transformers | Apache 2.0 |
| Re-ranking | sentence-transformers (cross-encoder) or API | Apache 2.0 |
| Knowledge Graph | NetworkX | BSD |
| GitHub API | httpx | BSD |
| HTML → Markdown | markdownify + trafilatura | MIT / Apache 2.0 |
| LLM (local) | LM Studio via OpenAI-compatible API | - |
| LLM (cloud) | Claude via Anthropic client | - |
| MCP Server | mcp | MIT |

**100% local capable. Zero API costs (with LM Studio).** Licensed under [BSL 1.1](LICENSE) (converts to MIT).

## Quick Start

Trove supports Python 3.11, 3.12 and 3.13. Code indexing needs tree-sitter-languages, which has no wheels for 3.13, so on 3.13 everything works except code indexing.

```bash
# Get the code and install dependencies
git clone https://github.com/tomgould/trove.git
cd trove
python -m venv .venv && source .venv/bin/activate

# Install the CPU build of PyTorch first. Without this, pip pulls the much
# larger CUDA build. Skip it if you want GPU support.
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

# The full stack: document parsers, LLM clients, MCP server, code indexing
pip install -r requirements.txt

# Create a new client (interactive wizard)
python scripts/create_client.py

# Option A: Add local documents
cp /path/to/docs/* clients/myproject/documents/

# Option B: Connect external sources
python scripts/sources.py connect github --client myproject --owner myorg
python scripts/sources.py crawl --client myproject --url https://docs.example.com

# Ingest and index
python scripts/ingest.py --client myproject

# Use via MCP server (see MCP Server section below)
python scripts/mcp_server.py
```

### Reproducible installs

`requirements.txt` and `pyproject.toml` give tested version ranges. For the exact versions CI tests with, add the constraints file for your Python version:

```bash
pip install -r requirements.txt -c constraints/py312.txt   # or py311.txt, py313.txt
```

`constraints/py311.txt` needs Python 3.11.5 or later, because one pinned package (soupsieve 4.0.1) does. On an earlier 3.11 release, install without the constraints file.

### Installing as a package

Trove also installs as a package, with extras so you only pull what you use:

| Extra | Adds |
|-------|------|
| (none) | Hybrid search, ingestion of text, HTML and Markdown, source connectors, knowledge graph store |
| `docs` | PDF, Word, Excel, PowerPoint and the other formats unstructured can parse |
| `llm` | The OpenAI-compatible (LM Studio) and Anthropic clients, for agents, graph extraction and generated answers |
| `mcp` | The MCP server |
| `code` | Code indexing with tree-sitter (Python 3.11 and 3.12) |
| `dev` | pytest, ruff and build |
| `all` | `docs`, `llm`, `mcp` and `code`: the same stack as `requirements.txt` |

```bash
pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
pip install ".[mcp,docs]"
```

An installed copy provides the commands `trove-server`, `trove-search`, `trove-ingest`, `trove-sources` and `trove-create-client`. They look for clients in a `clients/` folder in the current directory. Run from a checkout, the scripts in `scripts/` use the checkout's own `clients/` folder. `TROVE_CLIENTS_DIR` and `--clients-dir` override both, see [Where clients live](#where-clients-live).

## Creating a Client

The easiest way to create a new client is with the interactive wizard:

```bash
python scripts/create_client.py
```

The wizard will prompt you for:

- **Client name** - Display name (e.g., "Acme Corp RAG")
- **Machine name** - Directory name (auto-generated from display name)
- **Description** - Optional description
- **Embedding model** - MiniLM (fast) or BGE (better quality)
- **LLM settings** - Provider (lmstudio/claude), context limits, max tokens
- **Search settings** - BM25/vector weights, top_k, rerank settings (optional)
- **Tokenization** - Stemmer selection (Porter, Snowball, or none)
- **Reference domain** - Compliance, legal, technical, or academic patterns
- **Audit settings** - Enable/disable logging, log level
- **System prompt** - Custom prompt or use default

After completion, it creates the full directory structure and config files:

```
clients/your_client/
├── client.yaml       # Configuration
├── documents/        # Add your source files here
├── index/            # Search index (created on ingest)
└── prompts/          # System prompt
    └── system.txt
```

### Without the wizard

Scripts and pipelines can create a client in one command:

```bash
python scripts/create_client.py --name news-site --description "Scraped news" \
    --documents-dir /data/scrapes/news-site --embedding-model all-MiniLM-L6-v2
```

Only `--name` is required. A name is 3 to 64 letters, digits, `_` or `-`, starting and ending with a letter or digit. The command writes a minimal `client.yaml` (everything else uses the defaults), creates `index/` and `prompts/` (and `documents/` unless `--documents-dir` is given), and refuses to overwrite an existing `client.yaml` unless you pass `--force`. With no arguments it starts the wizard.

### Manual Setup

Alternatively, create a client manually:

```bash
mkdir -p clients/myproject/{documents,index,prompts,data}
```

Then create `clients/myproject/client.yaml` (see Client Configuration below).

## How It Works

### 1. Document Ingestion

```
PDF/DOCX/XLSX/CSV/MBOX/MD → Unstructured → Chunks with metadata → Vector DB + BM25 Index
```

Supported file types (the default `documents.extensions` list):
- **PDF** - Text extraction with tables, lists preserved
- **Word** - DOCX, DOC
- **Spreadsheets** - XLSX, XLS, CSV
- **Email archives** - MBOX (Gmail exports, etc.)
- **Markdown** - `.md`, split at headings by Trove's own Markdown splitter instead of Unstructured. `.markdown` files are handled the same way once `.markdown` is added to `documents.extensions`
- **HTML** - Web pages
- **Plain text** - TXT files

Features:
- Document structure preserved (headings, sections)
- Chunking by title for parsed documents, by heading for Markdown
- Email metadata preserved (subject, from, to, date)
- Recursive directory scanning (all subdirectories)

### 2. Search (Hybrid + Agentic)

```
Query → Contextualize (if follow-up)
      → BM25 (keyword) + Vector (semantic) in parallel
      → Merge (weighted or RRF)
      → Cross-encoder re-rank (adaptive depth) + heading match bonus
      → MMR diversity (if enabled)
      → Reference resolution (fetch appendices/sections)
      → [If recursive: analyze → retry with better query]
      → Data-first ordering (tiers, then rerank score)
      → Page diversity (at most 2 chunks per page before other pages)
      → Final results + quality indicators
```

- BM25 finds exact matches ("Appendix C" → actual Appendix C)
- Vector finds semantic matches ("equivalent jurisdictions" → related content)
- Merge via weighted scores (default) or RRF rank fusion
- Cross-encoder picks the best results. Adaptive depth reranks only the top 30 candidates (`adaptive_rerank_depth`) when the same chunk tops both the keyword and the vector list, and never fewer than `top_k`
- Heading match: each reranked chunk gets `heading_boost` (default 1.0) times the share of the query's words found in its page title or section path, with stop words ignored and plurals matched loosely. Chunks with no title or section get nothing
- `top_k` is honoured up to 200 results: the candidate pool (`rerank_top_n`) grows to `top_k` when it is smaller
- MMR reduces redundancy by penalizing similarity to already-selected chunks
- Reference resolver fetches content when chunks say "see Appendix X"

The last step sorts results into tiers and orders by rerank score within each tier:

1. Spreadsheets (`.xlsx`, `.xls`, `.csv`) whose relative path contains `reference`, `list`, `countries` or `data` (a folder name counts, such as `data/`)
2. Other spreadsheets, and non-spreadsheet files whose path contains `reference` or `list`
3. Spreadsheets whose relative path contains `form`, `assessment` or `template`, and non-Markdown chunks that look like data: more than 10 non-blank lines, more than 5 commas, or the text "high" anywhere in the chunk together with one of a few country names hard-coded in `data_tier`
4. Everything else
5. Chunks outside tiers 1 to 3 that contain a pointer phrase ("see appendix", "refer to appendix", "listed in appendix", "detailed in appendix" or "as per appendix"), even alongside other real content. They rank below every tier-4 chunk

Chunks from the Markdown heading splitter are never ranked as data: whatever their path or shape, they rank as tier 4, or tier 5 when they contain a pointer phrase. The tiers come from `data_tier` in `core/search/searcher.py`.

After the tiers, page diversity keeps at most `max_chunks_per_source` (default 2) chunks of one page ahead of other pages. A page is a chunk's `url`, or its source file when it has none. A file whose frontmatter `canonical_url` is another indexed file's `url` on the same host (a print copy, for example) counts as that page; URLs compare without scheme, fragment or trailing slash, with the host in any case and default ports ignored. A `canonical_url` on another host is ignored. The page's further chunks are moved, in order, behind every reranked chunk from other pages rather than dropped. They reach the results only when fewer than `top_k` chunks from other pages remain after `min_score` filtering, so with `min_score: null` a page usually shows at most `max_chunks_per_source` chunks even when it is the only page that matches. Set `max_chunks_per_source: 0` (or `null`) in `search:` to turn it off.

Null `top_k`, `rerank_top_n`, `adaptive_rerank_depth` or `heading_boost` in `search:` means the default. A value of the wrong kind (text, a fraction where a whole number is needed, a number below the minimum, or `.nan`, `.inf` or `-.inf`) stops `search.py`, `evaluate.py`, `run_agent.py` and the MCP server with an error naming the setting.

### 3. Response Generation

```
Top chunks → Context filter (drop low-scoring)
           → Token budgeting (doc/code/reference allocations)
           → LLM prompt with citation instructions
           → Answer with inline [1] [2] citations
           → Citation footer + Follow-up suggestions
```

## Project Structure

```
Trove/
├── core/
│   ├── data/agents/          # Agent YAML templates (shipped in the package)
│   ├── paths.py              # Where the clients and agents folders are
│   ├── ingest.py             # Document processing with Unstructured, file manifest
│   ├── markdown.py           # Markdown frontmatter, image stripping, heading splitter
│   ├── pipeline.py           # Producer-consumer ingestion pipeline
│   ├── model_cache.py        # Keeps embedding models and rerankers loaded, shared across clients
│   ├── resource_manager.py   # Probes CPU, RAM and GPU to pick concurrency settings
│   ├── health.py             # Circuit breaker for the LM Studio connection
│   ├── search/               # Hybrid search package
│   │   ├── types.py          # SearchQuality, SearchResult dataclasses
│   │   ├── tokenizer.py      # EnhancedTokenizer, stemming, stop words
│   │   ├── cache.py          # Search result LRU+TTL cache
│   │   ├── merge.py          # Merging results across indexes and across clients (RRF)
│   │   ├── chroma_paging.py  # Reads a ChromaDB collection in pages (a full get() fails past 32,000 rows)
│   │   └── searcher.py       # HybridSearcher (BM25 + vector + re-ranking + recursive)
│   ├── llm.py                # LLM providers (LM Studio, Claude)
│   ├── client.py             # Client configuration + validation
│   ├── refinement.py         # Query refinement & contextualization
│   ├── reference_resolver.py # Appendix/section reference resolution
│   ├── reference_config.py   # Configurable reference patterns per domain
│   ├── logging_config.py     # Audit configuration loading
│   ├── reranker.py           # Reranker backends (local, API, fallback)
│   ├── evaluation.py         # Retrieval metrics + LLM-as-judge RAG quality
│   ├── chunking.py           # Chunking strategies (title, semantic, recursive)
│   ├── ingest_progress.py    # Progress tracking and resumable ingestion
│   ├── ingest_job.py         # Ingestion job management for MCP tools
│   ├── memory.py             # Persistent memory store (SQLite + vector search)
│   ├── workflow.py           # Workflow DAG engine (task pipelines)
│   ├── agents.py             # Agent template loader (YAML configs)
│   ├── runner.py             # Local agent runner (search → RAG → LLM generate)
│   ├── plugins/              # Plugin system
│   │   ├── interfaces.py     # ABCs: BaseEmbedder, BaseChunker, BaseParser, etc.
│   │   ├── registry.py       # Plugin registry (register/lookup by name)
│   │   └── loader.py         # Auto-discovery from plugins/ directory
│   ├── sources/              # Source connectors
│   │   ├── base.py           # BaseSourceConnector ABC, SyncResult
│   │   ├── github_connector.py # GitHub repo discovery, clone, sync
│   │   ├── github_client.py  # GitHub API client (rate limiting, pagination)
│   │   ├── github_models.py  # GitHubSourceConfig, RepoInfo
│   │   ├── git_ops.py        # Git clone/fetch with token auth
│   │   ├── web_crawler.py    # BFS web crawler with scope control
│   │   ├── web_converter.py  # HTML → clean Markdown conversion
│   │   └── web_cache.py      # Incremental crawling cache (ETag/content hash)
│   ├── code/                 # Code indexing subsystem
│   │   ├── chunker.py        # Code file chunking orchestration
│   │   ├── parser.py         # Tree-sitter parser management
│   │   ├── git_utils.py      # Git repository detection
│   │   ├── incremental.py    # Incremental indexing state
│   │   └── extractors/       # Language-specific extractors
│   └── graph/                # Knowledge graph subsystem
│       ├── models.py         # Entity, Relationship dataclasses
│       ├── store.py          # KnowledgeGraph (NetworkX + SQLite)
│       ├── extractor.py      # LLM-based entity/relationship extraction
│       ├── progress.py       # Graph build progress tracking
│       ├── search.py         # Graph search modes
│       └── export.py         # Stats and D3.js export
├── plugins/                   # User plugin directory (drop .py files here)
├── mcp_server/
│   ├── server.py             # MCP server entry point, tool definitions, dispatcher
│   ├── shared.py             # Shared state, cache, helpers
│   └── handlers/             # Tool handler modules (clients, search, ingest, graph, memory, workflows, agents, prompts, sources)
├── clients/
│   ├── {name}/
│   │   ├── client.yaml       # Config (including sources: section)
│   │   ├── documents/        # Local source files
│   │   ├── sources/          # Auto-managed source directories
│   │   │   ├── github/       # Cloned git repos
│   │   │   └── web/          # Crawled markdown files
│   │   ├── index/            # Search indices
│   │   ├── prompts/          # System prompts
│   │   └── tests/            # Evaluation datasets
├── scripts/
│   ├── create_client.py      # Interactive client creation wizard
│   ├── ingest.py             # CLI for document ingestion
│   ├── search.py             # CLI search tool
│   ├── evaluate.py           # CLI evaluation tool
│   ├── build_graph.py        # CLI for building knowledge graphs
│   ├── sources.py            # CLI for source connectors (GitHub, web)
│   ├── mcp_server.py         # MCP server launch script
│   ├── run_agent.py          # CLI for running agents locally
│   ├── benchmark_embeddings.py # Compare embedding models on an eval dataset
│   ├── watch_and_reindex.py  # Source sync + file watcher
│   ├── sync_prompts.py       # Sync prompts from prompts.chat API (needs the optional client)
│   └── list_docs.sh          # Print an inventory of the office documents and PDFs in a folder
├── overnight-rebuild.sh      # Reindex and rebuild graphs for a list of clients
├── constraints/              # Exact versions from a working install, per Python version
└── tests/
    ├── conftest.py           # Shared fixtures
    ├── test_*.py             # One file per area: client, ingest, search, ranking, graph, ...
    └── smoke/                # Install checks for a built wheel
```

## Client Configuration

```yaml
# clients/example/client.yaml
name: "My RAG System"
machine_name: example

paths:
  documents: "documents"
  index: "index"
  prompts: "prompts"

# File types to ingest. The list replaces the default list, so name every type you want
documents:
  extensions: [".pdf", ".docx", ".doc", ".xlsx", ".xls", ".csv", ".html", ".txt", ".md", ".mbox"]
  exclude_extensions: []           # e.g. [".csv", ".xlsx"] to skip spreadsheets

# Embedding model settings
embedding:
  model: "all-MiniLM-L6-v2"  # or "BAAI/bge-base-en-v1.5" for better quality
  dimension: 384             # 384 for MiniLM, 768 for BGE
  query_prefix: ""           # Use "query: " for E5 models
  passage_prefix: ""         # Use "passage: " for E5 models
  contextualize: false       # Prepend doc context to chunks before embedding (requires reindex)
  hnsw_ef_construction: 200  # HNSW build-time neighbor list (requires reindex)
  hnsw_max_neighbors: 16     # HNSW max connections per node (requires reindex)
  hnsw_ef_search: 100        # HNSW query-time neighbor list (no reindex needed)

search:
  bm25_weight: 0.5
  vector_weight: 0.5
  top_k: 20
  rerank_top_n: 50
  # Merge strategy: "weighted" (default) or "rrf" (rank fusion)
  merge_strategy: "weighted"
  rrf_k: 60                         # RRF constant (only used with merge_strategy: "rrf")
  # MMR diversity: null (disabled) or 0.0-1.0 (0.0=max diversity, 1.0=pure relevance)
  mmr_lambda: null
  # Adaptive reranking: reduce rerank depth on high-confidence matches
  adaptive_rerank: true
  adaptive_rerank_threshold: 0.9     # Combined score threshold to trigger
  adaptive_rerank_depth: 30          # Reduced depth when triggered (never below top_k)
  # Result diversity and heading match
  max_chunks_per_source: 2           # Chunks per page before other pages (0 or null = no limit)
  heading_boost: 1.0                 # Rerank bonus for query words in the title or section (0 = off)
  # BM25 tuning
  bm25_k1: 1.5                        # Term frequency saturation (>0)
  bm25_b: 0.75                        # Document length normalization (0-1)
  score_normalization: "minmax"        # "minmax", "zscore", or "raw"
  # Advanced query processing (requires LLM provider)
  hyde: false                          # Generate hypothetical doc for vector search
  multi_query: false                   # Generate query variants and merge via RRF
  multi_query_count: 3                 # Number of query variants (2-7)
  # Search result caching
  cache_ttl: 300                       # TTL in seconds (0 = disabled)
  cache_max_size: 128                  # Max cached queries per client
  # Reranker backend: "local" (CrossEncoder) or "api" (Cohere-compatible)
  reranker: "local"
  reranker_model: null               # Custom reranker model (null=default)
  reranker_endpoint: null            # API endpoint for "api" reranker
  # Tokenization settings for BM25
  tokenization:
    stemmer: "porter"  # "porter", "snowball", or null
    synonyms:          # Domain-specific synonyms
      aml: ["anti-money laundering", "financial crime"]
      kyc: ["know your customer"]

# Reference resolution settings
references:
  enabled: true
  mode: "extend"        # "extend" adds to domain defaults, "replace" uses only custom
  domain: "compliance"  # "compliance", "legal", "technical", or "academic"
  # Optional: custom patterns
  patterns:
    custom_ref:
      priority: high
      patterns:
        - '(?:see\s+)?REF-(\d+)'
      query_templates:
        - "Reference {id}"

llm:
  provider: "lmstudio"                                    # OpenAI-compatible (recommended for LM Studio)
  lmstudio_model: "mistralai/mistral-nemo-instruct-2407"  # must be set explicitly
  max_chunks: 10
  max_context_chars: 12000
  max_tokens: 2000
  citations: false                # Enable inline [N] citations in responses
  context_filter: false           # Drop low-scoring chunks before LLM context
  context_filter_threshold: 0.1   # Minimum rerank score for context inclusion
  doc_budget_ratio: 0.6           # Token budget share for document chunks
  code_budget_ratio: 0.3          # Token budget share for code chunks
  reference_budget_ratio: 0.1     # Token budget share for reference chunks

# Optional: Knowledge graph (entity/relationship extraction)
graph:
  enabled: false                # Opt-in per client
  entity_types:                 # Entity types to extract
    - person
    - organization
    - concept
    - location
  extract_batch_size: 10        # Chunks per LLM call
  max_description_chars: 200    # Truncate descriptions
  search_mode: "naive"          # Default search: naive, local, global, hybrid, mix

# Optional: Chunking strategy. Client.chunking_config returns these, but ingestion does
# not pass them to the chunkers (see Known limits): parsed documents use the title chunker
# and Markdown the heading splitter, both at 1500 characters with 100 overlap
chunking:
  strategy: "title"             # "title", "semantic", or "recursive"
  max_chunk_size: 1500
  min_chunk_size: 100
  overlap: 100                  # Overlap between chunks (recursive strategy)
  semantic_threshold: 0.5       # Cosine similarity for semantic split points
  separators: ["\n\n", "\n", ". ", " "]  # Priority-ordered separators (recursive strategy)

# Optional: Source connectors (auto-fetch from GitHub/web)
sources:
  github:
    - owner: myorg                  # GitHub user or org
      token_env: GITHUB_TOKEN       # Env var name (never store tokens in YAML)
      include_repos: [api, frontend] # Optional whitelist
      exclude_repos: [legacy-app]    # Optional blacklist
      sync_on_ingest: true           # Auto-sync before reindexing
    - repo: owner/repo              # Single repo shorthand
      token_env: GITHUB_TOKEN
  web:
    - url: https://docs.example.com
      stay_in_path: true            # Don't leave starting URL path
      max_pages: 500                # Safety limit
      sync_on_ingest: true

# Optional: Code indexing (tree-sitter AST parsing)
code:
  enabled: true
  paths: ["src/"]                   # Directories to index (relative to client root)
  extensions: [".py", ".ts", ".js"] # File extensions to include
  exclude_patterns: ["**/test_*", "**/__pycache__/**"]

# Optional: Code priority in search results (when docs may be outdated)
# Add to search: section
#   code_priority: true             # Boost code chunks above docs
#   code_priority_boost: 0.5        # Score boost for code chunks
#   cross_reference_code: true      # Auto-fetch related code for doc results

# Optional: Audit settings
audit:
  enabled: true
  log_level: "full"  # "full", "minimal", or "none"
  include_chunks: true
  include_scores: true
  include_timing: true
```

### Embedding Models

| Model | Dimensions | Speed | Quality | Notes |
|-------|------------|-------|---------|-------|
| `all-MiniLM-L6-v2` | 384 | Fast | Good | Default, good for most use cases |
| `BAAI/bge-base-en-v1.5` | 768 | Medium | Better | Better retrieval, recommended for production |
| `intfloat/e5-base-v2` | 768 | Medium | Better | Requires query/passage prefixes |

### Reference Domains

| Domain | Patterns Detected |
|--------|-------------------|
| `compliance` | Appendix, Annex, Schedule, Section, Table |
| `legal` | Exhibit, Clause, Article, Recital, Appendix, Schedule |
| `technical` | RFC, Issue #, PR #, Section |
| `academic` | Equation, Theorem, Lemma, Definition, Figure, Table, Chapter |

## CLI Commands

### Ingestion

```bash
# Ingest documents for a client
python scripts/ingest.py --client myproject

# Rebuild from client's documents directory
python scripts/ingest.py --client myproject --rebuild

# Incremental run for every client (add --rebuild for a full rebuild of each)
python scripts/ingest.py --all

# Every client, documents only (skip code)
python scripts/ingest.py --all --no-code

# Resume an interrupted ingestion (skips already-processed files)
python scripts/ingest.py --client myproject --rebuild --resume

# Force BM25 cache rebuild (use after changing tokenization settings)
python scripts/ingest.py --client myproject --rebuild --force-bm25-rebuild

# List available clients
python scripts/ingest.py --list

# Print a JSON summary of what changed (for scripts and schedulers)
python scripts/ingest.py --client myproject --json
```

The default run is incremental. Unchanged files are not parsed or embedded again, including files a scraper rewrote with identical content, because the manifest keeps a hash of each file. A file that fails is reported and retried on the next run. `--json` prints:

```json
{"client": "myproject", "mode": "incremental", "new": 2, "changed": 1, "unchanged": 40,
 "deleted": 0, "failed": 0, "failed_files": [], "chunks_added": 9, "chunks_removed": 3}
```

`mode` is `rebuild` on the first run (or with `--rebuild`), when every file counts as new. With `--all` the summaries are listed under `"clients"`.

### Search

```bash
# Basic search
python scripts/search.py --client myproject "your search query"

# Verbose output with timing and metadata
python scripts/search.py --client myproject --verbose "your query"

# Use RRF merge instead of weighted
python scripts/search.py --client myproject --merge-strategy rrf "your query"

# Enable MMR diversity (0.0=max diversity, 1.0=pure relevance)
python scripts/search.py --client myproject --mmr-lambda 0.7 "your query"

# Disable adaptive reranking
python scripts/search.py --client myproject --no-adaptive "your query"

# More results (at most 200; a larger number is lowered with a note)
python scripts/search.py --client myproject --top-k 50 "your query"

# JSON output (for scripting/comparison)
python scripts/search.py --client myproject --json "your query"

# Full content (no truncation)
python scripts/search.py --client myproject --full "your query"

# Search several clients, merged by rank, each result labelled with its client
python scripts/search.py --clients docs,blog "your query"
python scripts/search.py --all --json "your query"

# List available clients
python scripts/search.py --list
```

Results show the source file and, when the document has them, its title, URL and section (heading path). `--json` results carry `client`, `source`, `title`, `url` and `section` fields; `title`, `url` and `section` are `null` when unknown.

### Source Connectors

Connect external sources (GitHub repos, websites) to a client. Source files are stored in `clients/{name}/sources/` and indexed alongside `documents/`.

```bash
# Connect a GitHub org's repos to a client
python scripts/sources.py connect github --client myproject --owner myorg

# Connect a single repo
python scripts/sources.py connect github --client myproject --repo owner/repo

# Filter repos
python scripts/sources.py connect github --client myproject --owner myorg \
    --include api frontend --exclude legacy --public-only

# Crawl a website into a client
python scripts/sources.py crawl --client myproject --url https://docs.example.com

# Crawl with options
python scripts/sources.py crawl --client myproject --url https://api.example.com/docs \
    --max-pages 200 --wait-time 2.0 --no-stay-in-path

# Sync all configured sources for a client
python scripts/sources.py sync --client myproject

# Sync all clients at once
python scripts/sources.py sync --all

# List configured sources and on-disk state
python scripts/sources.py list --client myproject
```

After connecting sources, ingest them:

```bash
python scripts/ingest.py --client myproject --rebuild
```

Sources with `sync_on_ingest: true` in `client.yaml` automatically sync before each ingestion.

### Automatic Reindexing (File Watcher)

The watcher compares files on disk against a manifest saved during ingestion. It detects new, modified, and deleted files, and only changed files are reindexed. If a client has `source_sync` configured, files are synced from the source directory before checking for changes.

```bash
# Dry-run to see what changed
python scripts/watch_and_reindex.py --dry-run

# Run for a specific client (sync + reindex)
python scripts/watch_and_reindex.py --client myproject

# Set up as cron (every 15 minutes)
*/15 * * * * cd /path/to/trove && .venv/bin/python scripts/watch_and_reindex.py
```

#### Source Sync

Clients can auto-sync files from an external directory into their `documents/` folder. Add to `client.yaml`:

```yaml
source_sync:
  enabled: true
  source: ../..                   # Relative to client directory
  include: ["*.py", "*.md", "*.txt", "*.yaml", "*.json"]
  exclude: [clients/, venv/, .git/, __pycache__/]
```

The sync copies new/updated files, removes deleted files, and respects include/exclude patterns. It runs automatically as part of `watch_and_reindex.py`.

### Evaluation

Measure search quality with the evaluation tool. Requires a test dataset at `clients/{name}/tests/eval_dataset.json`.

```bash
# Run retrieval evaluation
python scripts/evaluate.py --client myproject

# Filter by tags
python scripts/evaluate.py --client myproject --tags search

# Include LLM-as-judge RAG quality metrics (requires LM Studio running)
python scripts/evaluate.py --client myproject --rag

# A/B comparison of merge strategies
python scripts/evaluate.py --client myproject --compare "weighted,rrf"

# JSON output for scripting
python scripts/evaluate.py --client myproject --json

# Write markdown report
python scripts/evaluate.py --client myproject --output reports/eval.md

# List clients with eval datasets
python scripts/evaluate.py --list
```

### Embedding Benchmark

Compare embedding models on your own evaluation dataset.

```bash
# Compare two models on a client's eval dataset
python scripts/benchmark_embeddings.py --client myproject \
    --models "all-MiniLM-L6-v2,BAAI/bge-base-en-v1.5"

# Save results to JSON
python scripts/benchmark_embeddings.py --client myproject \
    --models "all-MiniLM-L6-v2" --output results.json
```

### Knowledge Graph

Build a knowledge graph from indexed document chunks using LLM entity/relationship extraction.

```bash
# Build graph for a client (incremental - skips already-processed chunks)
python scripts/build_graph.py --client myproject

# Rebuild from scratch
python scripts/build_graph.py --client myproject --rebuild

# Dry-run (show chunk count and estimated LLM calls)
python scripts/build_graph.py --client myproject --dry-run

# Test with a sample of 50 random chunks
python scripts/build_graph.py --client myproject --sample 50

# Build for all graph-enabled clients
python scripts/build_graph.py --all

# Build even if graph.enabled is false in client.yaml
python scripts/build_graph.py --client myproject --force

# The graph saves every 5 batches (graph.extract_save_every), so cancelling loses at most
# the batches since the last save; the next incremental run resumes
```

### Local Agent Runner

Run Trove agents end-to-end using a local LLM (LM Studio). Combines agent system prompt + RAG context search + LLM generation.

```bash
# List available agents
python scripts/run_agent.py --list-agents

# List available clients
python scripts/run_agent.py --list-clients

# One-shot query
python scripts/run_agent.py --agent code-reviewer --client example "Review the search pipeline"

# Interactive REPL with conversation history
python scripts/run_agent.py --agent code-reviewer --client example --interactive

# JSON output for scripting
python scripts/run_agent.py --agent security-reviewer --client myproject --top-k 15 --json "Check auth patterns"
```

Available agents: `code-reviewer`, `security-reviewer`, `performance-analyzer`, `documentation-writer`, `test-generator`, `architecture-analyst`

### Prompt Library (optional)

`scripts/sync_prompts.py` and the three `trove_prompts_*` MCP tools work with a prompts.chat API client at `clients/prompts/api_client.py`. That client is not part of this repository. Without it the script exits with "The prompt library is not available" and the MCP tools return the same message instead of failing. `trove_prompts_find` still searches a local `prompts` client if you have indexed one.

If you add your own client module there, the script syncs prompts into `clients/prompts/documents/`:

```bash
python scripts/sync_prompts.py --dry-run   # preview
python scripts/sync_prompts.py --reindex   # sync, then reindex the prompts client
```

### Overnight Rebuild

Reindex all clients and rebuild knowledge graphs in one run. Requires LM Studio running with the configured model loaded.

```bash
# Incremental (changes only, fast on most nights)
./overnight-rebuild.sh

# Full rebuild from scratch
./overnight-rebuild.sh --rebuild
```

Edit the script to uncomment/comment clients in the `REINDEX_CLIENTS` and `GRAPH_CLIENTS` arrays.

### Client Creation

```bash
# Interactive wizard
python scripts/create_client.py

# Without questions
python scripts/create_client.py --name myproject --documents-dir /path/to/docs
```

Every Python script in `scripts/` except `mcp_server.py` accepts `--clients-dir PATH`; `sources.py` takes it before the subcommand (`sources.py --clients-dir PATH sync --all`). See [Using Trove with a scraper](#using-trove-with-a-scraper).

## Using Trove with a scraper

Trove works well as the search half of a scrape pipeline: the scraper writes Markdown, Trove indexes it, and answers cite the page they came from.

### Where clients live

By default clients are folders in the repository's `clients/` (or `./clients` for an installed copy). To keep them elsewhere, set `TROVE_CLIENTS_DIR` or pass `--clients-dir` to a CLI script. The flag wins over the environment variable, which wins over the default. A relative path, from either, is resolved once to an absolute path against the folder you run the script in, and `~` is expanded. `..` is collapsed but symlinks in the path are kept as written.

The manifest and the chunks key each file by its full path. A clients folder reached through a symlink therefore stores its files under the symlink path, and every switch between the symlink path and the real path makes every file look deleted under one form and new under the other: the next ingest removes the old chunks and embeds all the files again. Use one path form consistently.

The MCP server has no `--clients-dir` flag. It reads `TROVE_CLIENTS_DIR` once, when it starts.

A client is a folder with a `client.yaml`. Folders whose names start with `_` or `.`, and folders without a `client.yaml`, are not clients, so a workspace can keep a `_template/` folder next to its projects:

```
workspace/
└── projects/                  # TROVE_CLIENTS_DIR=workspace/projects
    ├── _template/             # ignored
    ├── news-site/
    │   ├── client.yaml
    │   ├── documents/         # the scraper writes Markdown here
    │   └── index/             # Trove's index, created on ingest
    └── docs-site/
        ├── client.yaml        # paths.documents may be absolute or a symlink
        └── index/
```

```bash
export TROVE_CLIENTS_DIR=$PWD/workspace/projects
python scripts/create_client.py --name news-site
# ... scraper writes workspace/projects/news-site/documents/*.md ...
python scripts/ingest.py --client news-site --json
python scripts/search.py --client news-site --json "what changed in the pricing"
```

### The Markdown contract

Each page is one `.md` file (`.markdown` works too once it is added to `documents.extensions`). It may start with YAML frontmatter:

```markdown
---
title: Pricing changes for 2027
url: https://example.com/blog/pricing-2027
date: 2026-10-01
domain: example.com
page_type: article
scraped_at: 2026-11-01T09:30:00Z
content_hash: 4f1c2e
---
# Pricing changes for 2027

## What is changing
...
```

| Field | Type | Stored as | Used for |
|-------|------|-----------|----------|
| `title` | string | `title` (string, at most 300 characters) | Shown with results and citations. Falls back to the first level-1 heading |
| `url` | string | `url` (string) | Shown with results and citations |
| `date` | date or string | `date` (ISO string) | Metadata |
| `domain` | string | `domain` | Metadata |
| `page_type` | string | `page_type` | Metadata |
| `scraped_at` | datetime or string | `scraped_at` (ISO string) | Metadata |
| `content_hash` | string | `page_content_hash` | Metadata (`content_hash` is the chunk's own hash) |
| `canonical_url` | string | `canonical_url` (string) | Groups a copy of a page (a print version, for example) with that page for [page diversity](#2-search-hybrid--agentic) when it equals another file's `url` on the same host |

All fields are optional and other keys are ignored. `title`, `url` and `canonical_url` are always stored as text: a number becomes a string (`url: 12345` becomes `"12345"`), and a value that is not a plain string or number (a list, a mapping, a date, `.nan`, `true`, or blank) is ignored, so the title falls back to the first level-1 heading (`# Title` or a title underlined with `===`). A title from either source is cut to 300 characters, because it is copied into every chunk. Dates and datetimes in the other fields are stored as ISO strings and lists as comma-separated text. The frontmatter block is removed from the indexed text. If it is not valid YAML, is not a mapping, or has no closing `---`, Trove logs a warning and indexes the whole file as text; a file is never dropped because of its frontmatter.

The body is split at its headings (`#` to `######`, and headings underlined with `===` or `---`). Each chunk records its heading path as `section`, for example `Pricing changes for 2027 > What is changing`. A section longer than 1500 characters is split further with the recursive splitter, with up to 100 characters of overlap; the overlap shrinks, to nothing if need be, so a chunk never goes over 1500 characters and no text is cut off. The overlap starts on a word boundary: a partial word at its start is left out, and the overlap is empty when no whole word fits. Inside a run of more than 30 characters with no whitespace (text written without spaces, such as Chinese or Japanese, or a long URL) there is no boundary to start on, so the overlap keeps the raw characters.

Headings inside fenced code blocks are ignored. A fence opened with three or more backticks closes only on a line holding the same character repeated at least as many times and nothing else (`~~~` fences likewise). A backtick line whose info string contains a backtick (```` ```js `x` ````) is ordinary text, not a fence. A fence that is never closed runs to the end of the file.

No text is dropped for being short. A section under 80 characters joins the chunk before it if both fit in 1500 characters, otherwise the chunk after it, and if neither fits it stays a small chunk of its own. When a chunk holds more than one section, its `section` is the heading path of the section that gives it the most text (the first one on a tie). The other sections' headings stay in the chunk text, so a search for them still finds it.

Images are not indexed: `![alt](url)` becomes its alt text and a line holding only images is dropped. To keep image syntax, set this in `client.yaml`:

```yaml
markdown:
  strip_images: false
```

Other file types (PDF, HTML, Word and so on) go through Unstructured and the title chunker, and have no `section`.

An index keeps each file's chunks as they were made until the file changes. After upgrading Trove, run `ingest.py --client C --rebuild` to chunk every file with the current rules (word-boundary overlaps, `canonical_url` metadata); until then unchanged files keep their old chunks, and a print copy counts as a page of its own. Search works on either.

### Re-running the scraper

Run `ingest.py` after each scrape. Only new and changed files are embedded. Each file's SHA-256 is kept in the manifest, so a file the scraper rewrote with identical content counts as `unchanged` and is not parsed or embedded again. A changed file has its old chunks removed before it is indexed again. A page that disappears from `documents/` is reported as `deleted` and the incremental run removes its chunks. A file that fails is listed in `failed_files`, left out of the manifest and tried again on the next run. Use `--json` to decide what to do next, for example to alert on `failed`.

Trove removes chunks only for files the manifest lists as changed or deleted. Orphan chunks, whose file is not in the manifest at all (for example a file removed between an interrupted run and its `--resume`), stay in the index until a `--rebuild` without `--resume`. See [Known limits](#known-limits) and the [roadmap](docs/REVIVAL-PLAN.md).

## Known limits

These are facts of how the code works today. The [roadmap](docs/REVIVAL-PLAN.md) lists the ones with planned work.

- Ingestion does not read the `chunking:` section of `client.yaml`. Parsed documents (PDF, Word, HTML, text, spreadsheets) always use the title chunker and Markdown always uses the heading splitter, both at 1500 characters with 100 overlap, whatever `strategy`, `max_chunk_size`, `overlap` or `separators` say. `.mbox` archives use their own email splitter and code files the tree-sitter code chunker.
- `ingest.py --client C --path P` without `--resume` clears the client's whole index before it indexes `P`, so everything not under `P` is gone from the index. To index another folder, point `paths.documents` (or `--documents-dir` on `create_client.py`) at it, or give it its own client.
- Unchanged-file skipping compares the SHA-256 of the whole file, so only byte-identical files are skipped. A scraper that rewrites a page's frontmatter (a new `scraped_at`, for example) with the same body makes that file count as changed and be embedded again.
- In a client that mixes Markdown with spreadsheets, or with non-Markdown text that has more than 10 non-blank lines or more than 5 commas, those chunks sit in a higher [ranking tier](#2-search-hybrid--agentic) and can rank above a better Markdown answer.
- The recursive splitter drops the separator where it splits, so a split at `". "` loses that full stop. A run of more than 1500 characters with no whitespace is cut at exactly 1500 characters, which splits it.
- Search has no "no confident answer" signal. Rerank scores do not separate unanswerable questions from answerable ones: on a test set of 31 answerable and 5 unanswerable plain-language questions, some questions whose top result was the right answer scored lower than questions whose answer was not in the corpus at all.
- Plain-language questions can still miss official wording that shares no words with them (for example "computer won't start" against "system fails to boot"). Heading match helps only when the query's words appear in a title or heading.
- Page diversity moves a page's third and later chunks behind other pages even when they are the better answer to the question.
- The sufficiency check in `core/refinement.py` counts raw line breaks, blank lines included, when it decides whether a chunk holds real content. It affects only whether recursive search runs another refinement round.
- A clients folder reached through a symlink re-embeds every file whenever the path form changes between the symlink path and the real path, see [Where clients live](#where-clients-live).
- The MCP server reads `TROVE_CLIENTS_DIR` once at start-up and has no `--clients-dir` flag.
- Orphan chunks (chunks whose file is not in the manifest, for example after an interrupted run) are not pruned. Chunks of deleted and changed files are removed.
- `.markdown` is not in the default `documents.extensions` list, so `.markdown` files are skipped unless the client lists it.
- The plugin loader is not called at start-up, `TROVE_PLUGINS_DIR` is not read, and plugin names in `client.yaml` (for example a plugin embedder as `embedding.model`) are not looked up in the registry.
- Code indexing is not available on Python 3.13, because tree-sitter-languages has no wheels for it.

## MCP Server (Claude Code Integration)

Trove includes an MCP (Model Context Protocol) server that allows Claude Code to query your indexed documentation directly.

### Use Case

1. Scrape API documentation and convert to markdown
2. Create a Trove client for the API (`python scripts/create_client.py`)
3. Ingest the docs (`python scripts/ingest.py --client api_name`)
4. Configure Claude Code to use the Trove MCP server
5. Claude Code searches your API docs when helping you code

### Setup

Add the server to a project's `.mcp.json` (or for all projects with `claude mcp add --scope user`, see [docs/CLAUDE_CODE_INTEGRATION.md](docs/CLAUDE_CODE_INTEGRATION.md)):

```json
{
  "mcpServers": {
    "trove": {
      "command": "python",
      "args": ["/path/to/trove/scripts/mcp_server.py"],
      "env": {
        "TROVE_DEFAULT_CLIENT": "your_client_name"
      }
    }
  }
}
```

With Trove installed as a package (`pip install ".[mcp]"`), use `"command": "trove-server"` with no `args`. It reads clients from `clients/` in the folder it is started in. To point either form at another clients folder, add `"TROVE_CLIENTS_DIR": "/path/to/clients"` to `env`.

### Available Tools

| Tool | Description |
|------|-------------|
| `trove_list_clients` | List all available Trove clients/projects |
| `trove_get_client` | Get info about the currently loaded client |
| `trove_load_client` | Load a specific client for searching |
| `trove_search` | Search documents with mode presets and context budgets |
| `trove_search_client` | Load and search a client in one call |
| `trove_search_multi` | Search across multiple clients simultaneously (cross-client RRF merge) |
| `trove_search_code` | Search code only, with language/symbol type filtering |
| `trove_list_repos` | List git repositories in the loaded client's index |
| `trove_search_repo` | Search within a specific repository |
| `trove_find_symbol` | Find a function, class, or method by name |
| `trove_summarize` | Search and return concise summaries instead of full content |
| `trove_ingest_file` | Ingest a single file into a client's index |
| `trove_ingest_url` | Fetch and ingest content from a URL |
| `trove_reindex_client` | Trigger a full background reindex for a client |
| `trove_ingest_status` | Check status/progress of an ingestion job |
| `trove_graph_stats` | Get knowledge graph statistics (entity/relationship counts, top entities) |
| `trove_graph_export` | Export knowledge graph as D3.js-compatible JSON |
| `trove_search_entities` | Search knowledge graph entities by name |
| `trove_search_relationships` | Traverse knowledge graph relationships from an entity |
| `trove_build_graph` | Build/rebuild a knowledge graph (background) |
| `trove_connect_github` | Connect GitHub repos to a client (discover, clone, sync) |
| `trove_crawl_web` | Crawl a website and save pages as Markdown |
| `trove_sync_sources` | Sync all configured sources for a client |
| `trove_list_sources` | List configured sources with sync status |
| `trove_memory_store` | Store a key-value memory entry (with optional TTL and vector embedding) |
| `trove_memory_search` | Search memory by semantic similarity or keyword |
| `trove_memory_list` | List stored memories in a namespace |
| `trove_memory_delete` | Delete a memory entry |
| `trove_workflow_create` | Create a multi-step workflow DAG with dependencies |
| `trove_workflow_status` | Get workflow status and step progress |
| `trove_workflow_update` | Update a workflow step status (running/completed/failed/skipped) |
| `trove_workflow_list` | List all workflows with progress summaries |
| `trove_agents_list` | List available agent templates |
| `trove_agents_get` | Get full agent template details |
| `trove_run_agent` | Run an agent end-to-end: search context, apply system prompt, generate via local LLM |
| `trove_prompts_find` | Combined local index + live prompts.chat API search (deduplicated). Optional, see [Prompt Library](#prompt-library-optional) |
| `trove_prompts_search_live` | Search live prompts.chat API with type/category/tag/sort filters. Optional |
| `trove_prompts_get` | Fetch full prompt by ID with optional variable substitution. Optional |

### Example Usage in Claude Code

```
# List available documentation projects
Use trove_list_clients to see what's available

# Search API documentation
Use trove_search_client with client_name="stripe_api" and query="how to create a payment intent"

# Search currently loaded client
Use trove_search with query="authentication flow"
```

### Creating Skills That Use Trove

You can create custom Claude Code skills in other projects that query Trove for documentation context. See **[docs/CLAUDE_CODE_INTEGRATION.md](docs/CLAUDE_CODE_INTEGRATION.md)** for:

- Setting up global MCP configuration
- Creating project-specific skills
- Example skill templates
- Multi-client workflows

## Development

See `CLAUDE.md` for architecture details, code guidelines, and module reference, [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup, and [docs/REVIVAL-PLAN.md](docs/REVIVAL-PLAN.md) for the roadmap.

### Running Tests

```bash
pip install -r requirements.txt -c constraints/py312.txt
pytest                         # the whole suite, several minutes
pytest tests/test_search.py    # one area
```

The suite downloads two small models from Hugging Face on first run (all-MiniLM-L6-v2 and ms-marco-MiniLM-L-6-v2). `tests/smoke/` holds checks for a built wheel installed in a clean environment.