Skip to main content
Glama

Trove — Knowledge Engine for AI Development

A local-first knowledge engine with hybrid search, source connectors, plugin extensibility, and MCP integration for Claude Code. Trove discovers, indexes, and searches your documents, code, and web content so AI tools always have the right context.

Features

Search & Retrieval

  • Hybrid Search — BM25 keyword + vector semantic search with cross-encoder re-ranking

  • RRF Merge Strategy — Reciprocal Rank Fusion as alternative to weighted merging

  • MMR Diversity — Maximal Marginal Relevance reduces redundant results

  • HyDE — Hypothetical Document Embeddings for better vector search on vague queries

  • Multi-Query / RAG-Fusion — Generate query variants and merge results via RRF

  • Search Quality Indicators — Scores, timing, source diversity, and strategy metadata

  • Configurable Embeddings — Choose embedding model per client (MiniLM, BGE, E5)

  • Stemming & Synonyms — Porter/Snowball stemmer and domain-specific synonyms

  • Agentic/Recursive Search — Automatically refines queries when results are insufficient

  • Reference Resolution — Detects "see Appendix X" and fetches referenced content

  • Query Contextualization — Understands follow-up questions in conversation context

  • Search Result Caching — LRU + TTL cache with filter-aware keys

  • Score Normalization — MinMax, Z-score, or raw normalization for merge strategies

Source Connectors

  • GitHub Connector — Discover repos from GitHub accounts/orgs, clone/fetch, track changed files. API client with rate limiting, pagination, and retries. Tokens via environment variables only.

  • Web Crawler — BFS crawl with scope control, HTML-to-Markdown conversion (markdownify + trafilatura), incremental cache via ETag/content hash.

  • Auto-sync — Sources sync before ingestion when sync_on_ingest is configured. Manual sync via CLI or MCP.

  • Incremental Updates — Only changed files are re-crawled and reindexed.

Plugin System

  • 6 Extension Points — Custom embedding models, chunkers, parsers, rerankers, LLM providers, and merge strategies

  • Plugin Registry — Register and lookup plugins by name with duplicate detection

  • Auto-Discovery — Drop a .py file or package in plugins/ and it's loaded on startup

  • Built-in Refactoring — LLM providers and rerankers already implement plugin interfaces

Ingestion & Indexing

  • Producer-Consumer Pipeline — Parallel file parsing (ThreadPool) → GPU batch embedding → progressive ChromaDB indexing

  • Fast File Collection — git ls-files for git repos (near-instant), os.scandir fallback, parallel change detection

  • Ctrl+C Safe — Each GPU batch is indexed immediately; interrupt loses at most one batch, resume skips already-indexed files

  • Code Indexing — Tree-sitter AST parsing for Python, TypeScript, JavaScript, PHP, HTML, CSS, YAML, JSON, Gherkin, Dart

  • Semantic Chunking — Split by embedding similarity drops (alternative to title-based)

  • Recursive Chunking — Priority-ordered separators with fallback

  • Contextual Embeddings — Prepend document context to chunks before embedding

  • Chunk Quality Validation — Min-length, alphanumeric, and content-hash deduplication

Response Generation

  • Inline Citations — LLM responses cite sources with [1], [2] references

  • Configurable Reranker — Local CrossEncoder or API-based (Cohere-compatible) with automatic fallback

  • Token Budgeting — Separate context budgets for document, code, and reference chunks

  • Context Pre-Filtering — Drop low-scoring chunks before LLM context

Knowledge & Agents

  • Knowledge Graph — Entity/relationship extraction, graph-augmented search (local/global/hybrid/mix modes)

  • Persistent Memory — SQLite-backed store with vector search, TTL expiration, per-client or global

  • Workflow DAGs — Multi-step task pipelines with dependency ordering and auto-status tracking

  • Agent Templates — YAML-defined personas (code-reviewer, security-reviewer, etc.)

  • Local Agent Runner — Search for context, inject system prompt, generate via LM Studio

  • Prompt Discovery — 10k+ indexed templates + live prompts.chat API search

Related MCP server: coderag

Stack

Layer

Library

License

Document Processing

Unstructured.io

Apache 2.0

Keyword Search

rank-bm25

Apache 2.0

Stemming/NLP

NLTK

Apache 2.0

Vector Search

ChromaDB + sentence-transformers

Apache 2.0

Re-ranking

sentence-transformers (cross-encoder) or API

Apache 2.0

Knowledge Graph

NetworkX

BSD

GitHub API

httpx

BSD

HTML → Markdown

markdownify + trafilatura

MIT / Apache 2.0

LLM (local)

LM Studio via OpenAI-compatible API

-

LLM (cloud)

Claude via Anthropic client

-

MCP Server

mcp

MIT

100% local capable. Zero API costs (with LM Studio). Licensed under BSL 1.1 (converts to MIT).

Quick Start

Requires Python 3.10+

# Get the code and install dependencies
git clone https://github.com/tomgould/trove.git
cd trove
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Create a new client (interactive wizard)
python scripts/create_client.py

# Option A: Add local documents
cp /path/to/docs/* clients/myproject/documents/

# Option B: Connect external sources
python scripts/sources.py connect github --client myproject --owner myorg
python scripts/sources.py crawl --client myproject --url https://docs.example.com

# Ingest and index
python scripts/ingest.py --client myproject

# Use via MCP server (see MCP Server section below)
python scripts/mcp_server.py

Creating a Client

The easiest way to create a new client is with the interactive wizard:

python scripts/create_client.py

The wizard will prompt you for:

  • Client name - Display name (e.g., "Acme Corp RAG")

  • Machine name - Directory name (auto-generated from display name)

  • Description - Optional description

  • Embedding model - MiniLM (fast) or BGE (better quality)

  • LLM settings - Provider (lmstudio/claude), context limits, max tokens

  • Search settings - BM25/vector weights, top_k, rerank settings (optional)

  • Tokenization - Stemmer selection (Porter, Snowball, or none)

  • Reference domain - Compliance, legal, technical, or academic patterns

  • Audit settings - Enable/disable logging, log level

  • System prompt - Custom prompt or use default

After completion, it creates the full directory structure and config files:

clients/your_client/
├── client.yaml       # Configuration
├── documents/        # Add your source files here
├── index/            # Search index (created on ingest)
└── prompts/          # System prompt
    └── system.txt

Manual Setup

Alternatively, create a client manually:

mkdir -p clients/myproject/{documents,index,prompts,data}

Then create clients/myproject/client.yaml (see Client Configuration below).

How It Works

1. Document Ingestion

PDF/DOCX/XLSX/CSV/MBOX/MD → Unstructured → Chunks with metadata → Vector DB + BM25 Index

Supported file types:

  • PDF - Text extraction with tables, lists preserved

  • Word - DOCX, DOC

  • Spreadsheets - XLSX, XLS, CSV

  • Email archives - MBOX (Gmail exports, etc.)

  • Markdown - Ideal for scraped API docs

  • HTML - Web pages

  • Plain text - TXT files

Features:

  • Document structure preserved (headings, sections)

  • Automatic chunking by section

  • Email metadata preserved (subject, from, to, date)

  • Recursive directory scanning (all subdirectories)

2. Search (Hybrid + Agentic)

Query → Contextualize (if follow-up)
      → BM25 (keyword) + Vector (semantic) in parallel
      → Merge (weighted or RRF)
      → Cross-encoder re-rank (adaptive depth)
      → MMR diversity (if enabled)
      → Reference resolution (fetch appendices/sections)
      → [If recursive: analyze → retry with better query]
      → Final results + quality indicators
  • BM25 finds exact matches ("Appendix C" → actual Appendix C)

  • Vector finds semantic matches ("equivalent jurisdictions" → related content)

  • Merge via weighted scores (default) or RRF rank fusion

  • Cross-encoder picks the best results (adaptive depth reduces cost on high-confidence matches)

  • MMR reduces redundancy by penalizing similarity to already-selected chunks

  • Reference resolver fetches content when chunks say "see Appendix X"

3. Response Generation

Top chunks → Context filter (drop low-scoring)
           → Token budgeting (doc/code/reference allocations)
           → LLM prompt with citation instructions
           → Answer with inline [1] [2] citations
           → Citation footer + Follow-up suggestions

Project Structure

Trove/
├── core/
│   ├── ingest.py             # Document processing with Unstructured
│   ├── pipeline.py           # Producer-consumer ingestion pipeline
│   ├── search/               # Hybrid search package
│   │   ├── types.py          # SearchQuality, SearchResult dataclasses
│   │   ├── tokenizer.py      # EnhancedTokenizer, stemming, stop words
│   │   ├── cache.py          # Search result LRU+TTL cache
│   │   ├── merge.py          # merge_search_results() for multi-index merging
│   │   └── searcher.py       # HybridSearcher (BM25 + vector + re-ranking + recursive)
│   ├── llm.py                # LLM providers (LM Studio, Claude)
│   ├── client.py             # Client configuration + validation
│   ├── refinement.py         # Query refinement & contextualization
│   ├── reference_resolver.py # Appendix/section reference resolution
│   ├── reference_config.py   # Configurable reference patterns per domain
│   ├── logging_config.py     # Audit configuration loading
│   ├── reranker.py           # Reranker backends (local, API, fallback)
│   ├── evaluation.py         # Retrieval metrics + LLM-as-judge RAG quality
│   ├── chunking.py           # Chunking strategies (title, semantic, recursive)
│   ├── ingest_progress.py    # Progress tracking and resumable ingestion
│   ├── ingest_job.py         # Ingestion job management for MCP tools
│   ├── memory.py             # Persistent memory store (SQLite + vector search)
│   ├── workflow.py           # Workflow DAG engine (task pipelines)
│   ├── agents.py             # Agent template loader (YAML configs)
│   ├── runner.py             # Local agent runner (search → RAG → LLM generate)
│   ├── plugins/              # Plugin system
│   │   ├── interfaces.py     # ABCs: BaseEmbedder, BaseChunker, BaseParser, etc.
│   │   ├── registry.py       # Plugin registry (register/lookup by name)
│   │   └── loader.py         # Auto-discovery from plugins/ directory
│   ├── sources/              # Source connectors
│   │   ├── base.py           # BaseSourceConnector ABC, SyncResult
│   │   ├── github_connector.py # GitHub repo discovery, clone, sync
│   │   ├── github_client.py  # GitHub API client (rate limiting, pagination)
│   │   ├── github_models.py  # GitHubSourceConfig, RepoInfo
│   │   ├── git_ops.py        # Git clone/fetch with token auth
│   │   ├── web_crawler.py    # BFS web crawler with scope control
│   │   ├── web_converter.py  # HTML → clean Markdown conversion
│   │   └── web_cache.py      # Incremental crawling cache (ETag/content hash)
│   ├── code/                 # Code indexing subsystem
│   │   ├── chunker.py        # Code file chunking orchestration
│   │   ├── parser.py         # Tree-sitter parser management
│   │   ├── git_utils.py      # Git repository detection
│   │   ├── incremental.py    # Incremental indexing state
│   │   └── extractors/       # Language-specific extractors
│   └── graph/                # Knowledge graph subsystem
│       ├── models.py         # Entity, Relationship dataclasses
│       ├── store.py          # KnowledgeGraph (NetworkX + JSON)
│       ├── extractor.py      # LLM-based entity/relationship extraction
│       ├── search.py         # Graph search modes
│       └── export.py         # Stats and D3.js export
├── plugins/                   # User plugin directory (drop .py files here)
├── mcp_server/
│   ├── server.py             # MCP server entry point, tool definitions, dispatcher
│   ├── shared.py             # Shared state, cache, helpers
│   └── handlers/             # Tool handler modules (clients, search, ingest, graph, memory, workflows, agents, prompts, sources)
├── agents/                    # Global agent YAML templates
├── clients/
│   ├── {name}/
│   │   ├── client.yaml       # Config (including sources: section)
│   │   ├── documents/        # Local source files
│   │   ├── sources/          # Auto-managed source directories
│   │   │   ├── github/       # Cloned git repos
│   │   │   └── web/          # Crawled markdown files
│   │   ├── index/            # Search indices
│   │   ├── prompts/          # System prompts
│   │   └── tests/            # Evaluation datasets
│   └── prompts/
│       └── api_client.py     # prompts.chat REST API client
├── scripts/
│   ├── create_client.py      # Interactive client creation wizard
│   ├── ingest.py             # CLI for document ingestion
│   ├── search.py             # CLI search tool
│   ├── evaluate.py           # CLI evaluation tool
│   ├── build_graph.py        # CLI for building knowledge graphs
│   ├── sources.py            # CLI for source connectors (GitHub, web)
│   ├── mcp_server.py         # MCP server launch script
│   ├── run_agent.py          # CLI for running agents locally
│   ├── watch_and_reindex.py  # Source sync + file watcher
│   └── sync_prompts.py       # Sync prompts from prompts.chat API
└── tests/
    └── test_core.py          # 724 tests

Client Configuration

# clients/example/client.yaml
name: "My RAG System"
machine_name: example

paths:
  documents: "documents"
  index: "index"
  prompts: "prompts"

# Embedding model settings
embedding:
  model: "all-MiniLM-L6-v2"  # or "BAAI/bge-base-en-v1.5" for better quality
  dimension: 384             # 384 for MiniLM, 768 for BGE
  query_prefix: ""           # Use "query: " for E5 models
  passage_prefix: ""         # Use "passage: " for E5 models
  contextualize: false       # Prepend doc context to chunks before embedding (requires reindex)
  hnsw_ef_construction: 200  # HNSW build-time neighbor list (requires reindex)
  hnsw_max_neighbors: 16     # HNSW max connections per node (requires reindex)
  hnsw_ef_search: 100        # HNSW query-time neighbor list (no reindex needed)

search:
  bm25_weight: 0.5
  vector_weight: 0.5
  top_k: 20
  rerank_top_n: 50
  # Merge strategy: "weighted" (default) or "rrf" (rank fusion)
  merge_strategy: "weighted"
  rrf_k: 60                         # RRF constant (only used with merge_strategy: "rrf")
  # MMR diversity: null (disabled) or 0.0-1.0 (0.0=max diversity, 1.0=pure relevance)
  mmr_lambda: null
  # Adaptive reranking: reduce rerank depth on high-confidence matches
  adaptive_rerank: true
  adaptive_rerank_threshold: 0.9     # Combined score threshold to trigger
  adaptive_rerank_depth: 15          # Reduced depth when triggered
  # BM25 tuning
  bm25_k1: 1.5                        # Term frequency saturation (>0)
  bm25_b: 0.75                        # Document length normalization (0-1)
  score_normalization: "minmax"        # "minmax", "zscore", or "raw"
  # Advanced query processing (requires LLM provider)
  hyde: false                          # Generate hypothetical doc for vector search
  multi_query: false                   # Generate query variants and merge via RRF
  multi_query_count: 3                 # Number of query variants (2-7)
  # Search result caching
  cache_ttl: 0                         # TTL in seconds (0 = disabled)
  cache_max_size: 128                  # Max cached queries per client
  # Reranker backend: "local" (CrossEncoder) or "api" (Cohere-compatible)
  reranker: "local"
  reranker_model: null               # Custom reranker model (null=default)
  reranker_endpoint: null            # API endpoint for "api" reranker
  # Tokenization settings for BM25
  tokenization:
    stemmer: "porter"  # "porter", "snowball", or null
    synonyms:          # Domain-specific synonyms
      aml: ["anti-money laundering", "financial crime"]
      kyc: ["know your customer"]

# Reference resolution settings
references:
  enabled: true
  mode: "extend"        # "extend" adds to domain defaults, "replace" uses only custom
  domain: "compliance"  # "compliance", "legal", "technical", or "academic"
  # Optional: custom patterns
  patterns:
    custom_ref:
      priority: high
      patterns:
        - '(?:see\s+)?REF-(\d+)'
      query_templates:
        - "Reference {id}"

llm:
  provider: "lmstudio"                                    # OpenAI-compatible (recommended for LM Studio)
  lmstudio_model: "mistralai/mistral-nemo-instruct-2407"  # must be set explicitly
  max_chunks: 10
  max_context_chars: 12000
  max_tokens: 2000
  citations: false                # Enable inline [N] citations in responses
  context_filter: false           # Drop low-scoring chunks before LLM context
  context_filter_threshold: 0.1   # Minimum rerank score for context inclusion
  doc_budget_ratio: 0.6           # Token budget share for document chunks
  code_budget_ratio: 0.3          # Token budget share for code chunks
  reference_budget_ratio: 0.1     # Token budget share for reference chunks

# Optional: Knowledge graph (entity/relationship extraction)
graph:
  enabled: false                # Opt-in per client
  entity_types:                 # Entity types to extract
    - person
    - organization
    - concept
    - location
  extract_batch_size: 10        # Chunks per LLM call
  max_description_chars: 200    # Truncate descriptions
  search_mode: "naive"          # Default search: naive, local, global, hybrid, mix

# Optional: Chunking strategy
chunking:
  strategy: "title"             # "title", "semantic", or "recursive"
  max_chunk_size: 1500
  min_chunk_size: 100
  overlap: 100                  # Overlap between chunks (recursive strategy)
  semantic_threshold: 0.5       # Cosine similarity for semantic split points
  separators: ["\n\n", "\n", ". ", " "]  # Priority-ordered separators (recursive strategy)

# Optional: Source connectors (auto-fetch from GitHub/web)
sources:
  github:
    - owner: myorg                  # GitHub user or org
      token_env: GITHUB_TOKEN       # Env var name (never store tokens in YAML)
      include_repos: [api, frontend] # Optional whitelist
      exclude_repos: [legacy-app]    # Optional blacklist
      sync_on_ingest: true           # Auto-sync before reindexing
    - repo: owner/repo              # Single repo shorthand
      token_env: GITHUB_TOKEN
  web:
    - url: https://docs.example.com
      stay_in_path: true            # Don't leave starting URL path
      max_pages: 500                # Safety limit
      sync_on_ingest: true

# Optional: Code indexing (tree-sitter AST parsing)
code:
  enabled: true
  paths: ["src/"]                   # Directories to index (relative to client root)
  extensions: [".py", ".ts", ".js"] # File extensions to include
  exclude_patterns: ["**/test_*", "**/__pycache__/**"]

# Optional: Code priority in search results (when docs may be outdated)
# Add to search: section
#   code_priority: true             # Boost code chunks above docs
#   code_priority_boost: 0.5        # Score boost for code chunks
#   cross_reference_code: true      # Auto-fetch related code for doc results

# Optional: Audit settings
audit:
  enabled: true
  log_level: "full"  # "full", "minimal", or "none"
  include_chunks: true
  include_scores: true
  include_timing: true

Embedding Models

Model

Dimensions

Speed

Quality

Notes

all-MiniLM-L6-v2

384

Fast

Good

Default, good for most use cases

BAAI/bge-base-en-v1.5

768

Medium

Better

Better retrieval, recommended for production

intfloat/e5-base-v2

768

Medium

Better

Requires query/passage prefixes

Reference Domains

Domain

Patterns Detected

compliance

Appendix, Annex, Schedule, Section, Table

legal

Exhibit, Clause, Article, Recital, Appendix, Schedule

technical

RFC, Issue #, PR #, Section

academic

Equation, Theorem, Lemma, Definition, Figure, Table, Chapter

CLI Commands

Ingestion

# Ingest documents for a client
python scripts/ingest.py --client myproject

# Rebuild from client's documents directory
python scripts/ingest.py --client myproject --rebuild

# Rebuild ALL clients at once
python scripts/ingest.py --all

# Rebuild all clients, documents only (skip code)
python scripts/ingest.py --all --no-code

# Resume an interrupted ingestion (skips already-processed files)
python scripts/ingest.py --client myproject --rebuild --resume

# Force BM25 cache rebuild (use after changing tokenization settings)
python scripts/ingest.py --client myproject --rebuild --force-bm25-rebuild

# List available clients
python scripts/ingest.py --list
# Basic search
python scripts/search.py --client myproject "your search query"

# Verbose output with timing and metadata
python scripts/search.py --client myproject --verbose "your query"

# Use RRF merge instead of weighted
python scripts/search.py --client myproject --merge-strategy rrf "your query"

# Enable MMR diversity (0.0=max diversity, 1.0=pure relevance)
python scripts/search.py --client myproject --mmr-lambda 0.7 "your query"

# Disable adaptive reranking
python scripts/search.py --client myproject --no-adaptive "your query"

# JSON output (for scripting/comparison)
python scripts/search.py --client myproject --json "your query"

# Full content (no truncation)
python scripts/search.py --client myproject --full "your query"

# List available clients
python scripts/search.py --list

Source Connectors

Connect external sources (GitHub repos, websites) to a client. Source files are stored in clients/{name}/sources/ and indexed alongside documents/.

# Connect a GitHub org's repos to a client
python scripts/sources.py connect github --client myproject --owner myorg

# Connect a single repo
python scripts/sources.py connect github --client myproject --repo owner/repo

# Filter repos
python scripts/sources.py connect github --client myproject --owner myorg \
    --include api frontend --exclude legacy --public-only

# Crawl a website into a client
python scripts/sources.py crawl --client myproject --url https://docs.example.com

# Crawl with options
python scripts/sources.py crawl --client myproject --url https://api.example.com/docs \
    --max-pages 200 --wait-time 2.0 --no-stay-in-path

# Sync all configured sources for a client
python scripts/sources.py sync --client myproject

# Sync all clients at once
python scripts/sources.py sync --all

# List configured sources and on-disk state
python scripts/sources.py list --client myproject

After connecting sources, ingest them:

python scripts/ingest.py --client myproject --rebuild

Sources with sync_on_ingest: true in client.yaml automatically sync before each ingestion.

Automatic Reindexing (File Watcher)

The watcher compares files on disk against a manifest saved during ingestion. It detects new, modified, and deleted files — only changed files are reindexed. If a client has source_sync configured, files are synced from the source directory before checking for changes.

# Dry-run to see what changed
python scripts/watch_and_reindex.py --dry-run

# Run for a specific client (sync + reindex)
python scripts/watch_and_reindex.py --client myproject

# Set up as cron (every 15 minutes)
*/15 * * * * cd /path/to/trove && ./venv/bin/python scripts/watch_and_reindex.py

Source Sync

Clients can auto-sync files from an external directory into their documents/ folder. Add to client.yaml:

source_sync:
  enabled: true
  source: ../..                   # Relative to client directory
  include: ["*.py", "*.md", "*.txt", "*.yaml", "*.json"]
  exclude: [clients/, venv/, .git/, __pycache__/]

The sync copies new/updated files, removes deleted files, and respects include/exclude patterns. It runs automatically as part of watch_and_reindex.py.

Evaluation

Measure search quality with the evaluation tool. Requires a test dataset at clients/{name}/tests/eval_dataset.json.

# Run retrieval evaluation
python scripts/evaluate.py --client myproject

# Filter by tags
python scripts/evaluate.py --client myproject --tags search

# Include LLM-as-judge RAG quality metrics (requires LM Studio running)
python scripts/evaluate.py --client myproject --rag

# A/B comparison of merge strategies
python scripts/evaluate.py --client myproject --compare "weighted,rrf"

# JSON output for scripting
python scripts/evaluate.py --client myproject --json

# Write markdown report
python scripts/evaluate.py --client myproject --output reports/eval.md

# List clients with eval datasets
python scripts/evaluate.py --list

Embedding Benchmark

Compare embedding models on your own evaluation dataset.

# Compare two models on a client's eval dataset
python scripts/benchmark_embeddings.py --client myproject \
    --models "all-MiniLM-L6-v2,BAAI/bge-base-en-v1.5"

# Save results to JSON
python scripts/benchmark_embeddings.py --client myproject \
    --models "all-MiniLM-L6-v2" --output results.json

Knowledge Graph

Build a knowledge graph from indexed document chunks using LLM entity/relationship extraction.

# Build graph for a client (incremental - skips already-processed chunks)
python scripts/build_graph.py --client myproject

# Rebuild from scratch
python scripts/build_graph.py --client myproject --rebuild

# Dry-run (show chunk count and estimated LLM calls)
python scripts/build_graph.py --client myproject --dry-run

# Test with a sample of 50 random chunks
python scripts/build_graph.py --client myproject --sample 50

# Build for all graph-enabled clients
python scripts/build_graph.py --all

# Build even if graph.enabled is false in client.yaml
python scripts/build_graph.py --client myproject --force

# Graph saves after every batch — safe to cancel mid-run, resumes on next incremental run

Local Agent Runner

Run Trove agents end-to-end using a local LLM (LM Studio). Combines agent system prompt + RAG context search + LLM generation.

# List available agents
python scripts/run_agent.py --list-agents

# List available clients
python scripts/run_agent.py --list-clients

# One-shot query
python scripts/run_agent.py --agent code-reviewer --client example "Review the search pipeline"

# Interactive REPL with conversation history
python scripts/run_agent.py --agent code-reviewer --client example --interactive

# JSON output for scripting
python scripts/run_agent.py --agent security-reviewer --client myproject --top-k 15 --json "Check auth patterns"

Available agents: code-reviewer, security-reviewer, performance-analyzer, documentation-writer, test-generator, architecture-analyst

Prompt Sync

Sync new and trending prompts from the live prompts.chat API into the local index.

# Preview what would be synced
python scripts/sync_prompts.py --dry-run

# Sync newest + popular prompts (2 pages per sort order)
python scripts/sync_prompts.py

# Sync more pages for deeper coverage
python scripts/sync_prompts.py --pages 100

# Sync and trigger incremental reindex
python scripts/sync_prompts.py --reindex

Overnight Rebuild

Reindex all clients and rebuild knowledge graphs in one run. Requires LM Studio running with the configured model loaded.

# Incremental (changes only — fast on most nights)
./overnight-rebuild.sh

# Full rebuild from scratch
./overnight-rebuild.sh --rebuild

Edit the script to uncomment/comment clients in the REINDEX_CLIENTS and GRAPH_CLIENTS arrays.

Client Creation

# Interactive wizard
python scripts/create_client.py

MCP Server (Claude Code Integration)

Trove includes an MCP (Model Context Protocol) server that allows Claude Code to query your indexed documentation directly.

Use Case

  1. Scrape API documentation and convert to markdown

  2. Create a Trove client for the API (python scripts/create_client.py)

  3. Ingest the docs (python scripts/ingest.py --client api_name)

  4. Configure Claude Code to use the Trove MCP server

  5. Claude Code can now search your API docs when helping you code

Setup

Add to your Claude Code settings (~/.claude/settings.json) or project .mcp.json:

{
  "mcpServers": {
    "trove": {
      "command": "python",
      "args": ["/path/to/trove/scripts/mcp_server.py"],
      "env": {
        "TROVE_DEFAULT_CLIENT": "your_client_name"
      }
    }
  }
}

Available Tools

Tool

Description

trove_list_clients

List all available Trove clients/projects

trove_get_client

Get info about the currently loaded client

trove_load_client

Load a specific client for searching

trove_search

Search documents with mode presets and context budgets

trove_search_client

Load and search a client in one call

trove_search_multi

Search across multiple clients simultaneously (cross-client RRF merge)

trove_search_code

Search code only, with language/symbol type filtering

trove_list_repos

List git repositories in the loaded client's index

trove_search_repo

Search within a specific repository

trove_find_symbol

Find a function, class, or method by name

trove_summarize

Search and return concise summaries instead of full content

trove_ingest_file

Ingest a single file into a client's index

trove_ingest_url

Fetch and ingest content from a URL

trove_reindex_client

Trigger a full background reindex for a client

trove_ingest_status

Check status/progress of an ingestion job

trove_graph_stats

Get knowledge graph statistics (entity/relationship counts, top entities)

trove_graph_export

Export knowledge graph as D3.js-compatible JSON

trove_search_entities

Search knowledge graph entities by name

trove_search_relationships

Traverse knowledge graph relationships from an entity

trove_build_graph

Build/rebuild a knowledge graph (background)

trove_connect_github

Connect GitHub repos to a client (discover, clone, sync)

trove_crawl_web

Crawl a website and save pages as Markdown

trove_sync_sources

Sync all configured sources for a client

trove_list_sources

List configured sources with sync status

trove_memory_store

Store a key-value memory entry (with optional TTL and vector embedding)

trove_memory_search

Search memory by semantic similarity or keyword

trove_memory_list

List stored memories in a namespace

trove_memory_delete

Delete a memory entry

trove_workflow_create

Create a multi-step workflow DAG with dependencies

trove_workflow_status

Get workflow status and step progress

trove_workflow_update

Update a workflow step status (running/completed/failed/skipped)

trove_workflow_list

List all workflows with progress summaries

trove_agents_list

List available agent templates

trove_agents_get

Get full agent template details

trove_run_agent

Run an agent end-to-end: search context, apply system prompt, generate via local LLM

trove_prompts_find

Combined local index + live prompts.chat API search (deduplicated)

trove_prompts_search_live

Search live prompts.chat API with type/category/tag/sort filters

trove_prompts_get

Fetch full prompt by ID with optional variable substitution

Example Usage in Claude Code

# List available documentation projects
Use trove_list_clients to see what's available

# Search API documentation
Use trove_search_client with client_name="stripe_api" and query="how to create a payment intent"

# Search currently loaded client
Use trove_search with query="authentication flow"

Prompt Discovery Examples

The prompt discovery tools search 10k+ indexed prompt templates and the live prompts.chat API.

Find a prompt (combined local + live search):

Find me a prompt template I can use as a senior code reviewer
I need an AI prompt for generating midjourney-style portraits

Browse live prompts with filters:

Search prompts.chat for the newest IMAGE generation prompts
What are the most popular prompts on prompts.chat right now?

Fetch a full prompt with variable substitution:

Fetch prompt cmj1zb0yc000ivl0rc9ka596e and set Position to "Data Engineer"

Use the /prompt-finder skill shortcut:

/prompt-finder react code review assistant
/prompt-finder cartoon illustration of a Mini Cooper
/prompt-finder SQL expert that explains query optimization

Multi-step workflow:

Is there a prompt template that acts as a job interviewer?
Fetch the full version and customize it for a DevOps Engineer position.

Creating Skills That Use Trove

You can create custom Claude Code skills in other projects that query Trove for documentation context. See docs/CLAUDE_CODE_INTEGRATION.md for:

  • Setting up global MCP configuration

  • Creating project-specific skills

  • Example skill templates (including /prompt-finder for AI prompt discovery)

  • Multi-client workflows

Development

See CLAUDE.md for architecture details, code guidelines, and module reference.

Running Tests

pytest tests/ -v

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables AI coding assistants to search and retrieve information from a locally ingested knowledge base using hybrid search, grounded in user-curated documentation.
    16
    MIT
  • A
    license
    Not graded
    quality
    A
    maintenance
    Enables private source-code repositories to be indexed and searched via hybrid retrieval (symbol, lexical, semantic, and code-graph), then packages token-budgeted, structure-aware context for LLMs such as Claude.
    73 PyPI
    Apache 2.0
  • A
    license
    B
    quality
    A
    maintenance
    Enables AI coding agents to build and query local repository memory, returning ranked files, context packs, relationship maps, semantic recall, and feedback-tuned search without uploading code.
    9
    1,459 PyPI
    1
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables AI coding agents to query a local semantic code graph for surgical context, cross-file resolution, framework-aware routes, and code intelligence while keeping indexes automatically up to date.
    77,341 npm
    MIT