CodeSight

AI-powered document search engine — hybrid BM25 + vector + RRF retrieval with pluggable LLM answer synthesis.

The live public site is holusight.com. That is Holusight's public marketing site; document indexing stays local-first, so indexed files are not published there and do not leave your environment by default.

Quick Start

# Install
pip install -e ".[dev]"

# Install with AST chunking support (Python/JS/TS — higher MRR for code)
pip install -e ".[dev,ast]"

# Index a folder of documents
python -m codesight index /path/to/documents

# Search (hybrid BM25 + vector)
python -m codesight search "payment terms" /path/to/documents

# Filter by file type
python -m codesight search "auth" /path/to/code --glob '*.py'

# Ask a question (requires LLM API key — see Configuration)
python -m codesight ask "What are the payment terms?" /path/to/documents

# Machine-readable output
python -m codesight search "query" /path --json

# Check index status
python -m codesight status /path/to/documents

# Launch the web chat UI
pip install -e ".[demo]"
python -m codesight demo

# Production server (FastAPI + browser UI)
pip install -e ".[server]"
export CODESIGHT_API_KEY=$(openssl rand -hex 24)
export CODESIGHT_DOCUMENTS_DIR=/path/to/documents
python -m codesight serve

# Or use Docker (see docs/playbooks/docker-deployment.md)
export CODESIGHT_DOCUMENTS_HOST_DIR=/path/to/documents
docker compose up --build

Related MCP server: semantic-search-mcp

Python API

from codesight import CodeSight

engine = CodeSight("/path/to/documents")
engine.index()                                     # Index all files
results = engine.search("payment terms")           # Hybrid search
answer = engine.ask("What are the payment terms?") # Search + LLM answer
status = engine.status()                           # Index freshness check

The package root exports CodeSight, ServerConfig, Answer, IndexStats, RepoStatus, and SearchResult for stable public imports.

Supported Formats

Architecture

• Document Parsing: PDF, DOCX, PPTX text extraction with page/section metadata

• Chunking: AST-based (tree-sitter) for Python/JS/TS — function/class boundaries preserve semantic units. Regex fallback for other languages. Paragraph-aware splitting for documents.

• Embeddings: voyage-code-3 (API, code files) / all-MiniLM-L6-v2 (local, docs). Auto-detected via VOYAGE_API_KEY.

• Vector Store: LanceDB (serverless, file-based)

• Keyword Search: SQLite FTS5 sidecar

• Retrieval: Hybrid BM25 + vector + code-vector with RRF merge → metadata filename boost → optional reranker

• Reranker: voyage rerank-2 (code-aware, auto-enabled with VOYAGE_API_KEY). Local ms-marco cross-encoder opt-in only.

• Answer Synthesis: Pluggable LLM backend (Claude, Azure OpenAI, OpenAI, Ollama)

See ARCHITECTURE.md for the full system tour.

Performance

Measured on the holusight codebase (96 files, 20 representative queries):

AST chunking is the largest single lever (+0.224 MRR). The local ms-marco cross-encoder hurts code retrieval — only enable it explicitly.

Deployment (pilot)

Single-team production shape: FastAPI server, browser UI, API key auth, read-only document mount.

export CODESIGHT_API_KEY=$(openssl rand -hex 24)
export CODESIGHT_DOCUMENTS_HOST_DIR=/path/to/documents
docker compose up --build

See docs/playbooks/docker-deployment.md and the capability matrix for what is shipped vs planned.

holusight.com is a static marketing site only — customer documents are indexed on the customer's deployment.

Configuration

See .env.example for all options.

Stack

• Python 3.11+

• LanceDB + SQLite FTS5

• sentence-transformers + voyage-code-3 (optional)

• tree-sitter (optional — AST chunking for Python/JS/TS)

• Anthropic Claude API / Azure OpenAI / OpenAI / Ollama

• Streamlit (local demo UI)

• FastAPI + uvicorn (single-team production server, optional [server] extra)

• pymupdf, python-docx, python-pptx (document parsing)