CoreRag

A local-first, privacy-preserving knowledge engine with semantic search, exposed via MCP (Claude Desktop) and REST API. Optimized for Apple Silicon.

Features

Search

• Hybrid Search: Vector (BAAI/bge-m3, 1024d) + BM25 full-text with RRF fusion

• Cross-Encoder Reranking: ms-marco-MiniLM-L-6-v2

• HyDE Expansion: Hypothetical document embeddings for better recall

• Multi-Query Fusion: Parallel query variants merged via RRF

• Time-Decay Scoring: Recent documents weighted higher

• Collection Tags: Filter searches by tagged document groups

Ingestion Pipeline

• Inbox Workflow: Drop files, auto-process via watchdog or dashboard batch

• Human-in-the-Loop: Web dashboard with skip/error management, quality report banner, per-detection redaction editor

• Dual RAG Databases: Main (redacted for cloud-safe search) + Restricted (unredacted for local-only access)

• Document Catalog: SQLite catalog tracking every file across all destinations (RAG, Obsidian, archive)

• Three-Layer PII Detection: Presidio NER + custom dictionary + LLM advisory, with per-detection Keep/Redact toggles

• Smart Filing: Archive to ~/Documents/PKM/, export markdown with LLM tags + summaries to Obsidian vault

• Per-Agent Access Control: Settings tab with per-action permission toggles per agent, API key management

• Archive Manager: Browse, search, filter cataloged documents; cold storage migration with folder structure replication

• Parent-Child Chunking: Context-preserving hierarchical chunks with quality scoring

• Corrective RAG: Post-retrieval relevance filtering (correct/ambiguous/incorrect)

Multi-Format Support

• Documents: PDF (with OCR fallback), DOCX, TXT, Markdown, JSON, YAML, CSV

• Spreadsheets: XLSX, XLS, XLSM (markdown table output per sheet)

• Code: Python, JavaScript, TypeScript, JSX, TSX, Go, Rust, Java, Ruby (AST + line-based chunking)

• Images: PNG, JPG, WebP, HEIC (Vision.framework OCR + VLM captioning)

• Audio: MP3, WAV, M4A (mlx-whisper transcription + topic segmentation)

• Video: MP4, MOV (keyframe extraction + scene detection + audio)

Quality Assurance

• LLM-Powered Tagging: Purpose-driven collection tags (replaces keyword auto-tagger), year as tag

• Duplicate Detection: Content hash + MinHash/LSH + semantic similarity

• Link Checker: Async URL validation with caching

• Freshness Indicators: Age classification + staleness warnings

• Conflict Detection: Find contradictions across documents

Advanced

• GraphRAG: Bitemporal knowledge graph with confidence decay

• Episodic Memory: User context and search pattern tracking

• Rate-Limited REST API: Authenticated v1 endpoints with slowapi

• MCP Server: Full tool suite for Claude Desktop integration

• Memory Safety: Auto-pause at high RAM usage, GC between files

• Security Hardening: CSRF protection, XSS escaping, PII redaction fail-safe, asyncio.to_thread for blocking I/O, LanceDB connection caching, thread-safe embedding singleton, OrderedDict LRU cache

Related MCP server: Claude RAG MCP Pipeline

Quick Start

# Clone and setup
git clone https://github.com/TJ-Neary/CoreRag.git
cd CoreRag
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_lg

# Copy and configure environment
cp .env.example .env
# Edit .env with your paths

# Install menu bar app (auto-starts server at login)
./scripts/install_menubar.sh

# Or start manually
./scripts/run_system.sh

See StartHere.md for detailed setup instructions.

Usage

CLI

python -m src.cli.main status                          # System status
python -m src.cli.main search "your query"             # Search knowledge base
python -m src.cli.main ingest /path/to/folder -r -t mytag  # Ingest with tags
python -m src.cli.main health                          # System health checks
python -m src.cli.main check-links /path               # Find broken links
python -m src.cli.main duplicates /path                # Find duplicates
python -m src.cli.main stale /path --days 365          # Find stale content
python -m src.cli.main tag /path                       # Auto-tag files
python -m src.cli.main pii list                        # Manage PII dictionary
python -m src.cli.main optimize-db                     # Optimize LanceDB
python -m src.cli.main backup create                   # Create backup
python -m src.cli.main graph stats                     # Knowledge graph stats
python -m src.cli.main memory list                     # Episodic memory

REST API (v1)

# Capability manifest (no auth required)
curl http://localhost:8000/api/v1/manifest

# Search (with optional tag filtering)
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $CORERAG_API_KEY" \
  -d '{"query": "authentication setup", "k": 5, "tags": ["sphr-study"]}'

# Ingest content
curl -X POST http://localhost:8000/api/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $CORERAG_API_KEY" \
  -d '{"content": "...", "source": "my-app", "metadata": {}}'

# Stats and deletion
curl -H "X-API-Key: $CORERAG_API_KEY" http://localhost:8000/api/v1/stats
curl -X DELETE -H "X-API-Key: $CORERAG_API_KEY" http://localhost:8000/api/v1/documents/{id}

MCP (Claude Desktop)

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "corerag": {
      "command": "/path/to/CoreRag/venv/bin/python",
      "args": ["-m", "src.mcp_server.server"],
      "cwd": "/path/to/CoreRag"
    }
  }
}

Dashboard

python -m src.server    # http://localhost:8000

Web UI for reviewing AI-proposed metadata, editing tags, marking sensitivity, and committing documents through the pipeline.

Configuration

Create .env from the example:

cp .env.example .env

Key variables:

Technology Stack

Testing

pytest                           # Full suite with coverage
pytest -m "not slow"             # Skip slow tests
pytest -m "not integration"      # Skip integration tests
pytest -k "test_name"            # Single test

Development

black src/ tests/ --line-length 100    # Format
ruff check src/ tests/                 # Lint
mypy src/                              # Type check
./scripts/security_scan.sh --staged    # Security scan before commit

See CONVENTIONS.md for coding standards and CLAUDE.md for AI agent instructions.

License

MIT