Darwin RAG

A local-first RAG engine that ingests documents, indexes them with BM25 + dense embeddings, and exposes search via an MCP server for AI agent integration.

Ingestion — PDF, Markdown, HTML, images (OCR), CSV, Excel, ODS, URLs
Indexing — BM25 keyword + dense embedding hybrid index with configurable chunking strategies
Search — Hybrid, semantic, or keyword retrieval with reranking, diversity rerank, and structural penalties
Generation — LLM-backed answer synthesis via LiteLLM (OpenAI, Anthropic, Gemini, etc.)
Observability — Structured logging with per-session history, queryable via MCP
Isolation — Multiple independent stores for tenant/project separation
Deployment — stdio (AI agent subprocess), SSE, or Streamable HTTP; Docker-ready
Local-first — Everything runs locally, fully offline-capable after setup

Built by BrightDotDev.
License: MIT with Attribution

Quick Start

1. Install

pip install darwin-rag

2. Set up models

# Interactive — detects hardware, pick your models
darwin-admin setup interactive

# Or one-shot (embedding-only, no prompts)
darwin-admin setup --preset required

3. Start the MCP server & connect

# stdio mode — for AI agent subprocess (Claude Desktop, Cursor, etc.)
darwin mcp

# Or HTTP mode — for remote clients
darwin mcp --http --port 8765

Configure your MCP client:

{
  "mcpServers": {
    "darwin": {
      "command": "darwin",
      "args": ["mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."  // At least one LLM provider key
      }
    }
  }
}

Or generate config automatically:

darwin config claude          # Claude Desktop config
darwin config cursor          # Cursor config
darwin config all --copy      # All clients + copy to clipboard

MCP Tools

Tool	Description
`search_darwin`	Query the knowledge base with hybrid/semantic/keyword search
`search_lists`	Search structured data (CSVs, JSON arrays) by field values
`get_search_results`	List saved search results
`get_search_result_by_id`	Load a saved search result by filename
`get_schema`	Inspect schemas for structured files (keys, types, record counts)
`run_pipeline`	Ingest + index documents from a path or URL
`purge_artifacts`	Delete pipeline artifacts for specific files
`create_store`	Create a new isolated data store
`list_files`	List all tracked files with pipeline status
`file_status`	Detailed status for a single file across all stages
`get_logs`	Query session logs (oldest first, INFO excluded)

Full documentation: docs/mcp.md

Remote / HTTP Mode

Start the server on a network-accessible endpoint:

# SSE transport (legacy)
darwin mcp --sse --host 0.0.0.0 --port 8765

# Streamable HTTP transport (recommended for production)
darwin mcp --http --host 0.0.0.0 --port 8765

Configure your MCP client with the URL:

{
  "mcpServers": {
    "darwin": {
      "url": "http://your-host:8765/mcp"  // or /sse for SSE mode
    }
  }
}

Environment Variables

Variable	Required	Description
`OPENAI_API_KEY`	No*	OpenAI provider key
`ANTHROPIC_API_KEY`	No*	Anthropic provider key
`GEMINI_API_KEY`	No*	Google Gemini provider key
`MISTRAL_API_KEY`	No*	Mistral AI provider key
`GROQ_API_KEY`	No*	Groq provider key
`COHERE_API_KEY`	No*	Cohere provider key
`TOGETHER_API_KEY`	No*	Together AI provider key
`OPENROUTER_API_KEY`	No*	OpenRouter provider key
`DEEPSEEK_API_KEY`	No*	DeepSeek provider key
`DARWIN_BASE_DIR`	No	Override the base data directory
`NO_COLOR`	No	Set to any value to disable ANSI color output

* At least one LLM provider key is required for answer generation. Search/indexing works without any.

Setup Details

Command	What it does
`darwin-admin setup interactive`	Guided setup — detect hardware, choose models
`darwin-admin setup --preset required`	Download embedding model only (fastest)
`darwin-admin setup --preset recommended`	Embedding + reranker + OCR models
`darwin-admin setup logging`	Reconfigure logging only
`darwin-admin setup validate`	Validate current setup

See docs/setup.md for the full walkthrough including Docker, from-source install, and API key configuration.

CLI Reference

darwin — User CLI

Command	Description
`darwin mcp`	Start MCP server (stdio, `--sse` or `--http` for network)
`darwin config [client]`	Generate MCP client config

darwin-admin — Power-user CLI

Command	Description
`darwin-admin setup`	Setup models, logging, and configuration
`darwin-admin status`	System status overview
`darwin-admin models`	Model registry: list, install, switch, keys
`darwin-admin store`	Data store: status, files, audit, health, repair
`darwin-admin pipeline`	Ingestion pipeline: run, ingest, index, purge
`darwin-admin search`	Interactive search
`darwin-admin logs`	Structured log viewer and management
`darwin-admin system`	System information
`darwin-admin uninstall`	Remove Darwin data and configuration

See docs/admin.md for the full command reference.

Python API

For embedding darwin-rag as a library in your own app:

from core import Darwin

d = Darwin()
d.ingest("./papers", recursive=True)
results = d.search("what is this paper about")
records = d.search_records(filters={"status": "active"})

Full reference: docs/api.md

Documentation

Doc	What
setup.md	Full setup walkthrough
mcp.md	MCP server, tools, resources, transports
admin.md	Admin CLI reference
architecture.md	For developers and contributors
pipeline.md	Ingestion & indexing
retrieval.md	Search engine
storage.md	DarwinStore
models.md	Model registry & inference
logger.md	Structured logging
orchestrators.md	High-level business logic
api.md	Python API (Darwin class)

Contributing

Found a bug? Want to add something? You're welcome here.

Issues — open one at github.com/BrightDotDev/DARWIN/issues
Code — fork, branch, PR. Keep it minimal.
AI-generated code is fine — but you own what you ship. Test it before submitting.

Read CONTRIBUTING.md for the full guidelines.

License

MIT with Attribution — see LICENSE.

Core architecture and implementation by BrightDotDev.