Skip to main content
Glama
Brightdotdev

io.github.Brightdotdev/darwin-rag

Official

Darwin RAG

A local-first RAG engine that ingests documents, indexes them with BM25 + dense embeddings, and exposes search via an MCP server for AI agent integration.

  • Ingestion — PDF, Markdown, HTML, images (OCR), CSV, Excel, ODS, URLs

  • Indexing — BM25 keyword + dense embedding hybrid index with configurable chunking strategies

  • Search — Hybrid, semantic, or keyword retrieval with reranking, diversity rerank, and structural penalties

  • Generation — LLM-backed answer synthesis via LiteLLM (OpenAI, Anthropic, Gemini, etc.)

  • Observability — Structured logging with per-session history, queryable via MCP

  • Isolation — Multiple independent stores for tenant/project separation

  • Deployment — stdio (AI agent subprocess), SSE, or Streamable HTTP; Docker-ready

  • Local-first — Everything runs locally, fully offline-capable after setup

Built by BrightDotDev.
License: MIT with Attribution


Quick Start

1. Install

pip install darwin-rag

2. Set up models

# Interactive — detects hardware, pick your models
darwin-admin setup interactive

# Or one-shot (embedding-only, no prompts)
darwin-admin setup --preset required

3. Start the MCP server & connect

# stdio mode — for AI agent subprocess (Claude Desktop, Cursor, etc.)
darwin mcp

# Or HTTP mode — for remote clients
darwin mcp --http --port 8765

Configure your MCP client:

{
  "mcpServers": {
    "darwin": {
      "command": "darwin",
      "args": ["mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-..."  // At least one LLM provider key
      }
    }
  }
}

Or generate config automatically:

darwin config claude          # Claude Desktop config
darwin config cursor          # Cursor config
darwin config all --copy      # All clients + copy to clipboard

Related MCP server: Corpus-KB

MCP Tools

Tool

Description

search_darwin

Query the knowledge base with hybrid/semantic/keyword search

search_lists

Search structured data (CSVs, JSON arrays) by field values

get_search_results

List saved search results

get_search_result_by_id

Load a saved search result by filename

get_schema

Inspect schemas for structured files (keys, types, record counts)

run_pipeline

Ingest + index documents from a path or URL

purge_artifacts

Delete pipeline artifacts for specific files

create_store

Create a new isolated data store

list_files

List all tracked files with pipeline status

file_status

Detailed status for a single file across all stages

get_logs

Query session logs (oldest first, INFO excluded)

Full documentation: docs/mcp.md


Remote / HTTP Mode

Start the server on a network-accessible endpoint:

# SSE transport (legacy)
darwin mcp --sse --host 0.0.0.0 --port 8765

# Streamable HTTP transport (recommended for production)
darwin mcp --http --host 0.0.0.0 --port 8765

Configure your MCP client with the URL:

{
  "mcpServers": {
    "darwin": {
      "url": "http://your-host:8765/mcp"  // or /sse for SSE mode
    }
  }
}

Environment Variables

Variable

Required

Description

OPENAI_API_KEY

No*

OpenAI provider key

ANTHROPIC_API_KEY

No*

Anthropic provider key

GEMINI_API_KEY

No*

Google Gemini provider key

MISTRAL_API_KEY

No*

Mistral AI provider key

GROQ_API_KEY

No*

Groq provider key

COHERE_API_KEY

No*

Cohere provider key

TOGETHER_API_KEY

No*

Together AI provider key

OPENROUTER_API_KEY

No*

OpenRouter provider key

DEEPSEEK_API_KEY

No*

DeepSeek provider key

DARWIN_BASE_DIR

No

Override the base data directory

NO_COLOR

No

Set to any value to disable ANSI color output

* At least one LLM provider key is required for answer generation. Search/indexing works without any.


Setup Details

Command

What it does

darwin-admin setup interactive

Guided setup — detect hardware, choose models

darwin-admin setup --preset required

Download embedding model only (fastest)

darwin-admin setup --preset recommended

Embedding + reranker + OCR models

darwin-admin setup logging

Reconfigure logging only

darwin-admin setup validate

Validate current setup

See docs/setup.md for the full walkthrough including Docker, from-source install, and API key configuration.


CLI Reference

darwin — User CLI

Command

Description

darwin mcp

Start MCP server (stdio, --sse or --http for network)

darwin config [client]

Generate MCP client config

darwin-admin — Power-user CLI

Command

Description

darwin-admin setup

Setup models, logging, and configuration

darwin-admin status

System status overview

darwin-admin models

Model registry: list, install, switch, keys

darwin-admin store

Data store: status, files, audit, health, repair

darwin-admin pipeline

Ingestion pipeline: run, ingest, index, purge

darwin-admin search

Interactive search

darwin-admin logs

Structured log viewer and management

darwin-admin system

System information

darwin-admin uninstall

Remove Darwin data and configuration

See docs/admin.md for the full command reference.


Python API

For embedding darwin-rag as a library in your own app:

from core import Darwin

d = Darwin()
d.ingest("./papers", recursive=True)
results = d.search("what is this paper about")
records = d.search_records(filters={"status": "active"})

Full reference: docs/api.md


Documentation

Doc

What

setup.md

Full setup walkthrough

mcp.md

MCP server, tools, resources, transports

admin.md

Admin CLI reference

architecture.md

For developers and contributors

pipeline.md

Ingestion & indexing

retrieval.md

Search engine

storage.md

DarwinStore

models.md

Model registry & inference

logger.md

Structured logging

orchestrators.md

High-level business logic

api.md

Python API (Darwin class)


Contributing

Found a bug? Want to add something? You're welcome here.

  • Issues — open one at github.com/BrightDotDev/DARWIN/issues

  • Code — fork, branch, PR. Keep it minimal.

  • AI-generated code is fine — but you own what you ship. Test it before submitting.

Read CONTRIBUTING.md for the full guidelines.


License

MIT with Attribution — see LICENSE.

Core architecture and implementation by BrightDotDev.

Maintenance

ActivityActive
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    D
    maintenance
    Local-first RAG indexing and semantic search MCP server. Enables document retrieval and context-aware queries using local embedding models.
    3
    16
    MIT
  • F
    license
    A
    quality
    B
    maintenance
    A local-first document retrieval engine that mounts as an MCP tool for agents to index files, search for relevant passages, and let the agent's own LLM answer.
    4
    -
  • A
    license
    Not graded
    quality
    A
    maintenance
    MCP server for local RAG over personal notes, PDFs, and documents, enabling plain-English querying and hybrid search with multi-hop context expansion.
    MIT