RAG Factory — AutoML for RAG

RAG Factory turns RAG configuration from guesswork into measurement. From a short conversation plus a sample of your own documents, it recommends candidate RAG configurations, builds an isolated sub-environment for each, evaluates it on your own data, and optimizes across the candidate grid — then registers the winning environments as a persistent fleet you can query directly or over MCP.

Status: v1.0 snapshot. This repository is a curated portfolio snapshot of a project that continues to evolve in a private development repository.

📸 Screenshots / demo of the web UI will be added here.

The problem

A good RAG setup cannot be reliably predicted up front. MTEB scores do not predict performance in your domain, and the chunking strategy can matter as much as the embedding model. The only trustworthy answer is to measure candidates on your own corpus — but doing that by hand (generating a QA set, building indexes, running ranking metrics, comparing) is slow and expensive.

The solution

RAG Factory closes the loop:

[Recommender]  ->  [Factory]  ->  [Evaluator]
   input            build          measure
     ^                                 |
     +-------- optimization loop ------+

• Recommender (input). A conversational interview plus a data-profile turns your intent and a document sample into a search space of candidate configurations — not a single guessed "truth".

• Factory (build). For a given configuration it builds a sub-environment: parsing → chunking → embedding → index.

• Evaluator (output). It measures each candidate on your own data (MRR, NDCG, faithfulness / completeness), and the result feeds back to select the winner.

• Registry (fleet). Winning environments are persisted and can be served — locally, via a Gradio web UI, or through an MCP server.

What is built here vs. adopted

Honesty about the boundary matters, so it is explicit:

Principle: reinvent zero metrics and zero optimizer; build exactly what is new.

Tech stack

Python · AutoRAG · LlamaIndex (OpenAI-compatible LLMs) · MCP (FastMCP) · Gradio · pandas / numpy · langdetect · PyYAML · LLM access via OpenRouter.

Package layout

Getting started

# 1. Environment
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Configure access (copy the template and fill in your key)
cp .env.example .env
#   then edit .env: OPENROUTER_API_KEY, OPENROUTER_BASE_URL, OPENROUTER_MODEL

Run the web application:

python -m ragfactory.app

Run the MCP server (stdio transport):

python -m ragfactory.mcp_server
# optional: RAGFACTORY_REGISTRY_DIR=/path/to/registry python -m ragfactory.mcp_server

Note: some build/evaluation paths run AutoRAG and may require an embedding model and a GPU. A corpus is not bundled — you point RAG Factory at your own documents.

Testing

pytest                    # unit / fast tests (default; costly tests deselected)
pytest -m integration     # slower tests that run AutoRAG / GPU
pytest -m costly          # tests that spend money on live LLM calls (OpenRouter)

The suite is substantial (24 test modules) and mirrors the production package closely. Markers are defined in pytest.ini.

Documentation

• docs/ARCHITECTURE.md — design decisions, the build-vs-adopt rationale, and evaluation methodology.

License

Released under the MIT License © 2026 Dariusz Poślad.