openfox-rag → fr

RAG integration for OpenFox. A skill file that teaches the OpenFox agent to search local knowledge bases (Obsidian vaults, technical docs, procedures) through a hybrid retrieval pipeline: BM25 + Vector → RRF → Cross-encoder Reranker.

No MCP, no plugin, no additional process. The RAG server runs independently (rag-system), and the agent controls it entirely via its built-in terminal using 10 shell shortcuts.

Why

Related MCP server: ragi

How it works

OpenFox Agent
    │
    ├──→ Built-in terminal
    │         │
    │         ├──→ rc          (health check)
    │         ├──→ llmers      (start full stack)
    │         ├──→ ragr void "query"   (precise search)
    │         ├──→ rag void "query"    (fast search)
    │         ├──→ rst         (tail logs)
    │         └──→ rsk         (stop server)
    │
    └──→ Skill file (skills/rag-search.md)
              └──→ Tells the agent when and how to use each command

Zero additional processes on the OpenFox side. The RAG server is a separate, independent service. The agent starts it, searches it, monitors it, and stops it — all from the terminal, exactly like any other command-line tool.

Installation

mkdir -p ~/rag
git clone https://github.com/cried-nutty-won/openfox-rag.git ~/rag/openfox-rag
cd ~/rag/openfox-rag
bash install.sh

The installer:

• Detects OS, RAM, GPU (NVIDIA, Apple Silicon, lspci)

• Detects shell (fish, bash, zsh, sh) and writes aliases to the correct config

• Offers 0.6B models (default) or 4B models (GPU only — hidden on CPU)

• Downloads GGUF from official Qwen + Voodisss

• Clones and configures rag-system

• Scans for Obsidian and documentation vaults (interactive loop)

• Installs the OpenFox skill (~/.config/openfox/skills/rag-search.md)

• Displays all commands and documentation paths at the end

Dry-run mode

Test without modifying anything:

bash install.sh --dry-run

Commands

Vaults

Configured interactively during installation. Each vault has a short name used in commands:

rag void "nftables configuration"        # search vault "void"
ragr linux "dracut hooks"                 # precise search in "linux"
ragr all "your query"                     # search all vaults
rag obsidian "your query"                 # search all Obsidian vaults

Model selection

GGUF models

Note: The MTEB scores of the embedding and reranker models are not comparable — they evaluate different tasks (vector retrieval vs. pair re-ranking). The actual pipeline gain from adding the reranker is +12 NDCG@10 points (FinanceQA benchmark), not the difference between the two MTEB scores above.

• Embedding: Qwen/Qwen3-Embedding-0.6B-GGUF or Qwen/Qwen3-Embedding-4B-GGUF (official)

• Reranker: Voodisss/Qwen3-Reranker-0.6B-GGUF-llama_cpp or Voodisss/Qwen3-Reranker-4B-GGUF-llama_cpp (mandatory — community GGUFs are broken, see llama.cpp #16407)

MTEB (Massive Text Embedding Benchmark) is the reference benchmark for evaluating embedding model quality. It measures a model's ability to produce vectors that capture text meaning, across 8 task types:

Why it matters for RAG

The MTEB Retrieval score is the most important for RAG: it directly measures the model's ability to find the right document. The higher the score, the less the RAG needs the reranker to compensate.

The 0.6B is sufficient for a local RAG with reranker. The 4B adds +5 points but requires a GPU.

Backend Configuration

The embedding and reranker models can be served by the same backend as the chat LLM, or by separate llama-server instances. See presets/models-llamacpp.ini for a ready-to-use configuration.

llamacpp (models.ini)

One server, one port, three models. The router swaps models in/out of VRAM on demand.

0.6B models (CPU-friendly, ~1 GB total)

[*]
n-gpu-layers = all
batch-size = 2048
ubatch-size = 2048
load-on-startup = Qwen3-Embedding-0.6B

[Qwen3-Embedding-0.6B]
model = /path/to/Qwen3-Embedding-0.6B-Q8_0.gguf
embedding = true
pooling = last
ctx-size = 8192

[Qwen3-Reranker-0.6B]
model = /path/to/Qwen3-Reranker-0.6B-Q4_K_M.gguf
reranking = true
pooling = rank
embedding = true
ctx-size = 1024

[deepseek-v4-flash]
model = /path/to/DeepSeek-V4-Flash.gguf
ctx-size = 32768

4B models (GPU recommended, ~5 GB total)

[*]
n-gpu-layers = all
batch-size = 2048
ubatch-size = 2048
load-on-startup = Qwen3-Embedding-4B

[Qwen3-Embedding-4B]
model = /path/to/Qwen3-Embedding-4B-Q4_K_M.gguf
embedding = true
pooling = last
ctx-size = 32768

[Qwen3-Reranker-4B]
model = /path/to/Qwen3-Reranker-4B-Q4_K_M.gguf
reranking = true
pooling = rank
embedding = true
ctx-size = 32768

[deepseek-v4-flash]
model = /path/to/DeepSeek-V4-Flash.gguf
ctx-size = 32768

llama-server --host 127.0.0.1 --port 8000 --models-max 1 --models-preset models.ini

Note on pooling: The official Qwen3 blog says last (hidden state of the final [EOS] token) for embedding. Some community guides say mean. Official docs take precedence: use last.

vLLM (GPU cluster)

vllm serve Qwen/Qwen3-Embedding-0.6B \
  --served-model-name Qwen3-Embedding-0.6B \
  --task embedding --port 8000 &

vllm serve Qwen/Qwen3-Reranker-0.6B \
  --served-model-name Qwen3-Reranker-0.6B \
  --task score --port 8001 &

sglang

python -m sglang.launch_server \
  --model-path Qwen/Qwen3-Embedding-0.6B \
  --is-embedding --port 8000 &

ollama (embedding only)

Ollama does not support reranking. The RAG server automatically falls back to pure RRF (BM25 + vector, no reranker).

ollama pull qwen3-embedding:0.6b

# RAG server configuration for ollama
export LLAMA_EMBED_URL="http://127.0.0.1:11434/api/embeddings"
# No reranker URL — automatic RRF fallback

Backend compatibility summary

The RAG server adapts automatically: if the reranker is unreachable, it falls back to pure RRF without error.

Workflow Integration

Custom workflow step

Add a "Research" step before "Build" in OpenFox's workflow editor:

[Research (RAG)] → [Build] → [Verify] → [Code Review] → [Summary]

General Instructions

Add to your OpenFox General Instructions:

Before coding, always consult the local knowledge base via the terminal.
Run: rc
Then: ragr obsidian "your query"
Use the retrieved passages as context. Always cite the source file.

Performance

Known Limitations

• Reranker GGUF: Only Voodisss GGUFs work. Community conversions are broken (missing cls.output.weight tensor). See llama.cpp #16407.

• Host prompt cache: llama.cpp PR #16391 defaults to 8 GiB host prompt cache. For embedding/reranker servers where prompts are never reused, add --cache-ram 0 to prevent OOM.

• Reranker candidates > 21 on CPU: The reranker crashes. 18 is the stable sweet spot.

• No hot-reload: Modifying an Obsidian file requires cache clear + re-index.

• Orphan cache: Deleted chunks remain in the JSON cache (no garbage collection yet).

• Agent must parse output: The agent reads the command output and extracts relevant passages. Works well with models ≥ 7B.

References

• OpenFox — local-LLM-first agentic coding assistant

• rag-system — standalone RAG server (Python)

• Qwen3 Embedding blog — official documentation

• Voodisss multi-model guide — llamacpp models.ini reference

• Voodisss Reranker GGUF — working GGUF conversions

• llama.cpp #16407 — why community reranker GGUFs are broken

• llama.cpp PR #16391 — host prompt cache (8 GiB default)

• Dave Ebbelaar, "Hybrid Retrieval from Scratch" (2026) — methodology

Recommended System

Void Linux with niri desktop. Faster boot, 1GB RAM occupied only, fastest and whole package manager. System.d free, runit instead wich is faster, lighter and private. Perfect balance between security and fluidity.

License

MIT