jarvis-rag-local

Local-first GraphRAG over a personal markdown knowledge base — with an LLM router, an MCP server and scheduled self-maintenance.

Built and operated solo, in production on a 6 GB consumer GPU since April 2026. Roughly 14 k lines of Python, one operator, zero cloud dependency by default.

The problem

A consultant's business knowledge (decisions, client context, market watch, project notes) lives in ~450 markdown files. Reading them by hand doesn't scale, and pasting them into a chat model produces confident answers that are not in the notes.

jarvis-rag-local turns that folder into a knowledge graph you can query in natural language, with three hard constraints: every answer cites its source file, the model refuses when the corpus doesn't contain the answer, and nothing leaves the machine unless you explicitly switch a backend to a hosted API.

Related MCP server: lightrag-mcp

Architecture

flowchart LR
    V[(Markdown vault<br/>~450 notes)] -->|frontmatter filter<br/>hash dedup · chunk scoring| I[scripts/ingest.py]
    I --> LR[LightRAG<br/>entity + relation extraction]
    LR --> G[(Knowledge graph<br/>14.5k entities · 12.3k relations)]
    LR --> VDB[(Vector store<br/>nano-vectordb)]
    subgraph Local models — Ollama
        Q[Qwen 2.5 3B/7B<br/>extraction + synthesis]
        E[nomic-embed-text 768d<br/>or bge-m3 1024d via sentence-transformers]
    end
    LR <--> Q
    LR <--> E
    G --> QP[Query pipeline<br/>anti-hallucination prompt v2.4<br/>citation validation]
    VDB --> QP
    QP --> CLI[scripts/query.py]
    QP --> MCP[scripts/mcp_server.py<br/>FastMCP → Claude Desktop]
    QP --> H[hermes/<br/>brain / worker router]
    H --> TG[Telegram voice bot<br/>separate repo]
    subgraph Self-maintenance
        AI[auto_ingest_weekly.py<br/>snapshot → ingest → smoke tests → rollback]
        C[agents/custodian<br/>health-check questions 3×/day]
    end
    AI --> I
    C --> QP

How a query flows. The question goes through LightRAG in naive mode by default (graph modes local / global / hybrid are available but overflow the 8k context of a 7B model on 6 GB VRAM). Retrieved chunks are handed to the synthesis model with a strict prompt: cite [source: file.md] for every claim, synthesise across chunks, never invent, answer in French. scripts/query.py then validates every citation against the real vault and strips the ones that don't exist.

Hermes — the router. hermes/ splits work between a brain (planning, decisions, multi-step reasoning) and a worker (summarise, extract, classify, RAG query). Both default to local Qwen; the brain can be switched to Claude Haiku with a monthly call budget that falls back to the worker when exhausted. Short factual instructions bypass decomposition entirely and go straight to one RAG query.

Self-maintenance. A scheduled job snapshots the storage, ingests new or modified notes, runs three smoke queries, and rolls back automatically if they fail. A custodian agent asks the graph a rotating pool of known-answer questions three times a day and only notifies (Telegram) on anomaly.

Stack

Quickstart

git clone https://github.com/LucasLeduc/jarvis-rag-local && cd jarvis-rag-local
python -m venv venv && source venv/bin/activate && pip install -r requirements.txt
ollama pull qwen2.5:3b && ollama pull qwen2.5:7b && ollama pull nomic-embed-text
cp .env.example .env            # set VAULT_PATH to a folder of .md files
python scripts/ingest.py        # first pass; add --dry-run to preview what would be ingested
python scripts/query.py "What did we decide about pricing in March?"

Notes are opt-in: a file is ingested only if its YAML frontmatter has rag: true. Folders and files listed in EXCLUDED_FOLDERS / EXCLUDED_FILES (config.py) are always skipped. Re-running ingest.py is incremental (SHA-256 per file).

Useful entry points:

Windows users: scripts/setup_windows.ps1 installs Python deps, pulls the Ollama models and creates .env.

Numbers (June 2026, real corpus)

The numbers above come from the state-of-the-graph reports the pipeline writes after each ingest. They are not benchmark claims; they describe one real corpus on one real machine.

Engineering notes — things that were not in the docs

• bge-m3 through Ollama returns NaN vectors on French texts longer than ~300 characters (87 % NaN on real entity descriptions). The same model through sentence-transformers gives 0 % NaN. embedding_st.py exists because of that; tools/16-test_bgem3_sanity.py reproduces it and tools/rebuild_vdb_st.py re-embeds the whole store from 768d to 1024d.

• A 7B model on 6 GB VRAM spills ~47 % to CPU. Hybrid graph queries then time out, so naive is the default and the 3B model does extraction. Cheaper hardware decisions matter more than prompt tricks.

• Graph pollution is a real failure mode. Small extraction models create "zombie" entities (an entity named after a stray word, a person merged with a city). tools/20-graph-cartography.py, 21-pollution-detect.py, 19-purge-hallucinated-entities.py and 22-cleanup-storage.py were written to measure and clean it; a cleanup in May 2026 removed 36 orphan document ids.

• Refusing too much is as bad as hallucinating. The first anti-hallucination prompt refused 6/25 legitimate questions. v2 explicitly allows multi-chunk synthesis and light inference, and separates "not in the corpus" from "phrased differently".

• Every ingest is reversible. Snapshot before, smoke test after, rollback on failure. Two months of daily runs, zero manual restores needed after the first week.

Known limits

• Retrieval is sensitive to wording; the strict golden set scored 4/10 in May 2026. The reranker (bge-reranker-v2-m3) is declared in config but not wired in yet.

• Citation validation checks that a cited file exists, not that the sentence is actually in it.

• RAGAS scripts are present (tools/07-run_ragas_eval_jarvis.py) but the metrics were never fully computed on this corpus; only latency and refusal rates were.

• The synthesis prompt and the Hermes system prompt are written in French and tuned for one operator's tone. Adapting them is a copy-edit, not a refactor.

• Single-user by design: no auth, no multi-tenant storage.

Roadmap

1. Wire the reranker and re-run the golden set.

2. Contextual chunk headers (document title + section path prepended to each chunk) to fix wording sensitivity.

3. Docker Compose (Ollama + app) and a small FastAPI layer so the MCP server and the query endpoint share one process.

4. Compute RAGAS faithfulness / context precision properly and publish the numbers, good or bad.

5. Langfuse tracing on the query path (already used on the sibling agent dashboard).

Repository layout

config.py            single source of truth: paths, models, chunking, anti-hallucination prompt
local_llm.py         multi-host Ollama client with failover and short connect timeouts
embedding_st.py      sentence-transformers embedding backend (bge-m3)
scripts/             ingest, query, MCP server, eval, scheduled auto-ingest, smoke tests
hermes/              brain/worker router, usage budget, run logger
agents/custodian/    health-check agent
hooks/               Claude Code session hooks (optional memory export)
tools/               storage snapshots, PII scan, graph cartography, pollution detection, cleanup

Author

Lucas Leduc — AI & automation project lead, Lille (France). LinkedIn · GitHub

Companion repositories: jarvis-telegram-voice-bot (voice interface to this graph) · jarvis-acquisition (multi-agent outreach drafting with a learning loop) · claude-skills.

License

MIT — see LICENSE.