rag-assertivo

Fail-closed RAG for PT-BR. Hybrid retrieve (dense + BM25) → cross-encoder rerank → answer with a literal citation per sentence, checked by code. Unverified text never leaves.

Senior-AI-engineer proof in ten seconds: the reranker is optional, the proof is not. A tempting “split pools” change was measured, rejected, and documented.

flowchart LR
  Q["question"] --> R["hybrid retrieve<br/>e5-large + BM25"]
  R --> RK["rerank<br/>jina-reranker-v2"]
  RK --> G["generate JSON<br/>sentence + literal quote"]
  G --> P["code proof<br/>citation_found + numbers"]
  P -->|fail| X["cut / fail-closed"]
  P -->|pass| V["optional LLM verifier"]
  V --> OUT["answer + proofs"]

LangGraph runs that chain (rag/graph.py). Tracing: docs/TRACING.md (Langfuse or LangSmith, both optional).

Why this is not a toy RAG

Phrase proof is the product. Measured on a private production corpus (anonymized below): a relevance-score cutoff added zero extra trap catches on the test split — the citation contract already refused what the cutoff would have refused. So MIN_RELEVANCE stayed off.

Related MCP server: Legal MCP Server

Eval: baseline vs rerank vs pools

Numbers below are from a private PT-BR production corpus (calibration split, evidence threshold L = 0.6, hybrid e5-large + BM25 + Jina v2). They are not measurements of the public golden set in eval/golden.jsonl. Collection names, hosts, and questions are withheld; the funnel code in eval/ is the same.

Each question dies in at most one step. viva = the right chunk reached top-k.

Deaths by step (both paths summed)

Latency (interleaved A/B, same pass, HNSW)

Decision: pools not merged. CLI gain was +2 questions (need ≥ 3, McNemar). Latency +12–22%. The hypothesis (“synthetic crowding causes step-2 deaths”) failed: 41 of 45 step-2 CLI misses were nenhum — neither dense-only nor sparse-only retrieved the evidence. Split pools mostly helped step 4 (MCP k=6 cutting a CLI hit). That is a k-policy problem, not a fusion problem.

Rerank itself is cheap in failure terms: step 3 is 1–2 deaths. Turning rerank off is not a baseline this eval uses in production; the shipped system is hybrid with Jina. The public harness (python -m eval.run_funnel --backend fake) lets you A/B RAG_RERANK=0 against 1 on the legislative golden set.

Release gate (answer rate / facet coverage / fail-closed) on the same private calibration:

Reproduce the public gate (lexical backend, no keys):

pip install -r requirements-dev.txt
python -m eval.run_gate run --tag demo --backend fake
python -m eval.run_funnel --backend fake --tag demo
pytest -q

docker compose up

cp .env.example .env
docker compose up -d qdrant rag
docker compose run --rm ingest    # first run downloads e5-large + BM25 (~minutes)

• Qdrant: http://localhost:6333

• MCP HTTP: http://localhost:8000/mcp

• Health: curl -fsS http://localhost:8000/healthz

Ingest reads only corpus/texts/*.md (official Planalto excerpts). No scrapers.

Without Docker (stdio MCP, after a local ingest):

python -m ingest.ingest          # needs Qdrant + models
python -m rag.mcp_server         # MCP_TRANSPORT=stdio

The extractive generator is the default (RAG_GEN_MODEL=extractive). Set OPENAI_API_KEY / ANTHROPIC_API_KEY and RAG_GEN_MODEL=gpt-4o-mini (or claude-haiku-4-5) for a real model. Tests never need those keys.

Talk to the MCP

Cursor / Claude Desktop — stdio (no Docker):

{
  "mcpServers": {
    "rag-assertivo": {
      "command": "python",
      "args": ["-m", "rag.mcp_server"],
      "env": {
        "QDRANT_URL": "http://localhost:6333",
        "COLLECTION_NAME": "publico_br"
      }
    }
  }
}

HTTP (after docker compose up):

{
  "mcpServers": {
    "rag-assertivo": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

If MCP_AUTH_TOKEN is set (≥16 chars), send Authorization: Bearer <token>.

Tools:

Contract: cite numbers and claims only from citavel=true. Ementas are a map, not a source.

CLI: python -m rag search "o que é dado pessoal" · python -m rag answer "prazo de vício no CDC"

Layout

rag/            retrieve, proof, answer, LangGraph, MCP
ingest/         chunk + secret scan + upsert (public files only)
corpus/texts/   CF/88, LGPD, CDC, LAI, Marco Civil (excerpts)
eval/           golden, traps, judge, run_gate, funnel
tests/          mocks; `pytest -q` on a clean checkout

Sanitization checklist: docs/LEFT_OUT.md. Tracing: docs/TRACING.md. Eval details: eval/README.md.

Resumo PT-BR

RAG que não inventa. Busca híbrida (e5-large + BM25 no Qdrant), rerank Jina v2, resposta com citação literal por frase conferida por código (rag.proof + guarda de números). Texto sem prova não sai. Eval com golden, armadilhas, juiz, portão de release e funil (em que degrau o trecho certo morre). Corpus só de legislação pública brasileira. Pools separados foram medidos em produção e reprovados (ganho insuficiente, latência +12–22%). Orquestração LangGraph; tracing Langfuse/LangSmith opcional. docker compose up + MCP em stdio ou HTTP.

MIT. Public legislative excerpts: official texts, vendored from planalto.gov.br for reproducible ingest.