Reliable Agentic RAG

A runnable, synthetic expense-policy workflow showing where model planning ends and deterministic decisions begin. It retrieves fictional policies, checks a claim, and requires a separate human step before recording a simulated payment.

Status: local portfolio demonstration. No real money, customer records, employer source, or legal rules. The default planner is scripted, not an LLM. An optional local Ollama planner selects validated tool calls.

Ownership

Portfolio owner: Esteban H. Román Catafau. The initial implementation was developed with Codex assistance for this portfolio. It demonstrates inspectable engineering patterns; it is not presented as an independently authored historical project or a copy of any production system.

Related MCP server: financial-evidence

Quick Start

Python 3.13 and uv are required. Dependency download is the only network use in the default demo; no model download or API key is needed.

uv sync --frozen --extra dev
uv run --frozen policy-demo demo
uv run --frozen pytest -q

The sample claim requests 42.50 synthetic credits for a meal with a receipt. The tool returns 30.00, cites the fictional policy IDs, and records no payment.

uv run --frozen policy-demo demo --propose
uv run --frozen policy-demo review PROPOSAL_ID
uv run --frozen policy-demo approve PROPOSAL_ID

Replace PROPOSAL_ID with the full ID printed by the first command. The last command displays the proposal and asks you to type its complete ID in an interactive terminal. It records only a synthetic payment. Repeated approval fails; repeated identical proposals reuse the same ID. A claim ID cannot be reused with different contents.

Architecture

flowchart TD
    C[Validated synthetic claim] --> P[Scripted or local model planner]
    P --> R[BM25 + LSA retrieval]
    R --> P
    P --> T[Deterministic Decimal policy tool]
    T --> O[Structured assessment and fixed citations]
    O --> S[Pending proposal in local SQLite]
    S --> H[Separate interactive human review]
    H --> A[Atomic simulated payment and audit events]
    M[MCP stdio client] --> R
    M --> T

• Planning: a maximum of five validated steps. Unknown tools and premature completion fail closed. Model output cannot replace the original claim or authorize approval.

• Retrieval: BM25 plus TF-IDF/SVD latent semantic vectors, fused with reciprocal rank fusion. LSA is trained on six fictional documents; it is not a pretrained neural embedding model. Out-of-vocabulary queries return no results.

• Calculation: Decimal arithmetic and fixed category caps. Missing receipts produce an ineligible decision. Retrieved prose is not parsed into executable policy; a small trusted policy module is authoritative.

• Evidence: output citations come from the deterministic tool, not generated model prose. Valid IDs here do not constitute a general solution to citation entailment.

• Approval: a SHA-256 ID binds the stored assessment to its contents. SQLite transactions enforce single-use approval and append lifecycle events atomically. The local database owner is trusted.

MCP and Optional Model

uv run --frozen policy-mcp

This starts a real MCP stdio server using the official Python SDK. Exposed tools are search_policy and assess_claim. There is deliberately no approval tool. The integration test launches a subprocess, initializes a client session, discovers tools, and calls the assessment tool with valid and invalid inputs.

The implementation pins SDK 1.30.0 and uses its v1 FastMCP API; upgrading to SDK v2 requires migration. See the SDK v1 branch.

For model-driven planning, install and run Ollama separately with a model supporting JSON schema output:

uv run --frozen policy-demo demo --ollama-model YOUR_INSTALLED_MODEL

Only the local loopback endpoint is used. Requests disable environment proxies and redirects. A model can fail to follow the plan; this is reported as failure, never converted into an approval. No real-model result is claimed by the offline demo.

Evaluation and Limits

Tests cover cap boundaries, missing receipts, malformed inputs, bounded planning, retrieved policy relevance, fabricated assessments, storage tampering, duplicate proposals, repeated approvals, and the real MCP protocol. See VALIDATION.md for the observed run. No production reliability rate is inferred from these tests.

This small corpus is not evidence of enterprise retrieval quality. The local approval step is a demonstration, not multi-user authentication: a person or agent with shell/database access can act as the owner. Audit rows are not externally tamper-proof. There is no payment integration, tenant isolation, OCR, live policy ingestion, or remote MCP deployment. Failed planner attempts are surfaced to the caller; the persistent event log covers proposal/payment lifecycle only.

Files and Development

src/policy_agent/
  policy.py       # input/output schemas and deterministic rules
  retrieval.py    # BM25 + LSA and rank fusion
  agent.py        # bounded planner/tool loop and local model adapter
  store.py        # proposals and transactional approval events
  server.py       # MCP stdio interface
  cli.py          # demo and human-review commands
tests/            # behavioral and protocol tests

uv run --frozen ruff check .
uv run --frozen ruff format --check .
uv build

See SECURITY.md for trust boundaries and DISCLOSURE.md for data provenance. MIT license; dependency licenses remain their respective owners'.