Coverage Determination Agent

Agentic RAG over payer medical-policy documents. The goal is that every claim in the output can be mechanically traced to a clause that exists, was in effect on the date in question, and actually says what it is cited for. When that can't be shown, the system sends the case to a human instead of answering.

The question it answers: does payer P, under plan L, cover procedure C for a patient with condition D as of date T, and which clause says so?

The corpus is entirely synthetic. Payers, plans, bulletins, riders and clause text are invented, modelled on the public format of medical-policy bulletins. No real payer document is used anywhere in this repository.

In one paragraph: a deterministic Python harness runs a Planner → Retriever → Grader loop over an MCP policy server. A Synthesizer writes a cited determination, and a Verifier re-fetches every cited clause cold and checks it. A pure-Python gate then decides DETERMINE, REVIEW, REFUSE or HALT. The models reason, but they never decide the outcome. A 152-case eval suite, most of it labelled by construction from the corpus rather than by a model, measures retrieval and answers separately.

Status

So the committed baseline is a stub-mode scorecard. The stub is a deterministic stand-in for the model: it performs real MCP retrieval, but its outcomes come from a keyword heuristic. Read the numbers accordingly:

Producing the live numbers takes one command, cda eval --set-baseline with a key. The measured projection is ~$21 and ~4 hours for the full suite on Sonnet 5 (docs/cost.md).

Results on a local 7B model

The 22 hand-written seed cases, run end to end on qwen2.5:7b through Ollama: free, offline, 81 minutes on an M3 Pro (median 234s per case). The scorecard is committed at evals/baselines/ollama-qwen2.5-7b.json.

A 7B model is a stress test, not a result. It checks the claim the design rests on: a weaker model should produce more REVIEWs, not more wrong answers. That held.

• The 8 wrong outcomes were all stopped. Four were over-refusals: the Grader kept declaring the evidence insufficient. It named gaps that weren't there, such as "fingerstick frequency unknown" when the narrative states it, or treated a bulletin-vs-FAQ conflict as blocking even though precedence settles it. The other four were wrong determinations, and the Verifier rejected a citation in every one of them, so none got past REVIEW.

• Retrieval wasn't the problem. Recall was 1.00 on every case, so all eight misses happened in grading or synthesis. The two-layer scorecard makes that attribution immediate. The Retriever did pull superseded versions in two cases (stale retrieval rate 0.06), but none reached a citation.

• The price was over-caution. Seven correct answers were also held at REVIEW: one by the iteration cap, six by Verifier rejections. Some rejections are legitimate: the Synthesizer writes claims that drop a condition or apply member facts beyond what the quote says. At least one is a Verifier misreading: it said a clause requiring four daily fingersticks "does not address the frequency".

• Confidence carried no signal. The Synthesizer reported ≥ 0.80 on every determination, and 1.00 on 15 of 21, so the 0.70 threshold never fired. The mechanical and entailment checks did all the catching. That is an argument for a gate that doesn't lean on self-reported confidence.

The obvious next run is the same seed set on a frontier model, or a mixed setup with the local model everywhere except the Verifier.

Related MCP server: insurance-mcp-poc

Why agentic rather than vanilla RAG

A single embedding lookup fails here, and each way it fails is built into the corpus as a trap with labelled eval cases:

So retrieval has to be planned, critiqued and re-run, and the answer has to be verified against the corpus rather than trusted.

Architecture

   request                ┌──────────────────────────────────────────────┐
   ─────────              │ HARNESS — deterministic Python, no LLM       │
   payer, plan,           │ owns control flow, budget, gate, traces      │
   codes, dx,             └──────────────────────────────────────────────┘
   as_of_date                             │
        │              ┌─────────────────▼──────────────────┐
        └────────────► │ 0. PRE-FLIGHT (no model)           │  skill: ocr-confidence-gate
                       │   PHI redaction → field extraction │
                       │   → confidence → code validation   │──── below threshold ──► ⛔ HALT
                       │   → injection defusal              │      (names the field)
                       └─────────────────┬──────────────────┘
                                         │
        ╔════════════════════════════════▼═══════════════════════════════╗
        ║  AGENTIC LOOP  (capped at 3 iterations, metered, traced)       ║
        ║                                                                ║
        ║   PLANNER ──► RETRIEVER ──► GRADER ──► sufficient? ─no─┐       ║
        ║   no tools    MCP tools     no tools                   │       ║
        ║                  │                              re-plan with   ║
        ║                  ▼                              the named gap  ║
        ║       ┌──────────────────────┐                          │      ║
        ║       │ MCP: policy-corpus   │◄─────────────────────────┘      ║
        ║       │ (read-only, 5 tools) │                                 ║
        ║       └──────────────────────┘                                 ║
        ╚════════════════════════════════┬═══════════════════════════════╝
                                         │ yes
                       ┌─────────────────▼──────────────────┐
                       │ SYNTHESIZER — no tools             │  skills: citation-format,
                       │ determination + [clause_id] cites  │  coverage-criteria-logic,
                       └─────────────────┬──────────────────┘  refusal-policy
                                         │
                       ┌─────────────────▼──────────────────┐
                       │ VERIFIER                           │
                       │  deterministic: cold MCP re-fetch  │  ◄── sees only the answer,
                       │    · clause exists?                │      never the reasoning
                       │    · quote verbatim?               │      trace
                       │    · in effect on as_of?           │
                       │  model-mediated:                   │
                       │    · does the quote entail it?     │
                       └─────────────────┬──────────────────┘
                                         │
                       ┌─────────────────▼──────────────────┐
                       │ GATE — pure Python, no model       │
                       │ faithfulness == 1.0                │
                       │ AND no unresolved contradiction    │
                       │ AND confidence ≥ threshold         │
                       └──┬──────────────┬──────────────┬───┘
                          │              │              │
                     ✅ DETERMINE    ⚠️ REVIEW      🚫 REFUSE
                     answer +        answer + the   "corpus does not
                     verified cites  gap named      answer this"
                                                    (a success state)

The core idea: the harness is deterministic and holds all control flow. Subagents reason; they never decide the outcome. The Verifier runs cold. It is given only the input and the final answer, so it can't be talked into agreeing with reasoning it never saw. Refusal counts as a success state, and the eval rewards it.

Quickstart

python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest -q          # generates the corpus on first run

Everything below runs without an API key using --stub:

.venv/bin/cda eval --stub --diff       # 152 cases, scorecard vs committed baseline

# Watch the gate reject a citation to a policy version not yet in effect
CDA_STUB_FAULT=stale .venv/bin/cda ask --stub \
  --payer MHP --plan MHP-HMO-BASE --code A9276 --dx E10.9 --as-of 2024-03-15 \
  --question "Is personal-use CGM covered?"

⚠️  GATE: REVIEW
   · clause MHP-MP-0142.v2.C3 was not in effect on the as-of date

With ANTHROPIC_API_KEY exported (or in .env), drop --stub to run the real subagents:

.venv/bin/cda ask \
  --payer MHP --plan MHP-HMO-BASE --code A9276 --dx E10.9 \
  --as-of 2024-03-15 \
  --narrative "Type 1 diabetes. Chart documents two fingerstick tests per day." \
  --question "Is personal-use CGM covered for this member?"

That is case T1-cgm-stale-02. The correct answer is NOT_COVERED, because on 2024-03-15 the governing version required four daily fingersticks. Change --as-of to 2025-01-20 and the same facts become COVERED, because the requirement was removed on 2024-07-01.

Free, on a local model

Any model id starting with ollama/ runs every role on a local Ollama server. No API key is needed and nothing leaves the machine:

ollama pull qwen2.5:7b
export CDA_MODEL=ollama/qwen2.5:7b CDA_TIMEOUT_SCALE=4
.venv/bin/cda ask --payer MHP --plan MHP-HMO-BASE --code A9276 ...   # ~4 min on an M3 Pro
.venv/bin/cda eval --suite seed --set-baseline                       # 22 cases

The client uses Ollama's native API so it can set a 32k context window per request. Ollama's default window is much smaller and truncates silently. Small models also need a tighter tool loop: on its first run, qwen2.5:7b spent five minutes emitting parallel tool calls. The local path therefore caps calls per turn, output per tool turn, and tool-result size. A reply that breaks the output contract becomes a REVIEW for that case rather than aborting the eval. Each model's baseline is written to evals/baselines/<model>.json, so a live run never overwrites the stub baseline that CI diffs against.

Components

MCP server: policy-corpus

This is the integration boundary. It is read-only, and nothing in the system can write. All corpus access goes through these five tools, which is what makes the retrieval stack swappable without touching agent logic.

The swap is real and shows up in the scorecard. Changing only the reranker, on the same 152 cases:

CDA_RERANKER=rrf .venv/bin/cda eval --stub --diff

  = gold-clause recall                 1.0000 ->       1.0000  (+0.0000)
  + determination accuracy             0.5526 ->       0.5987  (+0.0461)
  ! gate correctness                   0.8026 ->       0.5855  (-0.2171)
  + FALSE AUTO-DETERMINE                   59 ->           42  (-17)

Every knob is recorded in every trace, so a scorecard move can be traced to its cause: CDA_EMBEDDER=tfidf|sentence-transformers, CDA_RERANKER=rrf|weighted|date_aware, CDA_VECTOR_BACKEND=local|pinecone.

The dense half is pluggable: a local numpy backend by default, or Pinecone serverless with hosted embeddings. Clause text is never written to the vector index, and effective dates are filtered server-side before top-k rather than after. docs/vector-backends.md explains why both matter, and gives the live Pinecone result: identical quality metrics, with p95 latency going from 13ms to 2.6s.

Subagents

Per-role model, effort and token caps live in src/config.py, and CDA_MODEL swaps every role at once. Moving a role to a cheaper model is the cost lever. Make the change, rerun eval --diff, and read the scorecard instead of assuming the swap was safe.

Skills: .claude/skills/

Domain policy lives in versioned markdown, not in prompt strings, so a policy change is a reviewable diff. Byte lengths are fingerprinted into every trace, so when a scorecard moves you can tell whether a skill changed underneath it.

citation-format · coverage-criteria-logic · ocr-confidence-gate · refusal-policy · payer-taxonomy

Harness: src/harness/

Iteration cap, token/cost budget with a hard stop, PHI redaction, injection defusal, the gate, and JSONL trace emission. It makes no LLM calls. Anything that costs money or needs an audit trail is decided here.

The SDK retries transient errors but does not bound total wall clock, and it does not bound an agentic tool loop at all. So three caps sit on top: per request (timeout_s), per Retriever tool loop (300s), and per run (900s). A blown cap becomes a REVIEW with a stated reason, never an exception and never a silent hang. A test asserts that the worst case with retries still fits inside the run cap.

Repo hooks: .claude/

Two project-scoped Claude Code hooks protect the committed baseline. That baseline is what makes the regression gate meaningful.

Both hooks fail open on a malformed payload, so a broken hook can't block all edits.

Evals: two layers, scored separately

Keeping the layers separate is what lets you tell which part regressed. If recall and accuracy both drop, it's a retrieval problem. If recall is flat and accuracy drops, it's a synthesis problem.

Layer 1: retrieval (deterministic). Gold-clause recall, stale-retrieval rate, clauses retrieved per case.

Layer 2: answer. Determination accuracy, gate correctness, citation faithfulness (the Verifier makes this measurable), hallucinated-clause count, stale-citation count, appropriate-refusal rate, cost, p95 latency. Plus false_auto_determine: the system decided on its own and was wrong. That one must stay at zero. Anything the gate caught doesn't count against it.

The offline Judge scores reasoning quality, clarity and hedging, never correctness. Determination accuracy and faithfulness are measured; having a model also grade them would turn a measured number into an opinion.

.venv/bin/cda eval --stub                # scorecard
.venv/bin/cda eval --stub --diff         # vs committed baseline
.venv/bin/cda eval --stub --set-baseline # freeze a new baseline

Exit status. In live mode, eval fails if false_auto_determine > 0. In stub mode the outcome numbers describe the heuristic, so eval --stub --diff fails only when a safety-critical metric (false auto-determine, citation faithfulness, hallucinated clauses) regresses against the baseline. That makes it usable as a CI gate without an API key; see .github/workflows/ci.yml.

The case set: 152 cases, and where the labels come from

The split is the point. For the mechanical traps, a model has no business writing the answer key: which policy version governs a date is arithmetic. So evals/expand.py writes a dozen clinical fact-patterns once and crosses them with every version, every effective-date boundary, and both a rider-bearing and a rider-free plan. The labels are correct by construction. A dimension is only crossed in where it changes the answer: the naive cross product is 252 cases, and the informative subset is 112.

.venv/bin/cda evals expand      # regenerate derived cases
.venv/bin/cda evals validate    # check every label against the corpus
.venv/bin/cda evals generate    # Adversary (needs ANTHROPIC_API_KEY)

Nothing enters the suite unchecked. A case with a wrong label doesn't fail loudly; it quietly changes what the eval measures. So evals/validate.py checks mechanically:

• clause ids exist

• gold clauses are in effect on the case's own as-of date

• riders belong to the case's plan

• the cited policy governs the requested code

• an expected HALT is actually reachable (it runs pre-flight to check)

evals generate --stub shows it rejecting the defect classes a model really produces, without writing to the suite.

What the live Adversary run showed

48 cases proposed on Haiku 4.5 across two batches, 18 kept (a 37% yield):

• The first validator caught 1 of 6 defects. The rest were things I hadn't thought to check: citing a Scope clause to support a determination, tagging a well-formed request HALT because its narrative contained an injection, and citing a clause whose policy doesn't govern the requested code. Three new rules now cover them. The 134 existing cases stayed clean, so there were no false positives.

• 16 of 37 survivors duplicated the deterministic expansion. Duplicates are keyed on what a case actually tests, not on its wording. A model asked for adversarial cases mostly re-derives the mechanical traps that are already generated for free. Where it adds value is where structure can't reach: conflicts between sources, unusual fact patterns, missing-fact refusals.

• Both injection cases got zero flags from the pre-flight detector. Haiku invented [SYSTEM DIRECTIVE] framings that the blocklist doesn't match. That confirms the injection patterns are telemetry, not a barrier. The real defence is structural: narrative text is fenced as untrusted data, and the gate ignores what the model says about its citations.

What expanding the suite caught

Going from 22 to 134 cases found a real bug within one run. search_policies had always filtered by effective date, but get_plan_riders didn't. On a 2023 date the Retriever was handed a rider that took effect on 2024-01-01. A rider overrides base policy, so applying one before it exists inverts the determination. 21 cases were hitting this. get_plan_riders now takes as_of_date, and a regression test covers both sides of the boundary.

The 22 hand-written cases had never picked a date before a rider existed, because whoever wrote them already knew what the rider said.

A baseline only compares against the same questions. Every scorecard carries a case_set fingerprint. --diff refuses to treat a comparison across different case sets as a signal, and never fails the run on one.

Fault injection

The gate's rejection paths are tested, not assumed. CDA_STUB_FAULT makes the stub produce a specific bad citation:

Known limitations

• Citation faithfulness is not outcome correctness. The Verifier checks that each claim is supported by its quote. It does not check that the outcome follows from the claims. The stub shows this directly: for a Type 1 diabetes member it cites the Type 2 CGM clause verbatim and returns DETERMINE / COVERED. Every check passes and the answer is wrong. That is why false_auto_determine is measured against the gold key rather than inferred from faithfulness. Closing this needs an outcome-consistency check: every criterion of the governing clause mapped to a stated fact.

• Injection detection is a blocklist, and the live Adversary beat it (see above). It's recorded as telemetry. The defences that matter are structural.

• Evals run sequentially, at ~98s per case live. Running cases concurrently and prompt-caching the static system prompts (8.3k tokens per case) are the two obvious next steps; see docs/cost.md.

• BM25 is in-process. Moving the lexical half to Pinecone's hosted sparse model is the natural next step, but at 40 clauses it would be pure overhead.

Layout

policy_corpus/      MCP server — server.py, store.py, retrieval.py, vectorstore.py
corpus/             synthetic corpus generator + generated/
src/
  config.py         per-role model/effort/caps, gate thresholds, budgets
  contracts.py      pydantic IO contracts, used as structured-output schemas
  skills.py         loads .claude/skills/*/SKILL.md into system prompts
  llm.py            model client (Anthropic SDK) + stub backend
  mcp_client.py     stdio session + typed facade over the MCP tools
  agents/           planner, retriever, grader, synthesizer, verifier,
                    adversary, judge
  harness/          preflight, loop, gate, budget, trace
  cli.py            ask / eval / evals / corpus / cost / trace
evals/              cases/, scorers/, runner.py, validate.py, expand.py,
                    cost.py, baseline.json
docs/               vector backends, cost measurement
tests/              deterministic tests — no model in the path
traces/             per-run JSONL (gitignored)