Skip to main content
Glama
master997

cuad-audit

by master997
OverviewSchemaRelated ServersScoreDiscussions
Python
Local

cuad-audit

CI License: MIT

An MCP server that audits a contract liability clause against a derived company standard, and only produces a verdict when it can point at the exact retrieved evidence it relied on, and that verdict has passed a faithfulness check against that evidence. When the evidence is too weak, it says so — insufficient-grounding is a first-class result, not an error.

This is a study in when an agent should abstain, built as a small, fully-tested MCP server over a real legal-contracts dataset (CUAD, CC BY 4.0).

Quickstart

uv sync
make demo

make demo runs two pinned fixtures through audit_clause and prints the raw tool JSON — no MCP wiring, no full dataset download (the Chroma index auto-builds in seconds from a committed slice of the data). Without ANTHROPIC_API_KEY set, it runs the abstain case only (gate 1 needs no LLM) and tells you so:

{
  "verdict": "insufficient-grounding",
  "reason": "retrieval evidence too weak to ground a verdict (top similarity 0.423 < threshold 0.64) — escalate to a human reviewer",
  "citation": null,
  "gate1_score": 0.4228,
  "gate1_threshold": 0.64,
  "faithfulness": null,
  "failure_cause": "gate1"
}

With a key set, it also runs a cited verdict case (a mutual 3x work-order liability cap) and prints acceptable/risky/off-standard with a chunk_id citation the server resolved to an exact precedent span.

Related MCP server: Enterprise Financial Compliance Audit Framework

Architecture

            ┌──────────────────┐
   agent ──▶│   MCP server     │  stdio; logging to stderr only
            │   (server.py)    │  (stdout is reserved for the protocol)
            └──┬──────┬─────┬──┘
   search_clauses  get_standard  audit_clause (reuses both)
        │              │              │
        ▼              ▼              ▼
   ┌──────────┐  ┌───────────┐  ┌─────────────────────────────┐
   │BM25 +    │  │standard,   │  │ GATE 1 (pre-LLM): cosine    │
   │cosine    │  │derived from│  │   evidence score < 0.64     │
   │(Chroma)  │  │15 read     │  │   → insufficient-grounding  │
   │          │  │clauses     │  │ verdict LLM (Sonnet, t=0)   │
   └────┬─────┘  └───────────┘  │ GATE 2 (post-LLM): Haiku     │
        │                       │   faithfulness judge         │
        ▼                       └─────────────────────────────┘
   CUAD liability spans (Cap On Liability / Uncapped Liability)
   → chunked with stable chunk_ids

The three tools

  • search_clauses(query, clause_type="liability", k=5) — BM25-ranked precedent chunks (each with a stable chunk_id, source contract, char span, and score), gated by a cosine evidence-confidence score. Below threshold returns status: "below_threshold" with the scores — an abstention, not an error.

  • get_standard(clause_type="liability") — the liability "playbook": six positions (P1–P6, e.g. mutuality, cap basis, carve-outs) derived from 15 hand-read CUAD clauses, with provenance. Explicitly scoped — not legal advice, not corpus-wide extraction.

  • audit_clause(incoming_clause, clause_type="liability") — reuses both tools above, then runs the two-gate grounding contract below.

The grounding contract (two gates, both in code)

  1. Gate 1 — pre-LLM evidence gate. A leave-one-out cosine similarity score is checked before any LLM call. If the best match is below 0.64 (calibrated against a 13-query negative set — gibberish, out-of-domain clauses, cross-referenced caps), the tool returns insufficient-grounding immediately. No API key needed for this path.

  2. Gate 2 — post-LLM faithfulness judge. A single cheap Haiku call decomposes the verdict's reasoning into claims and checks each is supported by the cited chunk and the standard. An unfaithful verdict is downgraded to insufficient-grounding and counted as a hallucination — never silently shipped.

Citations are chunk_id lookups, not string matching: the verdict LLM picks an id from the evidence it was shown, and the server resolves it to the exact span. If the LLM names an invalid verdict or a chunk_id outside the retrieved evidence, that's also caught and downgraded to insufficient-grounding.

escalate-infra (API timeout/rate-limit/malformed output/refusal) is a separate verdict from insufficient-grounding — "the system is honest" and "the API is flaky" are never conflated.

Connect to Claude Code

claude mcp add cuad-audit -- uv run --directory /path/to/luminance python -m cuad_audit.server
claude mcp list   # health check — should show cuad-audit as connected

Then ask Claude Code to call search_clauses, get_standard, or audit_clause. Tool descriptions document the abstain semantics — agents should relay insufficient-grounding / below_threshold / escalate-infra verbatim, not retry until they get a verdict.

Symptom

Likely cause

Fix

Server doesn't appear in claude mcp list

wrong --directory or uv not on PATH

run uv run python -m cuad_audit.server directly from the repo root and check stderr

audit_clause always returns escalate-infra

ANTHROPIC_API_KEY not set

export ANTHROPIC_API_KEY=... (search_clauses/get_standard still work without it)

First call is slow / looks hung

first-run embedding model download (~90 MB)

progress prints to stderr; subsequent runs are cached

Setup

  • Python 3.11+, dependency management via uv with a committed lockfile (uv sync).

  • Embedding model: sentence-transformers/all-MiniLM-L6-v2 (~90 MB, downloaded once and cached).

  • Tested on macOS and Linux.

  • First make demo / make ingest: ~1–2 minutes (model download + index build from the committed slice of 680 chunks). Subsequent runs: seconds. Measured smoke test (uv sync && make demo && make eval-retrieval && make test, warm model cache): 97s total, 33 passed + 1 skipped (the skipped test needs ANTHROPIC_API_KEY).

  • ANTHROPIC_API_KEY — only required for audit_clause's verdict + judge calls and make eval-verdicts. Not required for search_clauses, get_standard, the abstain half of make demo, make eval-retrieval, or make calibrate.

Reproducing the measurements

make eval-retrieval   # keyless, deterministic — retrieval vs CUAD spans + kill criteria
make calibrate        # keyless — gate-1 threshold calibration distributions
make eval-verdicts    # needs ANTHROPIC_API_KEY — ~70 LLM calls, a few dollars

Retrieval vs CUAD expert spans (held-out split, docs/DAY2_RESULTS.md)

166 held-out queries against 680 library chunks (80% Cap On Liability / 20% Uncapped Liability):

Metric

BM25

MiniLM (cosine)

precision@1 (overall)

0.741

0.705

precision@1 (Uncapped Liability, n=34)

0.559

0.500

success@3 (≥1 relevant in top 3)

0.970

0.970

Embeddings did not beat the keyword baseline, so retrieval ranking and citations are BM25-first (pre-registered Day-2 rule); the semantic index stays in the repo, tested, as the measured comparison and feeds gate 1.

Gate-1 threshold (docs/DAY3_CALIBRATION.md)

Calibrated on leave-one-out cosine top-scores: library positives (n=669, p10=0.64) vs a 13-query negative set (gibberish, out-of-domain clauses, cross-referenced caps). Threshold 0.64 catches 8/13 negatives outright; the remaining 5 (cross-referenced caps, near-domain insurance/audit-rights text) are real liability-adjacent text with no auditable content — owned by the verdict path (standard position P5), not by gate 1. See docs/gate1_calibration.png.

Verdicts vs hand-labeled standard (docs/EVAL.md, results: docs/DAY5_RESULTS.md)

35-item eval set (28 hand labels + 3 cross-reference cases + 3 capped/uncapped confusion pairs + 1 prompt-injection probe), run via make eval-verdicts. Reported as counts and a failure taxonomy, never a headline accuracy — n is too small for that, and the report says so. Columns distinguish grounding abstentions (justified vs unjustified) from infra abstentions (API failures).

One live run (2026-06-10; ±1–2 expected on re-run):

n

% of 28

Non-abstained verdict

11

39%

Grounding abstain — justified (R9)

1

4%

Grounding abstain — unjustified

16

57%

Infra abstain

0

0%

6/11 non-abstained verdicts matched the hand label exactly. Of the 16 unjustified abstentions, 14 came from the gate-2 faithfulness judge — a hand-verified sample found the judge, not the verdict LLM, was usually the weak link (rejecting reasonable inferential claims as "unsupported"). Citations were valid on 100% of non-abstained verdicts; adversarial defenses (cross-referenced caps + prompt-injection probe) held 4/4. Full taxonomy, root-cause analysis, and "what I'd do next": docs/DAY5_RESULTS.md. One-page project writeup: docs/WRITEUP.md.

Known limitations (named on purpose)

  • Polarity risk: "Cap on Liability" and "Uncapped Liability" are a negation pair that embedding similarity can confuse. Gate 1 measures evidence strength, not correctness — the verdict LLM owns the capped/uncapped call, and confusion pairs are in the adversarial eval set.

  • Tool-level vs agent-level grounding: the server cannot stop a client agent from speculating after an insufficient-grounding result. The demo harness instructs verbatim relay and shows raw tool output.

  • Single lane (liability), single segmenter (CUAD's expert spans, not a production clause segmenter), standard derived from 15 read clauses — all scoped claims, not corpus-wide extraction. See PLAN.md for the full "Not in Scope" list and rationale.

  • The faithfulness judge (gate 2) is itself an ungated LLM call; a hand-verified sample of judge outputs is reported alongside the eval.

Project layout

src/cuad_audit/
  download.py       CUAD v1 download (pinned sha256)
  derive_slice.py   reproduces the committed data slice byte-identically
  ingest.py         chunking, token-length checks, Chroma index build
  retrieval.py      BM25 (KeywordIndex) + cosine (SemanticIndex)
  llm.py            CompleteFn seam — typed failures, no silent fallbacks
  audit.py          the three tools + both gates
  server.py         MCP stdio entrypoint
  demo.py           make demo
  calibrate.py      gate-1 threshold calibration
  eval_retrieval.py make eval-retrieval
  eval_verdicts.py  make eval-verdicts (resumable JSONL)
data/               committed slice (split, standard, labels, chunks)
docs/               split methodology, rubric, eval definitions, results
tests/              34 tests, LLM seam fully mocked — CI is free

Data & attribution

Built on the Contract Understanding Atticus Dataset (CUAD) v1, © The Atticus Project, licensed under CC BY 4.0. This repo commits a small derived slice (liability-clause spans, data/liability_spans_all.json and data/split.json) for reproducibility; make ingest can re-derive the index from a fresh download via src/cuad_audit/download.py.

Code is MIT licensed.

Available Tools

3 tools
audit_clauseA

Audit a clause against the company standard with grounded citations.

Verdicts: acceptable / risky / off-standard (each carries a citation chunk_id resolved by the server to the exact precedent span); insufficient-grounding (evidence too weak OR the verdict failed the faithfulness check — escalate to a human, do NOT guess a verdict yourself); escalate-infra (API failure — retry later; not a judgment about the clause). Relay abstentions verbatim; they are first-class results.

ParametersJSON Schema
NameRequiredDescriptionDefault
incoming_clauseYesThe clause text to audit (one clause, max 8000 chars)
clause_typeNoClause lane to audit against. Only 'liability' is implemented.liability

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description fully discloses behavior: possible verdicts, server resolution of citations, and instructions for handling special outcomes (insufficient-grounding, escalate-infra). It provides rich behavioral context beyond the schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear lead sentence followed by a bullet-like list of verdicts. Every sentence adds value, though slightly verbose. It is front-loaded and concise enough for an AI agent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (multiple verdicts, no output schema), the description is highly complete. It explains each verdict, how citations work, and how to handle abstentions. It covers all necessary behavioral aspects for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so baseline is 3. The description does not add additional meaning to the parameters beyond what the schema already provides (e.g., max chars for incoming_clause, const/default for clause_type). No extra value added.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool audits a clause against the company standard with grounded citations. It specifies the verb 'audit' and the resource 'clause', and distinguishes from siblings 'get_standard' and 'search_clauses' by emphasizing grounded citations and specific verdict types.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit guidance on when to use the tool and how to handle each verdict: it warns against guessing for 'insufficient-grounding' and instructs to retry for 'escalate-infra'. It also instructs to relay abstentions verbatim, making usage clear and actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_standardA

Return the company standard (playbook) for a clause type: positions P1-P6 with provenance. The standard is data derived from the clause library, not legal advice; see scope_claim in the result.

ParametersJSON Schema
NameRequiredDescriptionDefault
clause_typeNoClause lane to audit against. Only 'liability' is implemented.liability

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description carries burden. It discloses that the standard is data derived from the clause library, not legal advice, and references scope_claim in the result. However, it lacks details on idempotency, authentication needs, or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with main action, no wasted words. The caveat is placed efficiently.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (1 param, no output schema, no annotations), the description covers the key aspects: what is returned, disclaimers, and a reference. It is complete for its scope.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter, and description adds meaning by mentioning 'positions P1-P6 with provenance' and 'see scope_claim in the result', which goes beyond the schema's const definition.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it returns the company standard (playbook) for a clause type, specifying positions P1-P6 with provenance. It distinguishes the tool's purpose but does not explicitly differentiate from sibling tools like audit_clause or search_clauses.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives, no exclusions or context provided. Sibling tools exist but no comparison is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_clausesA

Search the precedent clause library (CUAD liability spans).

Returns chunks with stable chunk_ids, source contract, char offsets and scores. If evidence is too weak (status: "below_threshold"), that is an abstention, not an error: do NOT retry with rephrased queries to force a result — relay the abstention. top_scores shows how far below the threshold the best matches were.

ParametersJSON Schema
NameRequiredDescriptionDefault
queryYesFull clause text to find precedents for. Pass the complete clause: short paraphrases score lower against the evidence gate and abstain more often (the gate is calibrated on full clause texts).
clause_typeNoClause lane to audit against. Only 'liability' is implemented.liability
kNo

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description covers key behaviors: returns stable chunk IDs, source, offsets, scores; explains 'below_threshold' means abstention (not error); mentions top_scores. No destructive implications or contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four short sentences, front-loaded with the main purpose, each sentence adds specific value. No fluff or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema or annotations, the description is fairly complete but lacks explanation for the k parameter (only in schema with constraints) and does not fully describe the return structure beyond listing fields. Could be more thorough.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67%, and the description adds value: warns against short paraphrases for query, clarifies clause_type constraint. The k parameter lacks description, but overall the description enriches the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool searches a precedent clause library for CUAD liability spans, and lists what it returns (chunks with IDs, source, offsets, scores). It distinguishes from sibling tools by being explicitly a search function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides strong usage guidance: warns against retrying on weak evidence, advises passing complete clause texts to avoid abstentions. Does not explicitly contrast with siblings, but gives actionable instructions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

TDQS

A4.2/5.0
Disambiguation5/5

Each tool has a distinct purpose: audit_clause evaluates clauses, get_standard retrieves the standard playbook, search_clauses searches precedents. There is no overlap or ambiguity.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern: audit_clause, get_standard, search_clauses. No mixing of conventions.

Tool Count5/5

Three tools is appropriate for the focused scope of clause auditing. Each tool serves a necessary role without being redundant or insufficient.

Completeness5/5

The tool set covers the full workflow: retrieving the standard, searching precedents, and auditing clauses. There are no obvious gaps for the stated purpose.

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • hivelaw

    srotzin
    A
    license
    Not graded
    quality
    C
    maintenance
    MCP server for AI compliance auditing. Scores agent outputs for hallucination liability under the EU AI Act, issues verifiable compliance stamps, and tracks audit history by agent.
    MIT

  • Grounding Enforcer

    steveswain14
    F
    license
    Not graded
    quality
    C
    maintenance
    A standalone MCP server that validates model output against retrieved sources. It flags any claim, statistic, attribution, quote, or URL that cannot be traced back to a real source.
  • RecoSearch by recohut

    RecoSearchofficial

    recohut
    A
    license
    Not graded
    quality
    C
    maintenance
    A deterministic MCP server that governs read-only queries across multiple data sources, returning answers with full provenance (every row cited) or a typed refusal, ensuring LLM answers are traceable and contract-enforced.
    1
    Apache 2.0

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/master997/luminance'

If you have feedback or need assistance with the MCP directory API, please join our Discord server