Skip to main content
Glama
DisputeDesk MCP Server

DisputeDesk MCP Server

DisputeDesk

Multi-agent card-dispute resolution with MCP tools, RAG, guardrails, observability and evals built in. LangGraph + FastAPI, runs locally, deploys to AWS.

Portfolio project. All data is synthetic. See PROJECT_BLUEPRINT.md for the architecture and milestones.

Status

See PROGRESS.md for milestone status and details.

Related MCP server: Agentic RAG MCP Server

Quickstart

Requires uv and Docker.

cp .env.example .env
docker compose up -d --wait      # Postgres+pgvector, Langfuse (:3000), OTel collector (:4317/4318)
uv sync --all-extras --dev
uv run pytest -q

LLM_PROVIDER=fake (the default in .env.example) needs no API key. Set LLM_PROVIDER=anthropic and ANTHROPIC_API_KEY to use Claude, or LLM_PROVIDER=ollama for a local model.

Evals

uv run python evals/run_evals.py --thresholds evals/thresholds.yml runs 42 synthetic disputes (10 adversarial) through the real graph and exits non-zero if any threshold fails. Needs a real model and the rulebook ingested (uv sync --all-extras --dev --group evals, uv run python -m disputedesk.rag.ingest). A full run takes about 30 minutes.

Latest result (Haiku 5.5 as system and judge, 2026-11-20). The gate is red (3 of 9 thresholds fail): the system is not good enough yet, and the thresholds were fixed before the first run. Details, causes and what changed between runs are in PROGRESS.md.

Metric

Baseline

Now

Threshold

Router accuracy

1.000

1.000

≥ 0.90

Decision accuracy (strict)

0.133

0.781

≥ 0.80

Human-approval routing accuracy

1.000

1.000

≥ 1.00

Citation hit rate

0.133

0.969

≥ 0.70

Citation faithfulness (DeepEval)

0.952

0.969

≥ 0.85

Letter quality (LLM judge, 1-5)

3.23

4.03

≥ 4.00

Guardrail catch rate

0.333

0.333

≥ 0.80

Guardrail false-positive rate

0.033

0.031

≤ 0.05

Injection pass rate

0.400

0.900

≥ 1.00

Cases that crashed

20 of 40

0 of 42

Red-team: uv run npx promptfoo@0.126.0 eval -c evals/promptfoo/redteam.yaml --no-cache (16 of 18 pass after M7.2, up from 4; not re-run since).

Development

uv run ruff check . && uv run ruff format .
uv run mypy src
uv run pytest tests/unit -q
uv run pre-commit install        # optional: run the same checks on every commit

Architecture decisions

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • F
      license
      Not graded
      quality
      D
      maintenance
      A production-grade MCP server that exposes real-time banking data replicated via Oracle GoldenGate CDC as structured tools for AI agents, enabling read, score, and write operations on customer, account, transaction, and alert data.
      1
      -
    • F
      license
      Not graded
      quality
      D
      maintenance
      MCP server that provides tools for retrieving transaction context and recording AI decisions or creating human reviews for payment risk exceptions. Enables LLM agents to handle exception transactions in a hybrid payment-decisioning workflow.
      -
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP-native agentic platform orchestrating planner/executor/critic agents over hybrid RAG with three-tier memory, budget enforcement, safety guardrails, and full observability. It exposes all capabilities as MCP tools, enabling natural-language control of document ingestion, retrieval-augmented generation, and multi-step AI workflows.
      MIT