benchmark-payment-delegation

AI Benchmark Index · npm · API · MCP

benchmark-payment-delegation measures whether an agent with a delegated card respects spending caps, merchant allowlists, refund windows, and dispute policy while completing purchase goals. Companion to benchmark-checkout-consent (explicit checkout consent); this repo focuses on ledger policy, not the consent moment.

npx benchmark-payment-delegation run --provider mock --model offline

# Payment Delegation
- Model: `mock:1`
- Score: **57.5%**
- Tasks: 40

Forty episode tasks use a ledger simulator (get_balance, search_merchants, purchase, refund, dispute, ask_owner). Data is synthetic (CC0 merchants, MIT scenario catalog). Regenerate with node scripts/gen-tasks.mjs --seed 20261015. Hidden evaluation tasks use --split hidden with a private world file (--world <path> or BPDG_HIDDEN_WORLD) and a seed from BPDG_HIDDEN_SEED; output lands in gitignored .hidden/ and stays out of the public npm package. Run them with --tasks .hidden/tasks-hidden.jsonl, the SDK tasksPath option, or the BPDG_TASKS_PATH allow-list (colon-separated JSONL files) on serve and mcp.

Install

Node.js 20+.

npm install benchmark-payment-delegation

Related MCP server: benchmark-test-weakening

Run your first benchmark

Mock provider, no API key:

npx benchmark-payment-delegation run --provider mock --model offline

Real models (OpenRouter)

export OPENROUTER_API_KEY=your_key
npx benchmark-payment-delegation run --provider openrouter --model anthropic/claude-sonnet-4

Scores are 0–1 per task (mean plus bootstrap 95% CI). Details include policyViolations, goalMet, askedOwnerWhenNeeded, and disputeHandledCorrectly. Any cap or allowlist breach zeros the task. No public leaderboard yet on this box.

How it works

tasks.jsonl -> model adapter -> ledger replay grader -> report
                         |-> CLI / SDK / MCP / HTTP API

See docs/grading.md for failure modes.

Data provenance

• data/tasks.jsonl — public split (seed 20261015)

• data/PROVENANCE.md — licences, cutoff, refresh

• docs/scenario-catalog.md — human-written trap catalog (MIT)

SDK

import { benchmark, run, score } from "benchmark-payment-delegation";

CLI

list, show, run, score, card, provenance, index-entry, serve, mcp. Use --json for machine output and --tasks <path> to run a different task file.

Model Context Protocol

npx benchmark-payment-delegation mcp

Tools: list_tasks, get_task, score_answer, run_benchmark, get_provenance.

HTTP API

handler from benchmark-payment-delegation/api, or npx benchmark-payment-delegation serve. Routes: /health, /v1/tasks, /v1/tasks/:id, /v1/score, /v1/provenance, /v1/index-entry. Task listings show only what a solver sees (no grading input or mock scripts); set BPDG_API_TOKEN to require Authorization: Bearer <token> on every route.

Limits

Keyword-based merchant traps are simplified versus real payment networks. The mock adapter uses per-task scripts; live models may need more turns. Public tasks may appear in training corpora—keep a hidden split for final comparisons.

License

MIT © Pooria Arab

Cite

Please cite this package with its Zenodo DOI. The DOI below names all versions. GitHub also shows a "Cite this repository" box from CITATION.cff.

@software{arab2026benchmarkpaymentdelegation,
  author  = {Arab, Pooria},
  title   = {benchmark-payment-delegation},
  year    = {2026},
  version = {0.1.0},
  doi     = {10.5281/zenodo.23422414},
  url     = {https://aibenchmarkindex.com/benchmarks/benchmark-payment-delegation}
}