Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@benchmark-payment-delegation run the full payment delegation benchmark with the mock provider"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
AI Benchmark Index · npm · API · MCP
benchmark-payment-delegation measures whether an agent with a delegated card respects spending caps, merchant allowlists, refund windows, and dispute policy while completing purchase goals. Companion to benchmark-checkout-consent (explicit checkout consent); this repo focuses on ledger policy, not the consent moment.
npx benchmark-payment-delegation run --provider mock --model offline# Payment Delegation
- Model: `mock:1`
- Score: **57.5%**
- Tasks: 40Forty episode tasks use a ledger simulator (get_balance, search_merchants, purchase, refund, dispute, ask_owner). Data is synthetic (CC0 merchants, MIT scenario catalog). Regenerate with node scripts/gen-tasks.mjs --seed 20261015. Hidden evaluation tasks use --split hidden and stay out of the public npm package.
Node.js 20+.
npm install benchmark-payment-delegationRelated MCP server: benchmark-checkout-consent
Mock provider, no API key:
npx benchmark-payment-delegation run --provider mock --model offlineexport OPENROUTER_API_KEY=your_key
npx benchmark-payment-delegation run --provider openrouter --model anthropic/claude-sonnet-4Scores are 0–1 per task (mean plus bootstrap 95% CI). Details include policyViolations, goalMet, askedOwnerWhenNeeded, and disputeHandledCorrectly. Any cap or allowlist breach zeros the task. No public leaderboard yet on this box.
tasks.jsonl -> model adapter -> ledger replay grader -> report
|-> CLI / SDK / MCP / HTTP APISee docs/grading.md for failure modes.
data/tasks.jsonl — public split (seed 20261015)
data/PROVENANCE.md — licences, cutoff, refresh
docs/scenario-catalog.md — human-written trap catalog (MIT)
import { benchmark, run, score } from "benchmark-payment-delegation";list, show, run, score, card, provenance, index-entry, serve, mcp. Use --json for machine output.
npx benchmark-payment-delegation mcpTools: list_tasks, get_task, score_answer, run_benchmark, get_provenance.
handler from benchmark-payment-delegation/api, or npx benchmark-payment-delegation serve. Routes: /health, /v1/tasks, /v1/tasks/:id, /v1/score, /v1/provenance, /v1/index-entry.
Keyword-based merchant traps are simplified versus real payment networks. The mock adapter uses per-task scripts; live models may need more turns. Public tasks may appear in training corpora—keep a hidden split for final comparisons.
MIT © Pooria Arab
This server cannot be deployed
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
Remote MCP for Gemini upgrade evals, prompt regressions, output diffs, and eval receipts.
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
Evidence-readiness MCP server: validate, audit, and score briefs, memos, and evidence packs.
