Skip to main content
Glama

Bankstatemently

Evaluate Benchmark

evaluate_benchmark
Read-only

Score parsed bank statement transactions against the Bankstatemently benchmark ground truth. Accepts a statement_id (e.g. "bsb-001") or content_hash, plus your parsed transactions. Returns extraction accuracy, integrity score, and an overall score. Only statements marked published: true in the catalog can be evaluated — held-out statements return an error. transactions[].originalData is optional but strongly recommended: fetch it via get_statement with data_mode: "original" and pass it through verbatim — an absent originalData scores that transaction's raw-fidelity (parsed) dimension 0; never fabricate a value. Free to use — no credits consumed. Read the benchmark://catalog resource first to see available statements and their published status.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
accountsNoOptional account roster for multi-account statements. Each transaction references one via accountId.
content_hashNoSHA-256 hex digest of the PDF. Use statement_id instead if you know it.
statement_idNoBenchmark statement ID (e.g. "bsb-001"). Preferred over content_hash.
transactionsYesParsed transactions (1-2000)

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault
idYes
challengesYes
difficultyYes
contentHashNo
parsedScoreYes
datasetVersionNo
normalizedScoreYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed9 schema fields changed
    • addedOutput schema / $defs
      Added value: +{
      +  "__schema0": {
      +    "anyOf": [
      +      {
      +        "type": "string"
      +      },
      +      {
      +        "type": "number"
      +      },
      +      {
      +        "type": "boolean"
      +      },
      +      {
      +        "type": "null"
      +      },
      +      {
      +        "items": {
      +          "$ref": "#/$defs/__schema0"
      +        },
      +        "type": "array"
      +      },
      +      {
      +        "additionalProperties": {
      +          "$ref": "#/$defs/__schema0"
      +        },
      +        "propertyNames": {
      +          "type": "string"
      +        },
      +        "type": "object"
      +      }
      +    ]
      +  }
      +}
    • addedOutput schema / properties / contentHash
      Added value: +{
      +  "type": "string"
      +}
    • addedOutput schema / properties / datasetVersion
      Added value: +{
      +  "type": "string"
      +}
    • addedOutput schema / properties / normalizedScore / additionalProperties / $ref
      Added value: +"#/$defs/__schema0"
    • removedOutput schema / properties / normalizedScore / properties
      Removed value: -{}
    • addedOutput schema / properties / normalizedScore / propertyNames
      Added value: +{
      +  "type": "string"
      +}
    • addedOutput schema / properties / parsedScore / additionalProperties / $ref
      Added value: +"#/$defs/__schema0"
    • removedOutput schema / properties / parsedScore / properties
      Removed value: -{}
    • addedOutput schema / properties / parsedScore / propertyNames
      Added value: +{
      +  "type": "string"
      +}
  2. Changed18 schema fields changed
    • changedInput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
    • addedInput schema / properties / accounts / description
      Added value: +"Optional account roster for multi-account statements. Each transaction references one via accountId."
    • addedInput schema / properties / accounts / items / properties / accountNumber / description
      Added value: +"Verbatim printed account number — never normalized by the submitter."
    • addedInput schema / properties / accounts / items / properties / currency / description
      Added value: +"ISO 4217 currency code."
    • addedInput schema / properties / accounts / items / properties / id / description
      Added value: +"Submission-internal handle, referenced by transactions[].accountId."
    • addedInput schema / properties / accounts / items / properties / name / description
      Added value: +"Verbatim printed account label."
    • addedInput schema / properties / content_hash / description
      Added value: +"SHA-256 hex digest of the PDF. Use statement_id instead if you know it."
    • addedInput schema / properties / statement_id / description
      Added value: +"Benchmark statement ID (e.g. \"bsb-001\"). Preferred over content_hash."
    • addedInput schema / properties / transactions / description
      Added value: +"Parsed transactions (1-2000)"
    • addedInput schema / properties / transactions / items / properties / accountId / description
      Added value: +"References accounts[].id — the account this transaction belongs to. Omit for single-account statements."
    • addedInput schema / properties / transactions / items / properties / amount / description
      Added value: +"Transaction amount. Negative = debit, positive = credit (or use direction)."
    • addedInput schema / properties / transactions / items / properties / balance / description
      Added value: +"Running balance after this transaction, if known."
    • addedInput schema / properties / transactions / items / properties / currency / description
      Added value: +"ISO 4217 currency code for this transaction, if known."
    • addedInput schema / properties / transactions / items / properties / date / description
      Added value: +"ISO 8601 date (YYYY-MM-DD)"
    • addedInput schema / properties / transactions / items / properties / description / description
      Added value: +"Transaction description as printed on the statement."
    • addedInput schema / properties / transactions / items / properties / direction / description
      Added value: +"Explicit direction. If omitted, inferred from amount sign."
    • addedInput schema / properties / transactions / items / properties / originalData / description
      Added value: +"Raw column values as on the PDF. Omit if unavailable — never fabricate a value; an absence scores the parsed dimension's raw fields 0 rather than polluting the measurement."
    • changedOutput schema / $schema
      Previous value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
  3. Changed16 schema fields changed
    • removedInput schema / properties / accounts / description
      Removed value: -"Optional account roster for multi-account statements. Each transaction references one via accountId."
    • removedInput schema / properties / accounts / items / properties / accountNumber / description
      Removed value: -"Verbatim printed account number — never normalized by the submitter."
    • removedInput schema / properties / accounts / items / properties / currency / description
      Removed value: -"ISO 4217 currency code."
    • removedInput schema / properties / accounts / items / properties / id / description
      Removed value: -"Submission-internal handle, referenced by transactions[].accountId."
    • removedInput schema / properties / accounts / items / properties / name / description
      Removed value: -"Verbatim printed account label."
    • removedInput schema / properties / content_hash / description
      Removed value: -"SHA-256 hex digest of the PDF. Use statement_id instead if you know it."
    • removedInput schema / properties / statement_id / description
      Removed value: -"Benchmark statement ID (e.g. \"bsb-001\"). Preferred over content_hash."
    • removedInput schema / properties / transactions / description
      Removed value: -"Parsed transactions (1-2000)"
    • removedInput schema / properties / transactions / items / properties / accountId / description
      Removed value: -"References accounts[].id — the account this transaction belongs to. Omit for single-account statements."
    • removedInput schema / properties / transactions / items / properties / amount / description
      Removed value: -"Transaction amount. Negative = debit, positive = credit (or use direction)."
    • removedInput schema / properties / transactions / items / properties / balance / description
      Removed value: -"Running balance after this transaction, if known."
    • removedInput schema / properties / transactions / items / properties / currency / description
      Removed value: -"ISO 4217 currency code for this transaction, if known."
    • removedInput schema / properties / transactions / items / properties / date / description
      Removed value: -"ISO 8601 date (YYYY-MM-DD)"
    • removedInput schema / properties / transactions / items / properties / description / description
      Removed value: -"Transaction description as printed on the statement."
    • removedInput schema / properties / transactions / items / properties / direction / description
      Removed value: -"Explicit direction. If omitted, inferred from amount sign."
    • removedInput schema / properties / transactions / items / properties / originalData / description
      Removed value: -"Raw column values as on the PDF. Omit if unavailable — never fabricate a value; an absence scores the parsed dimension's raw fields 0 rather than polluting the measurement."
  4. First observed

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds meaningful behavioral context beyond that: the published-only restriction, the held-out error behavior, the scoring rule that absent originalData zeroes the raw-fidelity dimension, the explicit 'never fabricate a value' instruction, and the 'free to use — no credits consumed' note. These are exactly the behavioral traits an agent needs to know and cannot infer from annotations or schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense but every sentence earns its place: purpose, identifiers, return values, published-only constraint, originalData guidance, cost note, and catalog pointer. It is front-loaded with the core purpose and the most decision-relevant constraint (published-only) appears early. It loses one point for being slightly long and for burying the 'read the catalog first' instruction at the end, which is arguably the first thing an agent should do.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a benchmark-evaluation tool with a rich output schema, the description covers everything an agent needs to call it correctly: what to pass, which identifier to prefer, the published-only gate, the originalData fidelity rule, the no-fabrication constraint, and the cost implication. The output schema exists, so the description needn't enumerate return fields. The only minor gap is that it doesn't explicitly say what happens if both statement_id and content_hash are omitted, but the schema's required field is transactions, so that's not a real gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all four parameters well. The description adds value by clarifying the relationship between statement_id and content_hash ('Use statement_id instead if you know it'), by explaining the originalData scoring consequence, and by emphasizing that originalData should be passed verbatim. It doesn't add much about accounts or transactions, but the schema already covers those thoroughly, so the baseline-3-plus-incremental-value score of 4 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb ('Score') and resource ('parsed bank statement transactions against the Bankstatemently benchmark ground truth'), and immediately distinguishes this from sibling tools like rate_statement or categorize_statement by naming the benchmark context. It also names the two accepted identifiers (statement_id, content_hash) and the returned metrics, so an agent can tell exactly what this tool does and what it produces.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit when-to-use guidance: only statements marked published: true can be evaluated, held-out statements error, and the agent should read the benchmark://catalog resource first. It also gives a concrete workflow hint — fetch originalData via get_statement with data_mode: 'original' and pass it through verbatim — which routes the agent to the correct sibling tool and mode. This is unusually actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.