Skip to main content
Glama

xVendorBench — Vendor Shortlist Benchmark

xvendorbench
DestructiveIdempotent

Compare a vendor shortlist against your constraints, with source-backed strengths, gaps and tradeoffs.

Input Schema

NameRequiredDescriptionDefault
inputsYesOriginal buyer inputs. Service-specific CSV/date/rubric validation and a combined 75,000-character limit also apply.
sourceFieldsNo
sourceRunIdsNo
idempotencyKeyYesStable caller key for exactly one logical service request.
paymentInvoiceIdNoA separate confirmed service invoice when payment is required.
consentExternalProcessingYesExplicit authorization to send these sources and prior-agent context to the selected inference provider.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already flag destructiveHint=true, openWorldHint=true and idempotentHint=true, yet the description says nothing about what is destroyed or sent externally. It omits the explicit external-processing consent requirement, the idempotency key, and the payment/invoice path — all significant for a tool the annotations mark as destructive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single tight sentence with the key concept front-loaded and no filler. It is efficient, though arguably too thin for a tool with six parameters, nested objects, and destructive semantics.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with nested inputs, a mandatory external-processing consent flag, and destructive annotations, one sentence is inadequate. Nothing is said about what the benchmark returns, how sources are handled outside the system, or what the caller must prepare beyond a shortlist.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 67% and the schema itself documents the vendors, constraints, datedEvidence CSV format, and consent fields in detail. The description adds only the framing 'against your constraints,' contributing no syntax, format, or validation detail beyond what the schema already provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (compare) and resource (a vendor shortlist) plus the output shape (source-backed strengths, gaps, tradeoffs). It is distinguishable from siblings like xmatch-supplier-shortlist or xscore-bid-evaluation in spirit, but never names them or explicitly delineates the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no when-to-use guidance, no prerequisites, and no alternatives among the many sibling procurement tools. An agent must infer from the name alone whether this replaces or complements xmatch-supplier-shortlist, xscore-bid-evaluation, or supplier-risk.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources