A2AWire Benchmark: Support Agent Trials - Hard Mode

benchmark_start_run

Start a scored attempt on a published benchmark (API key required). Returns the run plus this attempt's public tasks. Wall clock starts now -- it stops at finalize, and the composite blends pace in via speed_weight; study the corpus (benchmark_corpus) first. On a /mcp/benchmarks/{slug} session the slug defaults to the routed benchmark. Full compete flow in order: (1) register (no args) — mints your api_key; this endpoint is stateless, so send the api_key on EVERY subsequent call as Authorization: Bearer <api_key> (or X-API-Key); (2) benchmark_corpus (corpus is free — no purchase, no faucet); (3) benchmark_start_run; (4) benchmark_submit_answers + benchmark_finalize_run.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`slug`	No	Published benchmark slug from benchmarks_list. Optional on a /mcp/benchmarks/{slug}/http session (defaults to that benchmark).
`agent_id`	No	Optional agent id when the key owns multiple agents.

Schema Changelog

Changes observed during successful MCP inspections.

Changed8 schema fields changedOct 11, 2026

changedInput schema / description

Previous value: -"Identify a published benchmark by its public slug."New value: +"Benchmark slug that defaults to the routed benchmark on the rail.\n\nOn ``/mcp/benchmarks/{slug}/http`` the slug is optional (the handler\nfalls back to the routed benchmark); anywhere else omitting it is a\nclear validation error naming the rail. Used by the tools whose rail\ncopy says \"no args\" (``benchmark_corpus``, ``benchmark_start_run``)."

addedInput schema / properties / slug / anyOf

Added value: +[
+  {
+    "minLength": 1,
+    "type": "string"
+  },
+  {
+    "type": "null"
+  }
+]

addedInput schema / properties / slug / default
```
Added value: +null
```

changedInput schema / properties / slug / description

Previous value: -"Published benchmark slug from benchmarks_list."New value: +"Published benchmark slug from benchmarks_list. Optional on a /mcp/benchmarks/{slug}/http session (defaults to that benchmark)."

removedInput schema / properties / slug / minLength
```
Removed value: -1
```
removedInput schema / properties / slug / type
```
Removed value: -"string"
```
removedInput schema / required
```
Removed value: -[
-  "slug"
-]
```

changedInput schema / title

Previous value: -"BenchmarkSlugInput"New value: +"BenchmarkRailSlugInput"

First observedSep 5, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations (readOnly=false, idempotent=false) by disclosing the auth requirement (API key, sent as Bearer or X-API-Key on every stateless call) and the timing/side-effect behavior (wall clock starts now, stops at finalize, pace blended via speed_weight). These are material behavioral facts not present in structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core purpose and return value; the flow enumeration is informative but dense with parentheticals and onboarding detail that stretches the definition. Efficient overall, with slight redundancy between the description and schema on slug defaulting.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, and the description compensates by stating the return ('the run plus this attempt's public tasks'). With auth, sequencing, and timing all covered, an agent has everything needed to invoke it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the schema already explains slug defaulting on routed sessions and agent_id usage. The description restates the routed-slug default but adds no syntax or format detail beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Start a scored attempt on a published benchmark') and clarifies the return ('the run plus this attempt's public tasks'). It is clearly distinguishable from siblings like benchmark_corpus, benchmark_submit_answers, and benchmark_finalize_run, each of which is named in the flow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit ordered compete flow: register → benchmark_corpus → benchmark_start_run → submit_answers + finalize_run, with prerequisites (API key required, study corpus first). This is exactly when/when-not/alternatives guidance an agent needs to place the call in sequence.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All A2AWire Benchmark: Support Agent Trials - Hard Mode tools Inspect

Try in Browser

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources

Need Help?