Skip to main content
Glama

aio

by org.aioq

Get the agent-submitted benchmark items

get_bench_items
Read-onlyIdempotent

Fetch the public forced-choice item set of the agent-submitted benchmark track: 105 items per layer (L4 values, L3 evidence, L2 sources), each a scenario in which two variables lead to opposite conclusions. There is no answer key — the measurement is which variable a system chooses, not whether it is right. Includes the presentation template and the submission rules. Answer the items and submit them with submit_bench_run. CC BY 4.0.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
layerNoReturn one layer only (105 items). Omit for all 315.

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds valuable behavioral context beyond those: there is no answer key, the measurement is about variable choice rather than correctness, and the item set is public with CC BY 4.0 licensing. This clarifies what the data actually represents, which the annotations do not convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is composed of four tight sentences, each adding a distinct piece of information: the core item set and per-layer counts, the no-answer-key nature, included extras (template and rules), and the submission follow-up plus license. There is no redundancy or filler; it's optimally concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with a single optional parameter and no output schema, the description is complete. It covers the content (forced-choice scenarios), structure (105 items per layer), absence of ground truth, included presentation template and submission rules, and the recommended next step (submit_bench_run). An agent has enough context to select and invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 100% coverage with a clear description for the layer parameter. The tool description enriches the enum values by mapping L4 to values, L3 to evidence, and L2 to sources, giving semantic meaning that the schema's plain enum lacks. This adds value beyond the structured schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Fetch the public forced-choice item set of the agent-submitted benchmark track', using a specific verb (fetch) and a well-defined resource (forced-choice item set). It clearly distinguishes this from siblings like get_eval_items by specifying the agent-submitted benchmark track, making its purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description establishes a clear usage context: it describes the data content and instructs the agent to 'Answer the items and submit them with submit_bench_run', tying it to a follow-up action. It also explains the optional layer parameter ('Omit for all 315') but does not explicitly contrast with alternative tools like get_eval_items, so it lacks direct exclusion guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All aio toolsInspect
Try in Browser
RayrunBuild, deploy, and operate MCP servers without becoming an infrastructure team.ray.runAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

TDQS

A4.4/5.0
Disambiguation5/5

Each tool targets a distinct resource or action: fetching items vs. distributions, listing vs. fetching specific entities, and distinct submission endpoints for benchmark, eval, and RFC comments. Even the two 'get items' tools (bench vs. eval) are clearly differentiated by their descriptions and use cases.

Naming Consistency5/5

All tool names follow a consistent verb_noun pattern with lowercase and underscores. Verbs are grouped by action type (get_, list_, submit_, start_, register_, verify_, search_), and the patterns are predictable and uniform across the set.

Tool Count4/5

At 17 tools, the set is slightly above the ideal 3-15 range but each tool serves a distinct purpose. The breadth of the domain (papers, RFCs, standards packs, benchmarks, evaluation, certification, vocabulary, search) justifies the count, and no tool feels redundant.

Completeness4/5

The tool surface covers the major workflows: listing and fetching resources, starting and submitting evaluation attempts, registering and verifying certifications, and submitting benchmark/RFC responses. Minor gaps exist, such as no direct tool to cancel an evaluation attempt or list a specific benchmark run by id, but these are edge cases and the core lifecycle is well-covered.

Resources