Skip to main content
Glama

Yardstick ATS

Preview built-in-agent settings changes (nothing changed yet)

yardstick_built_in_agents_configure_preview
Idempotent

Preview configuring a built-in agent (config is validated against the built-in agent's declared schema; enabled is a first-class key, so { enabled: false } is the off switch). Side-effect free; pass config via --body-json.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
configYes
request_idNo
idempotency_keyNo
built_in_agent_idYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.5/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description explicitly claims 'Side-effect free', but the annotations declare readOnlyHint=false, which signals the tool may modify state. This is a direct contradiction between the description and the annotations, making it unsafe for an agent to rely on the stated side-effect behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single dense sentence with no filler. Purpose, validation behavior, the key 'enabled' semantics, side-effect safety, and invocation method are all front-loaded and useful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The core usage is covered well, especially the config semantics and side-effect-free guarantee. But there is no output schema and the description never explains what the preview returns or how the agent should interpret the response, leaving a meaningful gap for a tool whose whole purpose is to produce a preview.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description adds real meaning to the config parameter: it is validated against the agent's declared schema, enabled is a first-class key, and { enabled: false } is the off switch. However, request_id and idempotency_key remain completely undocumented, and built_in_agent_id is only implied by the tool name.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly says it previews configuring a built-in agent, names the resource, and emphasizes that nothing is changed yet. This makes the dry-run nature obvious and distinguishes it from the applying sibling (configure_bind).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear context: use this to preview and validate config before applying, with no side effects, and pass config via --body-json. It does not explicitly name the apply alternative or state when not to use it, so it stops short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.