ab_test

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full disclosure burden. It does valuable work explaining the return contract (per-scenario CEP deltas, verdict values, and how to interpret 'mixed' as a trade-off), but it never says whether the patch is persisted or purely experimental, how many runs are executed, or whether the operation is expensive or read-only in effect.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action, followed by the A/B definition, the return contract, and one interpretive caveat. Every sentence adds information an agent would otherwise have to guess, and nothing is repeated from the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, and the description compensates by describing the result shape and verdict vocabulary. But for a 4-parameter tool with a nested free-form object and zero schema coverage, the failure to explain presets/scenarios scoping or whether the patch is persisted leaves real gaps for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate. It explains config_patch well with a JSON merge-patch example, but 'presets', 'scenarios' and 'seeds' are never described in either place — the agent cannot tell that presets/scenarios select the comparison set or that seeds is a repetition count (default 5).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('A/B test a configuration change across scenarios on identical seeds') and defines exactly what A and B mean in terms of the config_patch. It doesn't explicitly distinguish itself from the similar-sounding sibling compare_filters, so the agent must infer the boundary, but the core action is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implicit guidance: it's clearly for evaluating a proposed config change against current defaults on comparable seeds. However, there is no explicit when-to-use/when-not, no mention of the alternative paths (editing config via engine_config, or single-scenario validation via simulate_scenario), and no note about cost/runtime relative to a single simulation.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.