emit_probe_set

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden. It discloses that the tool writes a JSONL file to a configurable output path (defaulting to <path>.<probe>.jsonl), describes the output format, and explains that the server requires no API key. However, it does not explicitly state whether existing output files are overwritten or if there are any side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear first sentence defining purpose, followed by output details, workflow, cost note, and argument list. It is somewhat verbose but each sentence adds value; a slight trim could improve conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema and the complexity of the tool, the description adequately covers output format and workflow. It explains the round-trip and cost benefit. It could briefly mention the input format expected for path, but overall it is sufficiently complete for effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, which it does thoroughly. It explains each parameter: path (existing judge run), probe (enumeration with context on which probes need variants), out_path (default derivation), and hint (examples for sycophancy). This adds significant meaning beyond the bare schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool generates variant files for probe scoring, specifically writing a JSONL of items to judge with variant tags and original item_id. It distinguishes itself from siblings by noting which probes need variants and which don't (distribution, length_confound).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly explains the workflow: run emit_probe_set, then run a judge, then pass results to bias_probe. It specifies when to use (for position, verbosity, sycophancy probes) and when not (for distribution, length_confound), and references sibling bias_probe for the next step.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.