Server Quality Checklist

Tool Scores

• run_pipelineA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark readOnly false and idempotent false, so the mutation/non-idempotent nature is covered. The description adds context that execution uses each stage's persistent baseline config (vs ad-hoc config), but it doesn't disclose side effects, latency, or external calls beyond the 'retrieve then rerank' example. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One compact, front-loaded sentence that states action, object, scope, and config behavior. Every clause earns its place; no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the operation and config behavior, and annotations plus output schema handle safety and return format. However, the sole required parameter external_inputs is left entirely undescribed, which is a meaningful gap for correct invocation. The missing run_stage contrast is minor because siblings imply it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description never mentions external_inputs or explains what shape/contents the object should have. With additionalProperties true, the agent gets no guidance on required keys or semantics from either source.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run'), names the resource ('the entire loaded pipeline graph'), and specifies the scope ('end to end', 'every stage'), distinguishing it from sibling run_stage without needing to inspect schemas. The 'current persistent baseline config' qualifier adds precision about how stages are configured.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives a clear context for use: when you want the full pipeline executed end to end rather than a single stage. It doesn't explicitly name run_stage as the alternative or state exclusions, but the 'entire loaded pipeline graph' phrasing makes the route evident.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_stage_configA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, so the safe-read nature is covered. The description adds the nuance 'current persistent baseline config', which clarifies it is not runtime/transient configuration, but it does not disclose further behavioral details such as error cases or response shape.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence that is front-loaded with the verb and resource, with no filler or redundant restatement. It earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter read tool with readOnly and idempotent annotations plus an output schema, the description is largely complete. The only context gap is explicit guidance on when to prefer sibling tools, but the get-vs-set/list semantics make this reasonably inferable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for stage_id. It ties the parameter to 'a stage's' config, but it does not explain how to obtain or format stage_id. The parameter name and title already do most of the work.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') and a specific resource ('a stage's current persistent baseline config'). This clearly distinguishes it from its siblings 'set_stage_config' (set vs. get) and 'list_stages' (list all vs. get one).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Get a stage's current persistent baseline config' implies use when you need the saved baseline for a single stage, but it does not explicitly say when to use this vs. list_stages or set_stage_config. No alternatives or exclusion conditions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_stageA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations only declare readOnlyHint=false, openWorldHint=false, and idempotentHint=false. The description adds meaningful behavior beyond that: config_override is merged over the persistent baseline for this call only, is not saved, and repeated calls avoid mutating shared config state. This gives an agent useful expectations about side effects and persistence.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The core purpose is front-loaded in the first sentence, and the config_override guidance is organized clearly. The explanation is slightly wordy but every sentence adds relevant operational detail, so it remains efficient without being bloated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description thoroughly covers the optional config_override behavior and its use case, but it omits semantics for the required stage_id and input parameters, and does not mention prerequisites like the graph needing to be loaded. The output schema exists, so return values need not be explained, but the parameter gaps leave the definition incomplete for a 3-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It richly explains config_override, including merge semantics and non-persistence, but stage_id and input receive no semantic explanation beyond their names. With two of three parameters unexplained, the description only partially compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run one stage from the loaded graph.' This clearly identifies the tool's action and scope, and distinguishes it from siblings like run_pipeline (whole pipeline) and list_stages/get_stage_config/set_stage_config (inspection/config tools).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives strong, explicit usage guidance for config_override: when to use it, how to compare configs, and that no reset is needed afterward. However, it does not explicitly state when to choose this tool over run_pipeline or other siblings, so it stops short of full alternative-based guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_stagesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish readOnlyHint=true and idempotentHint=true, so the description does not need to restate safety. Beyond those hints, it discloses that the operation is scoped to the 'currently loaded pipeline graph' and that the output includes full config and JSON schema details, which is useful contextual behavior. This exceeds the minimal annotation coverage without contradicting it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler: the first states the action and result fields, the second states its purpose. The key scoping phrase 'currently loaded pipeline graph' is front-loaded, and every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple zero-parameter list tool with an output schema available, the description provides enough context: scope, result contents, and why the agent would call it. The mention of run_stage and set_stage_config helps justify its role in a workflow. No critical information appears missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema is empty, so there are no parameter semantics to clarify; the baseline for zero-parameter tools is 4. The description does not mention arguments, but none are needed. It correctly focuses on output and context rather than parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'List every stage in the currently loaded pipeline graph', a specific verb and resource, and enumerates the returned fields (id, stage_type, current config, full JSON schemas). This clearly distinguishes list_stages from its siblings get_stage_config, set_stage_config, and run_stage, which operate on individual stages or perform mutations. It leaves no doubt about the tool's scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence gives a concrete use case: the output is 'Enough to call run_stage or set_stage_config without opening any adapter source file.' This tells the agent when to use this tool as a preparatory step. However, it does not explicitly name get_stage_config as an alternative for single-stage queries, so it lacks explicit when-not/alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_stage_configA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description reveals that the merge is validated before committing, is atomic (invalid overrides are rejected rather than partially applied), and has a persistent global effect on subsequent run_stage and run_pipeline calls. These are exactly the side effects an agent needs to anticipate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three focused sentences lead with the core operation, then cover side effects and the alternative. No filler or schema repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter mutation with an output schema present and idempotentHint set, the description covers the main decision, the effect, and the exception. Nothing critical is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description carries the burden. It defines stage_id as the target baseline config and config_overrides as the values being merged, and adds validation semantics, but it does not enumerate any allowed override keys. Given additionalProperties is true, that is acceptable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a precise action ('persistently merge') and a specific resource ('stage_id's baseline config'), so an agent knows exactly what the tool does. It also distinguishes this from run_stage by contrasting persistent changes with one-off overrides.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states when to use the tool (when a change should persist and affect future runs) and when not to (one-off comparison), explicitly directing the agent to pass config_override to run_stage instead. This is clear routing against a named sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.