Server Quality Checklist

Tool Scores

• run_moduleB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden for behavioral disclosure. It only advises check mode first, but fails to mention that the tool can execute arbitrary Ansible modules with potentially destructive side effects, requires permissions, or what happens on failure. This is a significant transparency gap for a tool running arbitrary code.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the core purpose and a safety tip. Every word contributes to the message, and it is easily scannable. It is concise without unnecessary verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite its conciseness, the description is severely incomplete for a tool with 9 parameters, no annotations, and no schema descriptions. It fails to explain how to use arguments, inventory, become, connection, or when this tool is appropriate versus run_playbook. The presence of an output schema does not compensate for the missing operational guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description is expected to compensate for missing parameter explanations. It does not explain any of the 9 parameters beyond implicitly referring to module and check_mode. Args, hosts, become, timeout, inventory, connection, and become_user are left entirely to the schema, which lacks descriptions. The description adds no meaningful semantic value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Run' and the resource 'Ansible module via local ansible CLI', distinguishing it from sibling tools like run_playbook. It is specific and unambiguous, making the tool's primary function immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes a usage tip to prefer check_mode=true, which is helpful, but it lacks explicit guidance on when to use this tool versus alternatives like run_playbook. The context of running a single module is implied but no exclusions or alternative suggestions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_playbookA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries responsibility for disclosing side effects. It mentions 'allowlisted roots' and suggests using check mode, implying the tool can make changes, but it does not explicitly state that it may modify systems or require elevated privileges.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, each contributing distinct information: what it does and how to use it safely. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 11 parameters and no annotations, the description is too sparse. It lacks information about parameter roles, error behavior, and the consequences of running playbooks with check_mode=false, making it incomplete for safe use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description provides no information about any of the 11 parameters. The parameter names are in the schema, but the description adds no semantic value to compensate for the missing descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs ansible-playbook with a specific path constraint (allowlisted roots). This is distinct from sibling tools which focus on modules (search_modules, run_module, etc.).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises to prefer check_mode=true first, which provides a clear usage guideline. It does not explicitly name alternative tools, but the tool's purpose is obvious against module-related siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_collectionsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It implies a read-only operation through the word 'List,' and 'committed gallery' adds scope context, but it does not disclose any side effects, permissions, or edge cases. The minimal disclosure is adequate for a simple listing tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that immediately states the action and scope. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless listing tool with an output schema, the description is sufficiently complete. It tells the agent what the tool does and what data it operates on, and the output schema will provide the return structure. No additional context is necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline of 4 applies. The description does not need to explain parameter semantics, and the input schema confirms no arguments are required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('collections') with a location qualifier ('committed gallery'), distinguishing it from sibling tools that operate on modules. It is slightly more informative than the tool name alone, though 'committed gallery' is domain-specific and not elaborated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives. The description only states what the tool does, with no context about when it is appropriate or how it compares to the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_module_schemaA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden of behavioral disclosure. It mentions 'slim' and 'when catalog has it,' but fails to specify what happens when the module is missing from the catalog (e.g., error, empty result) or what 'slim' entails. This leaves the agent uncertain about failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no filler. Every term adds meaning, including 'slim' and 'when catalog has it,' making it efficiently front-loaded and appropriately sized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter and has an output schema, so return values are presumably covered. However, the description lacks guidance on missing-module behavior and does not situate this tool relative to siblings like search_modules, leaving practical edge cases unaddressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only a 'module' string with no description (0% coverage). The description adds critical semantic meaning by specifying that the parameter expects a module FQCN (fully qualified class name), clarifying the required input format beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the clear verb 'Return' and specifies the resource: 'slim arg schema for a module FQCN'. This distinguishes it from sibling tools like search_modules and run_module, making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'when catalog has it' implies a conditional context but does not explicitly contrast with alternatives like search_modules. There is no explicit when-to-use or when-not-to-use guidance, though the intended usage is somewhat inferable from sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_modulesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses the 'allowlisted' restriction and searchable fields, which is useful behavioral context. However, it does not describe behavior like empty queries, default limits, or search matching semantics, leaving the agent with incomplete expectations for a read-only search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, sentence-length description that immediately conveys the tool's purpose and scope. No filler, front-loaded with the verb, and every word adds meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with an output schema and available sibling context, the description covers the essential purpose and searchable attributes. Minor gaps like edge-case search behavior and the meaning of 'allowlisted' are not critical for invocation, but there is slight room for more context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must clarify parameters. It does so for 'query' by specifying the search fields (name, FQCN, collection, description), but 'limit' is left to the schema's default without additional context. This provides partial compensation for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Search' and identifies the resource as 'allowlisted Ansible modules (gallery)' with explicit search criteria (name, FQCN, collection, description). This distinguishes it from sibling tools like list_collections and get_module_schema, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use for finding modules by various attributes, which is distinct from listing collections or retrieving schemas. However, it does not explicitly state when to prefer this over alternatives or mention exclusions, so guidance is somewhat implicit but still clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.