Server Quality Checklist

Tool Scores

• call_foreman_apiC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions a prerequisite ('Needs Foreman API resource to be available') but doesn't describe what the tool does beyond 'call an action', such as whether it performs read/write operations, authentication needs, rate limits, or error handling. This leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two short sentences, front-loaded with the main purpose. There's no wasted text, but it could benefit from more detail given the complexity implied by the parameters and lack of annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (3 parameters with nested objects, no output schema, and no annotations), the description is incomplete. It doesn't explain what the tool returns, how to structure parameters, or provide enough context for safe and effective use. The prerequisite hint is insufficient for a tool that likely performs API operations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter details. The description doesn't add any meaning to the parameters (resource, action, params) beyond what the schema titles imply. It doesn't explain what 'resource' or 'action' refer to, or what 'params' should contain, failing to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Call an action on a Foreman API resource', which provides a clear verb ('Call') and resource ('Foreman API resource'), but it's vague about what 'call an action' specifically entails. It doesn't distinguish from sibling tools like 'fetch_foreman_dsl_docs' or 'get_foreman_api_resource_docs', which appear to be documentation-related, but the distinction isn't explicitly stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes 'Needs Foreman API resource to be available', which implies a prerequisite but doesn't provide explicit guidance on when to use this tool versus alternatives. There's no mention of when-not-to-use or how it differs from sibling tools, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_foreman_dsl_docsC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states the action ('fetches') without detailing aspects like authentication needs, rate limits, error handling, or what the fetched documentation includes (e.g., format, scope). This is inadequate for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's front-loaded and appropriately sized for the tool's apparent simplicity, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations, no output schema, and low schema description coverage (0%), the description is incomplete. It lacks details on behavior, parameter usage, and output, which are essential for an agent to use the tool effectively in context with sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 1 required parameter ('section') with 0% description coverage, meaning the schema provides no details about this parameter. The description adds no information about what 'section' means, valid values, or how it affects the fetch operation, failing to compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the action ('fetches') and resource ('DSL documentation from Foreman'), which clarifies the basic purpose. However, it's vague about what 'DSL documentation' entails and doesn't distinguish this tool from sibling tools like 'get_foreman_api_resource_docs' or 'Get Foreman DSL Documentation', which appear to serve similar documentation-fetching purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'call_foreman_api', 'get_foreman_api_resource_docs', and 'Get Foreman DSL Documentation', there's no indication of context, prerequisites, or distinctions, leaving the agent without usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_foreman_api_resource_docsC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It states the tool 'fetches' documentation, implying a read-only operation, but doesn't specify whether it requires authentication, has rate limits, returns structured data, or handles errors. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste—it directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple single-parameter fetch), lack of annotations, 0% schema coverage, and no output schema, the description is incomplete. It doesn't address behavioral aspects like authentication needs, return format, or error handling, which are critical for an agent to use it correctly without structured data to rely on.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the parameter 'resource' is undocumented in the schema. The description adds minimal semantics by indicating it's for 'given Foreman API resource', but doesn't explain what constitutes a valid resource (e.g., format, examples, or constraints), failing to compensate for the low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('fetches') and target ('documentation for given Foreman API resource'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'fetch_foreman_dsl_docs' or 'Get Foreman DSL Documentation', which appear to serve similar documentation-fetching purposes but for different resource types.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description doesn't mention sibling tools like 'call_foreman_api' (which likely performs API calls rather than fetching docs) or clarify distinctions between API resource docs and DSL docs, leaving the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• Get Foreman DSL DocumentationB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'Reads from cache,' which hints at performance or data source behavior, but doesn't cover critical aspects like whether this is a read-only operation, potential errors, rate limits, or authentication needs. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with two sentences that efficiently convey the core functionality and a reference for further details. Every sentence adds value without redundancy, making it concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a documentation retrieval tool with one parameter), no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on return values, error handling, and behavioral traits, which are essential for effective tool invocation. The reference to external sections helps but doesn't suffice for full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate for the undocumented parameter 'section.' It adds some meaning by linking to 'foreman://documentation/dsl/sections for available sections,' which provides context for valid values. However, this is minimal and doesn't fully explain the parameter's purpose or usage, leaving it inadequately documented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Reads from cache and returns the documentation of available macros for template writing in Markdown format based on provided section.' It specifies the verb ('reads', 'returns'), resource ('documentation of available macros'), and format ('Markdown format'). However, it doesn't explicitly differentiate from sibling tools like 'fetch_foreman_dsl_docs' or 'get_foreman_api_resource_docs', which appear related to documentation retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context by mentioning 'Refer to foreman://documentation/dsl/sections for available sections,' which implies where to find valid inputs. However, it doesn't explicitly state when to use this tool versus alternatives like 'fetch_foreman_dsl_docs' or provide exclusions. The guidance is implied but not comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.