Server Quality Checklist

Tool Scores

• search-changelogA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only states that the tool 'searches' the changelog but does not explain matching behavior, result format, or limitations. This leaves important operational details unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action ('Search the SS6 changelog') followed by usage context. Every word earns its place with no unnecessary filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter search tool, the description covers what and when, and the schema handles parameter documentation. However, since there is no output schema, the description does not mention what results contain or how maxResults affects the output, leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides descriptions for both 'query' and 'maxResults,' covering 100% of parameters, so the baseline is 3. The description reinforces the purpose of the query but does not add new syntax or format details beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Search the SS6 changelog for specific terms,' clearly stating the verb, resource, and scope. It also distinguishes itself from siblings by referencing 'the themed documentation sections,' showing this tool is for targeted changelog lookups.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this when you need to find information about a specific class, method, or feature that may not be in the themed documentation sections,' giving clear usage context. However, it does not name alternative tools or state when not to use it, so it falls slightly short of full guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get-documentationA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It mentions the dependency on list-sections, which is useful. However, it does not disclose details about return format, error behavior, or what happens if the section ID is invalid, leaving some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, and includes a critical usage hint. Every word is purposeful with no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema, the description provides sufficient context: what it does and the required precursor. It does not explain the return value, but given the simplicity of fetching documentation, this is a minor gap and the description remains largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides full coverage of the 'section' parameter with a clear description and example. The tool description adds no additional parameter semantics beyond what the schema documents, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches Silverstripe 6 changelog documentation for a specific section, with a specific verb 'fetches' and resource. It distinguishes itself from sibling tools like search-changelog and list-sections by focusing on retrieving a documented section rather than listing or searching.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance to call list-sections first, establishing a required prerequisite. It doesn't explicitly mention when not to use this tool or contrast with alternatives, but the context is clear enough for an agent to understand when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-sectionsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry behavioral disclosure. It adds workflow context (use before fetching) but does not describe the return format, authentication needs, or any side effects. Since 'list' implies a read-only operation, this is minimally adequate but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The two sentences are efficient: the first states the action, the second provides usage guidance. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core purpose and usage workflow for a zero-parameter list operation. It does not explicitly state the output format, but the verb 'lists' implies the return of available sections, which is sufficient for such a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivially covered at 100%. The description correctly avoids adding parameter details because none exist, achieving the baseline for no-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Lists' with a concrete resource ('Silverstripe 6 changelog sections'), and then distinguishes the tool from sibling get-documentation by framing it as a discovery step. This clearly differentiates it from search-changelog and ss-validator.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence explicitly instructs when to use the tool: 'Use this to discover what documentation is available before fetching with get-documentation.' It names the sibling tool as the next step, providing clear usage context, though it doesn't formally state exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ss-validatorA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses key behavioral aspects: the types of issues detected and the iterative requirement. However, it does not describe the return format or state that the tool is read-only, which would be valuable for a validator. The absence of return details is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a clear purpose sentence, a strong imperative for mandatory usage, and a concise bulleted list of detection categories. Every sentence earns its place, and the front-loaded purpose ensures quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, when to use it, and what it detects, which is strong given the 4-parameter schema and lack of output schema. However, it does not explicitly describe the return value (e.g., a list of issues, a status indicator), which leaves a gap for agents needing to process results. This is a notable omission for a validator tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds minimal extra meaning beyond the schema, except linking 'generated Silverstripe PHP code' to the code parameter and mentioning BuildTask patterns relevant to filename context. It does not provide additional syntax or format details beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: analyzing PHP code for Silverstripe 6 compatibility issues. It uses a specific verb ('analyzes') and resource ('PHP code'), and the bulleted list of detection categories distinguishes it from sibling tools like get-documentation or search-changelog.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use the tool ('MUST be called on any generated Silverstripe PHP code before presenting to the user') and provides an iterative usage pattern ('Keep calling until no issues remain'). This gives clear context and exclusions, leaving no ambiguity about when it should be invoked.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.