Server Quality Checklist

Tool Scores

• search_casesB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It explains that it returns a list of cases with numbers and metadata, and includes pagination parameters (rows, start), which is useful. However, it does not explicitly state that it is read-only, nor does it mention authentication requirements, error behavior, or whether results are sorted. This is minimal but sufficient for a basic search tool, earning a 3.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a one-sentence summary followed by an Args block and a Returns line. It is front-loaded with the primary purpose, and every item adds value. The only minor redundancy is the Returns line restating what the initial sentence implies, but it is not wasteful. Overall, it is concise and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple (3 params, no output schema, no annotations). The description covers the essential purpose, parameters, and return type. However, it omits edge cases like empty results, maximum pagination bounds, or any error conditions. For an agent to use it effectively, a bit more detail on return structure and limitations would be helpful. This is adequate but not complete, hence a 3.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does provide a one-line explanation for each parameter: 'query' as a search query string, 'rows' as number of results, and 'start' as pagination index. These are clear but somewhat generic (e.g., 'query' just restates its type). The descriptions help an agent use the params correctly, but lack depth or examples, so a score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Search for Red Hat cases and return a list of case numbers,' which clearly states the verb (search) and resource (Red Hat cases). The mention of returning case numbers distinguishes it from sibling tools like get_case or get_case_raw, which likely fetch a single case. This meets the 5-level bar of specific verb+resource and sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool vs alternatives such as get_case or search_kcs. The description simply explains what it does, without stating situations where it should be preferred or avoided. There is no mention of 'use this for searching a list of cases, use get_case for a specific case.' This lack of direction warrants a score of 2.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_kcsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral transparency burden. It usefully discloses a default filter (documentKind and accessState) and the return type, but it does not mention authentication requirements, pagination behavior beyond the start parameter, error handling, or rate limits. This is moderate disclosure but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose sentence followed by compact Args and Returns sections, ending with a crucial default-filter caveat. It is concise, though the Args list partially duplicates schema field names and defaults, and the Returns section could arguably be inferred from the purpose statement.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a moderately complex tool with three simple parameters and no output schema, the description provides sufficient context: purpose, parameter explanations, return shape, and a key default-filter behavior. It does not explain total result counts, ordering, or relationship to get_kcs, but covers the essentials needed for typical invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It defines all three parameters: query as the search string, rows as the result count with default 50, and start as the pagination index with default 0. This adds meaning beyond the schema, which only lists titles and defaults. It lacks examples or constraints like maximum rows, so it is not a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with a specific verb and resource: 'Search for Red Hat KCS Solutions and return a list with Solution IDs.' This clearly states the tool's function and its output, distinguishing it from sibling tools like get_kcs (which likely retrieves a single KCS solution) and search_cases (for cases).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Search for Red Hat KCS Solutions' gives a clear context for when to use this tool, but it does not explicitly mention alternatives or exclusions. It only implies usage for query-based solution discovery, without saying 'use get_kcs for a single solution' or 'use search_cases for case search.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_caseA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It explains that the tool returns formatted case data with specific fields, which is useful. However, it does not mention potential errors, permissions, or other behavioral nuances, though for a simple read tool this may be sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and well-organized with clear Args and Returns sections. Every sentence provides necessary information without unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter, no output schema, and no annotations, the description covers all essentials: what the tool does, what input to provide, and what output to expect. It is complete for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description compensates by explaining the case_number parameter with a concrete example ('01234567'). This adds meaning beyond the plain schema and fully documents the only parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get case details by case number.' It uses a specific verb and resource, and the return specification of 'Formatted case data' distinguishes it from likely raw/alternative tools like get_case_raw. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a case number, but it does not explicitly mention when to prefer this over sibling tools like search_cases or get_case_raw. No exclusions or alternative recommendations are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_case_rawA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It does disclose the return behavior ('complete unfiltered API response with all available fields'), which is helpful. However, it doesn't mention side effects (though 'get' implies read-only), potential large payloads, or any security/privacy caveats about raw data. The transparency is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-organized: a one-line purpose statement followed by Args and Returns sections. Every sentence adds essential information, with no fluff or repetition. It is front-loaded and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 1-parameter tool with no output schema, the description is nearly complete. It covers purpose, parameter format, and return value. It does not describe error handling or rate limits, but these are not critical for a raw data fetch tool. The sibling context helps clarify its place, and the description itself provides sufficient detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines case_number as a string with no description. The tool description compensates by adding a concrete example ('e.g., "01234567"') and clarifying that it is a case number. This provides meaningful semantic value beyond the schema's bare type definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource+scope: 'Get ALL raw fields from a case for debugging/discovery.' It clearly distinguishes itself from sibling get_case by emphasizing 'ALL raw fields' and the debugging/discovery intent, making the tool's unique role obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states a clear usage context: 'for debugging/discovery.' This tells the agent when this tool is appropriate, but does not explicitly exclude alternatives like get_case or mention when to use a different tool. Since no explicit when-not guidance is provided, it falls short of a 5 but provides a clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_kcsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states that the tool returns a Dictionary with named fields (title, Environment, Issue, Resolution, Root Cause), which is meaningful. It stops short of describing error handling or access requirements, but for a simple read operation this is solid.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and well-structured: one opening sentence, a one-line parameter definition, and a one-line return description. It is front-loaded with the core action and contains no filler or redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema), the description covers the essential aspects: what it does, what parameter it takes, and what it returns. It doesn't address when to prefer this over search_kcs, but that is partially implied by sibling tool names and does not prevent correct selection or invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only provides the parameter name 'solution_id' with zero description coverage. The description's Args section adds 'The ID of the solution to retrieve', which gives the parameter clear meaning and compensates for the low schema coverage. For a single simple parameter, this is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Get') with a scoped resource ('a specific solution by ID') and clarifies the intent ('extract structured content'). This clearly distinguishes it from sibling tools like search_kcs (search vs. retrieve) and case-oriented tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a solution ID and need its structured content, but it does not explicitly state when to use this tool over search_kcs or mention any alternatives/exclusions. Usage context is present but not fully developed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.