Server Quality Checklist

Tool Scores

• get_cheaha_quick_startA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds minimal behavioral context beyond this, such as specifying it's for 'new users' and providing 'essential information,' but doesn't disclose additional traits like rate limits, authentication needs, or detailed output behavior. There's no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, with three sentences that efficiently convey purpose, target audience, parameter info, and return value. Each sentence adds clear value without redundancy, and it's front-loaded with the main purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, annotations covering safety, and an output schema present), the description is reasonably complete. It explains what the tool does, who it's for, and what it returns, though it could better differentiate from sibling tools. The output schema handles return values, so the description doesn't need to detail them.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the schema fully documents the input (none required). The description adds value by explicitly stating 'No parameters required,' which reinforces the schema and clarifies usage. This compensates adequately, though no parameter details are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get quick start information for accessing and using the Cheaha HPC cluster.' It specifies the verb ('Get') and resource ('quick start information'), and identifies the target system (Cheaha HPC cluster at UAB). However, it doesn't explicitly differentiate from sibling tools like 'get_documentation_page' or 'get_support_info', which might also provide informational content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by stating it's 'for new users getting started,' but it doesn't provide explicit guidance on when to use this tool versus alternatives like 'get_documentation_page' or 'search_documentation.' It mentions 'No parameters required,' which helps with invocation but doesn't clarify selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_documentation_sectionsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe, repeatable read operation. The description adds context about returning 'a structured list of main documentation sections and their purposes,' which provides useful output information, but it doesn't disclose additional behavioral traits like rate limits or authentication needs. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the purpose, the second explains the utility, and the third clarifies parameters and returns. Every sentence adds value without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, read-only, idempotent), the description is complete. It explains the purpose, usage context, parameter requirements, and return value. With annotations covering safety and an output schema likely detailing the structured list, no additional information is needed for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the schema fully documents the lack of inputs. The description explicitly states 'No parameters required,' which adds clarity and confirms the schema, earning a score above the baseline of 3 for zero-parameter tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List the main sections and categories available in the UAB Research Computing documentation.' It specifies the verb ('List') and resource ('main sections and categories'), though it doesn't explicitly differentiate from sibling tools like 'get_documentation_page' or 'search_documentation' beyond the scope of listing sections versus retrieving content or searching.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by stating it 'provides an overview of the documentation structure to help users understand what information is available,' which suggests it's for initial exploration. However, it doesn't explicitly say when to use this tool versus alternatives like 'search_documentation' or 'get_documentation_page,' leaving the guidance somewhat implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_documentationA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds valuable context beyond annotations by specifying the search scope ('through the documentation repository'), result formatting ('titles, URLs, and excerpts'), and default/max values for max_results, which are behavioral details not captured in annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized: it starts with a clear purpose statement, follows with usage context in a bulleted list, and ends with parameter and return details. Every sentence adds value without redundancy, and information is front-loaded for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, read-only operation), the description is complete: it covers purpose, usage examples, parameter semantics, and return format. With annotations providing safety hints and an output schema presumably detailing the formatted results, no additional information is needed for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter descriptions. The description compensates by explaining query ('search term or phrase to look for') and max_results ('Maximum number of results to return') with default and max values. However, it doesn't detail query syntax (e.g., wildcards, phrases) or result ordering, leaving some semantic gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('search'), resource ('UAB Research Computing documentation'), and scope ('for relevant content'). It distinguishes from siblings like get_documentation_page (retrieve specific page) and list_documentation_sections (list sections) by emphasizing search functionality across the entire repository.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('Useful for finding information about...') with specific examples like Cheaha HPC cluster usage and research computing policies. However, it doesn't explicitly state when not to use it or name alternatives among sibling tools (e.g., when to use get_documentation_page instead).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_support_infoA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds valuable behavioral context beyond what annotations provide. While annotations already indicate readOnlyHint=true, idempotentHint=true, and destructiveHint=false, the description specifies that 'No parameters required' and describes the return content (office hours, contact methods, links to support resources). This provides practical usage information that annotations don't cover.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise. It starts with the core purpose, adds details about what information it provides, explicitly states the parameter situation, and describes returns. Every sentence earns its place with no wasted words, and information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters), comprehensive annotations, and existence of an output schema, the description is complete. It covers purpose, what information it returns, and the parameter situation. With an output schema present, the description doesn't need to detail return values beyond the high-level summary it provides.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the baseline would be 4. The description explicitly states 'No parameters required', which adds clarity beyond the empty schema. This confirms the tool's simplicity and eliminates any ambiguity about whether parameters might be needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb 'Get' and resource 'information about how to get support from UAB Research Computing', including what information it provides (contact information, office hours, support channels). It distinguishes from sibling tools like get_documentation_page or search_documentation by focusing specifically on support information rather than general documentation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (when needing support information for UAB Research Computing services). However, it doesn't explicitly state when not to use it or name specific alternatives among the sibling tools, though the distinction is implied through the different resource types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_documentation_pageA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds valuable context beyond annotations: it specifies the source ('UAB Research Computing GitHub repository'), content format ('markdown'), and that it fetches 'complete' content, which helps the agent understand behavioral traits like data source and output type.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with a clear purpose statement, usage guideline, parameter details, and return value—all in four concise sentences. Each sentence adds value: the first states the action, the second adds source context, the third provides usage guidance, and the fourth and fifth detail parameters and returns efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no nested objects), rich annotations (readOnly, idempotent, non-destructive), and presence of an output schema (so returns needn't be explained in description), the description is complete. It covers purpose, usage, parameter semantics, and source context adequately for the agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides detailed semantics for the single parameter page_path: explains what it represents ('path to the documentation page'), gives examples (e.g., 'docs/cheaha/slurm/slurm_tutorial.md'), and clarifies format options ('relative path from the repository root or a GitHub URL'), adding significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Retrieve the full content'), target resource ('specific documentation page'), and format ('complete markdown content'). It distinguishes from sibling tools by specifying it fetches content after finding pages with the search tool, unlike list_documentation_sections which lists sections or search_documentation which searches.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool ('Use this after finding a relevant page with the search tool'), providing clear context and distinguishing it from search_documentation. It implies alternatives by referencing the search tool for initial discovery.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.