Server Quality Checklist

Tool Scores

• graphql_schema_root_typeC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but offers minimal behavioral context. It states the tool returns documentation content as a string but doesn't disclose authentication requirements, rate limits, error handling, or what specific documentation format to expect. The 'query' verb implies read-only, but this isn't explicitly confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with three sentences: purpose statement, parameter guidance, and return value. It's front-loaded with the core function. Minor improvements could include bullet points for Args/Returns, but overall it's efficient with minimal waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on authentication, error cases, the structure of returned documentation, or how this integrates with sibling tools. The return type 'str: Documentation content' is vague without examples or format specifications.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by explaining that 'type_name' should be 'QUERY or MUTATION' and represents the 'root type name.' This adds meaningful context beyond the schema's generic 'Type Name' title, though it doesn't detail format constraints or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'query GitHub GraphQL schema root types' with specific resource (GitHub GraphQL schema) and verb (query). It distinguishes from sibling 'graphql_schema_type' by focusing on root types only, though it doesn't explicitly contrast with 'call_github_graphql' or 'print_type_field'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'graphql_schema_type' for non-root types or 'call_github_graphql' for actual queries. The description mentions needing to provide root type name but doesn't explain use cases or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• print_type_fieldB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses that the tool returns documentation content as a string, which is useful behavioral information. However, it doesn't mention whether this is a read-only operation, what happens with invalid inputs, if there are rate limits, authentication requirements, or error behavior. For a query tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is reasonably concise but has structural issues. The first sentence clearly states the purpose, but the parameter explanations are formatted as 'Args:' and 'Returns:' sections which are somewhat redundant with the schema. The information is front-loaded but could be more efficiently integrated. It's not excessively verbose but has minor organizational inefficiencies.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters with 0% schema coverage and no output schema, the description provides adequate parameter semantics and specifies the return type as a string. However, it doesn't explain the format of the returned documentation content, error conditions, or authentication requirements. For a query tool with no annotations, this is minimally complete but lacks depth about operational behavior and integration context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides meaningful semantics for both parameters: 'type_name' is explained as 'root type like `QUERY` or `MUTATION`' and 'type_fields_name' as 'field name like `repository` based on root type documentation'. This adds substantial value beyond the bare schema, though it doesn't provide exhaustive examples or format details. With 2 parameters fully addressed, this exceeds the baseline expectation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'query GitHub GraphQL schema root type fields' with specific verbs ('query') and resources ('GitHub GraphQL schema root type fields'). It distinguishes from siblings like 'call_github_graphql' (which executes queries) and 'graphql_schema_type' (which might query non-root types), but doesn't explicitly contrast them. The purpose is specific but sibling differentiation is implied rather than explicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions 'based on root type documentation' but doesn't specify when this tool is appropriate compared to sibling tools like 'graphql_schema_root_type' or 'graphql_schema_type'. There are no explicit when/when-not instructions or named alternatives, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• graphql_schema_typeB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool queries documentation, implying a read-only operation, but doesn't specify if it's safe, requires authentication, has rate limits, or what happens on errors. For a query tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the purpose clearly. The 'Args:' and 'Returns:' sections are structured but slightly redundant with the schema. It's concise with no wasted words, though the formatting could be more integrated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is somewhat complete but has gaps. It covers the basic purpose and parameter semantics but lacks usage guidelines, behavioral details, and output explanation. For a simple query tool, it's adequate but not fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It explains the single parameter 'type_name' as 'Type name like `SecurityAdvisoryConnection`,' adding meaning beyond the schema's basic 'string' type. This clarifies the parameter's purpose and provides an example, though it could be more detailed about valid type names or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'query specific type documentation in GitHub GraphQL schema.' It specifies the verb ('query'), resource ('type documentation'), and domain ('GitHub GraphQL schema'). However, it doesn't explicitly differentiate from sibling tools like 'graphql_schema_root_type' or 'print_type_field' beyond mentioning 'specific type' documentation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance: 'You need to provide the type_name' and gives an example ('SecurityAdvisoryConnection'). It doesn't explain when to use this tool versus alternatives like 'graphql_schema_root_type' or 'print_type_field,' nor does it mention prerequisites or exclusions. The guidance is basic and lacks context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• call_github_graphqlB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions checking documentation and including ID fields, which adds some operational context, but fails to address critical traits like authentication requirements, rate limits, error handling, or mutation vs. query behavior. For a GraphQL API tool with zero annotation coverage, this leaves significant gaps in understanding how the tool behaves in practice.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose and usage recommendation in the first sentence. The Args and Returns sections are structured but could be more integrated. There's minimal waste, though the 'Returns: str: Execution result' is redundant given the output schema, slightly reducing efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (GraphQL API execution) and the presence of an output schema (which handles return values), the description is partially complete. It covers basic purpose and some usage tips but lacks details on authentication, error cases, or integration with sibling tools. Without annotations and with low schema coverage, it doesn't fully equip an agent for reliable tool invocation in a GitHub context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal meaning beyond the input schema: it names the single parameter 'graphql' and states it's 'The GraphQL query', which the schema already indicates as a string type. With 0% schema description coverage, the description doesn't compensate by explaining query syntax, validation rules, or examples. The baseline is 3 because the schema covers the parameter's existence and type, but the description fails to enhance understanding significantly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'execute GitHub GraphQL API queries' with a specific verb (execute) and resource (GitHub GraphQL API queries). It distinguishes itself from sibling tools like graphql_schema_root_type and graphql_schema_type by focusing on query execution rather than schema exploration. However, it doesn't explicitly contrast with print_type_field, leaving some sibling differentiation incomplete.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage guidance by recommending checking documentation first and including ID fields for follow-up operations, which suggests context for when to use this tool effectively. However, it lacks explicit when/when-not criteria or named alternatives to sibling tools, leaving the agent to infer optimal usage scenarios without clear boundaries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.