Server Quality Checklist

Tool Scores

• list_policiesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure. It clearly describes the output (list of policies with resource definitions and access rules) but does not mention any access requirements, side effects, or limitations beyond the return value. It is a read-only operation by implication but not explicitly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with the main action stated in the first line. The 'Returns:' section adds useful detail without redundancy or unnecessary length. Every sentence contributes value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and an output schema exists, the description provides sufficient context for a simple list operation. It describes the return format and the intended audience, though it could have explicitly stated its relationship to sibling tools for more complete context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the description correctly adds no parameter-specific information. Per the rubric, a baseline score of 4 is appropriate for 0 parameters, as there is nothing to describe beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all Ranger policies for the RAG service' with a specific verb and resource. It further explains the purpose as showing access mappings, which distinguishes it from sibling tools like list_knowledge_bases and check_access.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says 'Useful for administrators to understand the current access control setup,' which provides clear context for when to use the tool. However, it does not explicitly exclude alternatives or state when not to use it, so it lacks explicit comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_accessA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry full behavioral disclosure. It discloses that the tool does not query the knowledge base, which is a key behavioral trait. However, it does not state whether it modifies anything, requires special permissions, or how policy decisions are derived. It only says 'Returns: Access decision with policy details,' which is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a compact docstring with a purpose sentence, a usage note, and structured Args/Returns sections. Each part adds necessary information and nothing is redundant. It is front-loaded with the core purpose before diving into details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple access check tool, the description covers what, when, parameters, and return value. It does not mention edge cases like unknown users, unconfigured policies, or differences between the check and actual query behavior. However, the output schema exists to formalize return structure, so the description is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero description coverage, but the description includes an Args section explaining each parameter in plain language. It specifies that access_type accepts 'read' or 'write' and defaults to 'read', which is not discernible from the schema alone. This fully compensates for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Check if a user has access to a specific knowledge base without querying it,' using a specific verb and resource. The phrase 'without querying it' clearly distinguishes it from sibling query_knowledge_base. This is a strong, specific purpose statement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states 'Useful for pre-flight permission checks before attempting a query,' which provides clear usage context. It doesn't name alternatives or exclusions, but the scenario is clearly defined. Sibling tools are listed but not explicitly compared.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_knowledge_basesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden. It explicitly discloses that it checks Ranger authorization and returns only permitted KBs, which is a key behavioral trait. It does not detail failure modes or pagination, but for a read-only list operation this is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, with a clear purpose statement, a brief authorization note, and Args/Returns sections. Every sentence adds value, and there is no unnecessary repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has one parameter, a clear purpose, and an output schema (per context signals), the description fully enables an agent to select and invoke it correctly. It does not need to elaborate further on return values or error handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero description coverage for the 'user' parameter. The description compensates by specifying 'user: Username to check permissions for,' clarifying its purpose and type. This is sufficient for a single simple parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all knowledge bases the user has access to,' which is a specific verb (list) plus resource (knowledge bases) with a scope (user access). This distinguishes it from siblings like query_knowledge_base and list_policies by focusing on the set of accessible KBs rather than querying content or listing policies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for discovering KBs a user can query, and the mention of Ranger authorization gives clear context. However, it does not explicitly name alternative tools or state when not to use it, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_knowledge_baseA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavior: it checks Ranger read access, conditionally forwards to RAG Studio, returns an access denied message on failure, and describes the return values for both paths. This is rich and transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with an intro, a flow explanation, and clearly labeled Args and Returns sections. Each sentence adds valuable information, and the length is appropriate for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers the tool's purpose, authorization flow, parameters, and return values. Given the absence of annotations and the complexity involving RAG and Ranger, this description is complete and self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description includes an Args section that explains each parameter with examples (user, knowledge_base, query), adding meaningful semantics beyond the bare input schema. It even outlines the return behavior based on authorization, which ties to parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Query') and resource ('RAG knowledge base'), and distinguishes itself from siblings by explicitly mentioning the Ranger authorization check. It clearly communicates the core action and scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly explains the tool's context: it performs an authorization check before querying, and describes authorization flow. It does not explicitly mention when to use this tool instead of siblings like check_access, but it provides sufficient context for an agent to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.