Server Quality Checklist

Tool Scores

• search_knowledgeA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds one behavioral detail: 'Low-confidence context is removed in strict mode.' This is useful and not apparent from the schema. However, without annotations, the description should disclose more (e.g., permissions, read-only nature, pagination), which it does not.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, and the second sentence adds valuable behavioral context. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 6 parameters and no annotations, this description is too sparse. The output schema may define return values, but the description does not explain the tool's capabilities (e.g., filtering by source, project, or mode) well enough for an agent to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must compensate for parameter meaning. It clarifies that 'strict' (the confidence parameter) removes low-confidence context, but it does not explain mode, source, project, or limit parameters. This leaves a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Search governed knowledge' uses a specific verb and resource, clearly distinguishing it from sibling tools like list_knowledge_sources and rag_status. The action and object are immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied by the verb 'search', but no explicit guidance is given on when to use this tool vs. listing sources or checking rag status. Alternatives are not mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_knowledge_sourcesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It clearly states that the tool lists source classes and chunk counts and explicitly notes that it does NOT return chunk text—a key behavioral limitation. This is adequate for a read-only listing operation, though it doesn't mention any other side effects or prerequisites.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single clear sentence with no filler. It conveys the main purpose and a key limitation efficiently, making it easy to parse and act upon.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the output schema provides return structure, the description gives enough context for such a simple tool with no parameters. It doesn't define what 'source classes' exactly means, but the sibling tool names and the idea of knowledge sources provide adequate context. The omission of chunk text is explicitly addressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the schema provides no parameter details. The description adds meaningful context about the output (source classes and chunk counts) and the exclusion of chunk text, which helps the agent understand what the tool offers despite having no inputs to configure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and names the resource ('knowledge sources') with a clear output scope ('source classes and chunk counts'). The clarification 'without returning chunk text' differentiates it from the sibling search_knowledge tool, which likely returns chunk contents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use for obtaining an overview of sources without chunk text, but it does not explicitly state when to use this tool versus siblings like search_knowledge or rag_status. There is no direct comparison or exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rag_statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It discloses that the operation is 'safe' (implying read-only) and 'aggregate' (indicating summary rather than raw data), and explicitly states it does not return query or chunk text. These details go beyond a minimal description, though it lacks specifics on what health metrics are included, which is likely covered by the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence: 'Return safe aggregate health and telemetry without query or chunk text.' It contains zero filler words and every phrase adds value, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (no parameters), the presence of an output schema, and a description that clearly conveys its purpose and boundaries, the description is complete enough for an agent to understand when and how to invoke it. The statement 'without query or chunk text' provides crucial context for distinguishing it from sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so schema coverage is effectively 100% with nothing to explain. The baseline for zero parameters is 4, and the description correctly avoids any parameter-related claims that might be misleading.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'safe aggregate health and telemetry,' which is a specific verb+resource combination. It also distinguishes itself from siblings by noting it operates 'without query or chunk text,' differentiating it from search_knowledge and list_knowledge_sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: use this tool for aggregate health and telemetry, and it explicitly mentions it does not involve query or chunk text, implying it is not for content retrieval. However, it does not name alternative tools explicitly or provide a formal 'when to use' statement, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.