Server Quality Checklist

Tool Scores

• find_gapsC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description is brief and does not disclose any behavioral traits beyond the basic function. Since there are no annotations, it fails to inform the agent about side effects, read-only status, or expected output characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise (one sentence) and front-loads the purpose, but it may be too terse, sacrificing necessary detail for brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that finds gaps in a knowledge graph with two parameters and an output schema, the description lacks details on what constitutes a gap, how results are presented, and how to use the parameters correctly. It is insufficient for complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate by explaining parameter meanings. It does not; 'required_frameworks' and 'api_key' are not described, leaving the agent without guidance on what values to provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds gaps in the knowledge graph, specifying it looks for missing expected relationships or nodes. This conveys a specific purpose and distinguishes it from siblings that add nodes/edges or search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like semantic_node_search or trace_compliance_chain. There is no mention of prerequisites or specific contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trace_compliance_chainC
  2.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only hints at traversal direction but does not disclose side effects, auth requirements, error handling, or whether the tool is read-only. Not enough for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely terse (one sentence), but lacks essential details about parameters and behavior. Under-specification undermines conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the description fails to explain the tool's operation, output structure (beyond 'chain'), or how parameters affect results. Incomplete for a tool with three parameters and no annotation support.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides zero information about the three parameters: start_node_id, max_depth, and api_key. Agent cannot infer their formats, constraints, or purposes from the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool traces compliance chains from requirement through controls to evidence, specifying the directional path. This distinguishes it from sibling tools like add_edge (graph manipulation) and find_gaps (finding missing links).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool vs alternatives like find_gaps or semantic_node_search. No contexts, exclusions, or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_nodeC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description carries full burden but only states basic action. No mention of side effects, idempotency, required permissions, or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is concise, but lacks structure (e.g., no bullet points, no parameter descriptions). Could be expanded without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having output schema and nested objects, description omits return value details, parameter interactions, and usage context. Significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet description only adds vague references to embeddings and metadata. It fails to explain api_key, node_id, or the properties object structure. Parameters remain largely undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it adds a node to the knowledge graph, specifying included elements (properties, embeddings, metadata). This verb-resource pair distinguishes it from siblings like add_edge.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives (e.g., semantic_node_search for querying, trace_compliance_chain for traversing). No preconditions or exclusions mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• semantic_node_searchC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions semantic matching but fails to disclose prerequisites (e.g., embeddings must exist), required permissions, or behavior in edge cases like no matches. The api_key parameter is mentioned in schema but not explained in description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with key verb and resource. Appropriate length for the information provided, though lacks detail in other dimensions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists, so return format is covered, but the description omits critical context: how api_key is used, what top_k controls, what query format to use, and preconditions. With 3 parameters and no annotations, the description is insufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no information about the three parameters (query, top_k, api_key). Their semantics are entirely dependent on schema titles and defaults, which are minimal.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (Search), resource (nodes), and method (semantic similarity matching against stored embeddings), which distinguishes it from sibling tools like add_edge or trace_compliance_chain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool vs alternatives, or when not to use it. The description is just a single statement of function without context or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• add_edgeC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It states creation but omits side effects, authentication requirements (api_key not mentioned), error conditions, or what happens if nodes are missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loaded with the core action, but could be expanded slightly for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 5 parameters (3 required), no annotations, and an output schema not referenced, the 12-word description is inadequate. It lacks context on prerequisites (node existence), permission requirements, and return value format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description only restates parameter terms ('relationship type' for relation, 'weight') without adding meaning about formats, constraints, or usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Create a directed edge between two nodes with relationship type and weight,' which is a specific verb+resource combination and clearly distinguishes from siblings like add_node.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings (add_node, find_gaps, etc.), nor any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.