Server Quality Checklist

Tool Scores

• file_referenceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must communicate behavior. It states 'Save' which indicates a write operation, but does not disclose side effects such as overwriting, duplicate handling, or permission requirements. The description is not misleading, but lacks depth beyond the core action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two focused sentences. The first sentence states the purpose with examples, and the second lists specific use cases. Every sentence adds value, with no repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a simple 4-parameter schema and no output schema. The description covers the main purpose and use cases. However, it could be more complete by explicitly contrasting with sibling tools (e.g., 'for searching, use search_references') or clarifying what happens after saving (e.g., success message). Overall, it's sufficient for this low-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all four parameters (tags, title, content, filename) are already documented in the schema. The description adds no extra parameter semantics beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Save' and the resource 'reference document' to the knowledge base, with examples of content types (canonical pattern, version guide, architecture notes). This distinguishes it from sibling tools like search_references (read) and file_bug (bug-specific write).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence provides explicit use cases: 'working integration patterns, version compatibility matrices, and architectural decisions.' It does not explicitly name alternatives, but the 'save' vs. 'search' contrast with siblings implies when to use this tool. A brief exclusion (e.g., 'for bugs use file_bug') would elevate it to a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_past_bugsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavioral traits. It discloses that the knowledge base currently indexes 8 documents, giving a sense of coverage. It does not describe return format, error behavior, or performance characteristics, but for a read-only search tool, this is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is four sentences, each with a purpose: purpose, query guidance, indexing status, and usage recommendation. No redundant or vague phrases; front-loaded with the core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter search tool with no output schema, the description covers purpose, query construction, and when to use it. The lack of return format details is not critical given the simplicity, but a bit more about result format would help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with query and maxResults both having descriptive text including examples. The description adds context about what to query but does not add meaning beyond the schema; baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches a 'code_failures knowledge base' for bugs already solved, using the specific verb 'search' and naming the resource. It distinguishes itself from sibling tools like search_references and file_bug by focusing specifically on past bug fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs to use this tool before debugging any integration issue, providing strong when-to-use guidance. It also lists appropriate query types (error messages, symptoms, library names, keywords). It does not explicitly contrast with sibling tools, but the directive is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_referencesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It explains the content scope (canonical patterns, version guides, integration references) but does not explicitly state that it is a read-only operation, what the response format is, or any potential limitations. The description adds some behavioral context but misses explicit safety/return disclosures expected without annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) and front-loaded with the core action and resource. The 'USE THIS' section adds practical guidance without extraneous wording. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with two fully documented parameters and no output schema, the description provides sufficient context: what it searches, what kind of content, and example use cases. It could be more complete by mentioning the result format (e.g., list of reference snippets), but the named tool and examples make expectations reasonably clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents the 'query' and 'maxResults' parameters. The description's example queries align with the schema's parameter examples but do not add new semantic details beyond what is already in the input schema. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's function: searching a 'code_failures knowledge base' for canonical working patterns, version guides, and integration references. This specific verb+resource combination distinguishes it from sibling tools like search_past_bugs (which targets bugs) and file_reference (which files references).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes 'USE THIS' with concrete example queries ('what's the correct CopilotKit + AG-UI setup?', 'which versions are compatible?'), providing clear context on when to use the tool. However, it does not explicitly state when NOT to use it or mention alternatives, leaving the differentiation to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• file_bugA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full weight. It discloses the file write location ('code_failures/bugs/'), auto-indexing behavior, and the important user-confirmation requirement. However, it does not mention potential overwrite behavior for an existing shortName filename or any other edge-case side effects, so there is a slight gap for a write operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet informative: a clear purpose, a timing directive, storage details, and a crucial safety warning. Each sentence adds value without redundancy. The formatting with 'IMPORTANT' draws appropriate attention to the confirmation requirement, and the total length is appropriate for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 10 parameters, all documented in the schema, and no output schema, the description provides all necessary context: what it does, when to use it, where it writes, and a critical user-confirmation step. The absence of return-value details is acceptable given there is no output schema, and the usage context is fully covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the base score is 3. The description adds no parameter-specific insights beyond what are already in the schema, but it also does not need to since every parameter has a clear description. The tool-level details like file location and confirmation are separate from parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-resource pair: 'Save a newly discovered bug to the knowledge base.' It clearly distinguishes from siblings like search_past_bugs and file_reference by targeting bug reports specifically. The mention of writing to 'code_failures/bugs/' and auto-indexing further clarifies its unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is given: 'Call this AFTER solving a hard bug to capture the knowledge for future sessions.' It also includes a critical timing/consent directive: 'IMPORTANT: Show the user the generated content and get confirmation before calling.' This clearly indicates when to use the tool and the required prerequisite before invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.