Server Quality Checklist

Tool Scores

• keywords-forC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool returns a list but doesn't describe format, size limits, error handling, or performance traits (e.g., rate limits, caching). The description is minimal and lacks essential behavioral context for effective use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without redundancy. It's appropriately sized for a simple tool and front-loaded with the core action. There's no wasted verbiage, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter) and the presence of an output schema (which handles return values), the description is minimally adequate. However, it lacks details on behavioral traits and parameter meaning, which are needed for full contextual understanding despite the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 1 parameter with 0% description coverage, so the schema provides no semantic details. The description adds that the parameter is a 'syntax' but doesn't explain what constitutes valid syntax (e.g., programming language names, file formats, query types). This partial compensation is insufficient given the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'returns the keyword list for the given syntax', which provides a basic verb+resource relationship. However, it's vague about what 'keyword list' means (e.g., programming keywords, search terms, metadata tags) and doesn't distinguish from siblings like 'lookup-syntax' or 'search'. The purpose is understandable but lacks specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'lookup-syntax' or 'search'. The description implies usage when a syntax is known and keywords are needed, but it doesn't specify contexts, prerequisites, or exclusions. This leaves the agent with minimal direction for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup-syntaxB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the tool returns a record based on exact syntax matching, but doesn't describe what happens if no match is found (e.g., returns null, error), whether it's case-sensitive, performance characteristics, or authentication requirements. For a lookup tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core purpose and uses backticks to highlight the parameter name. Every element earns its place by conveying essential information about the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return values), one simple parameter, and no annotations, the description is minimally complete. It explains what the tool does but lacks behavioral details and usage guidance. For a lookup tool with output schema support, this is adequate but leaves the agent to infer important operational aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains that the 'syntax' parameter is used for exact matching, which adds meaning beyond the schema's basic type definition. However, it doesn't provide examples of valid syntax values, format requirements, or constraints. With one undocumented parameter, the description adds some but incomplete semantic context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Return') and resource ('record') with a specific matching condition ('whose `syntax` exactly matches the input'). It distinguishes from sibling tools like 'search' and 'semantic-search' by emphasizing exact matching rather than broader searching. However, it doesn't explicitly contrast with 'keywords-for', leaving some sibling differentiation incomplete.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search' or 'semantic-search'. It mentions the exact matching condition, but doesn't explain scenarios where this is preferred over other tools or any prerequisites for usage. The agent must infer usage context from the tool name and description alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the search is 'case-insensitive' and covers specific fields, but doesn't describe pagination behavior (implied by 'limit' parameter), error conditions, performance characteristics, or what the output contains. For a search tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that communicates the core functionality without any wasted words. It's appropriately sized for a simple search tool and front-loads the essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return values), simple parameters, and no annotations, the description covers the basic search scope and behavior. However, it lacks usage differentiation from siblings and doesn't fully compensate for the 0% schema coverage or missing annotations, making it minimally adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter descriptions. The tool description doesn't mention either parameter ('query' or 'limit') explicitly, though it implies the 'query' parameter through 'substring search'. It doesn't explain what 'limit' controls or provide format details. The description adds minimal value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a 'substring search' across specific fields (syntax, description, keywords) with case-insensitive matching. This is a specific verb+resource combination, though it doesn't explicitly differentiate from sibling tools like 'semantic-search' or 'lookup-syntax'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings ('keywords-for', 'lookup-syntax', 'semantic-search'). It doesn't mention any prerequisites, exclusions, or alternative scenarios, leaving the agent with no contextual usage information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• semantic-searchA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While it mentions 'semantic embeddings' and 'everyday language' queries, it doesn't describe important behavioral traits like whether this is a read-only operation, what permissions might be required, rate limits, error conditions, or how results are ranked. For a search tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise and well-structured: two sentences that directly state the tool's purpose and usage guidelines without any wasted words. Every sentence earns its place by providing essential information about what the tool does and when to use it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there's an output schema (which handles return values), the description doesn't need to explain output. However, for a search tool with 2 parameters, 0% schema description coverage, and no annotations, the description should provide more context about behavioral aspects and parameter usage. The description covers the basic purpose and usage context adequately but leaves gaps in behavioral transparency and parameter semantics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate for the lack of parameter documentation in the schema. The description mentions 'queries' which relates to the 'query' parameter, but doesn't explain the 'limit' parameter or provide any additional semantic context about parameter usage, formats, or constraints. With 2 parameters and no schema descriptions, the description adds minimal value beyond what's inferred from parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Natural language search using semantic embeddings.' It specifies the verb (search) and resource (semantic embeddings), and distinguishes it from exact keyword matching. However, it doesn't explicitly differentiate from sibling tools like 'search' or 'keywords-for', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool: 'Use this for queries in everyday language when exact keywords might not match.' This gives practical guidance on its intended use case. However, it doesn't explicitly state when NOT to use it or mention alternatives among the sibling tools, which would be needed for a score of 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.