Server Quality Checklist

Tool Scores

• test_match_code_ruleA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context by mentioning the internal call ('Internally calls: ast-grep scan --inline-rules <yaml> --json --stdin'), which hints at the tool's execution method and output format (JSON). However, it doesn't cover aspects like error handling, performance implications, or side effects, leaving gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, the second adds usage context, and the third provides internal implementation details. Every sentence earns its place by adding value without redundancy, making it efficient and well-structured for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is somewhat complete but has gaps. It explains the purpose and internal call but lacks details on output values, error cases, or integration with sibling tools. Without an output schema, the description should ideally hint at return types, but it only mentions JSON format indirectly, leaving room for improvement in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for both parameters ('code' and 'yaml'). The description doesn't add any parameter-specific semantics beyond what the schema provides, such as format details or examples. Since the schema does the heavy lifting, the baseline score of 3 is appropriate, as the description doesn't compensate or enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Test a code against an ast-grep YAML rule.' It specifies the verb ('test') and resources ('code' and 'ast-grep YAML rule'), making the action clear. However, it doesn't explicitly differentiate from sibling tools like 'find_code_by_rule', which might have overlapping functionality, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some usage context: 'This is useful to test a rule before using it in a project.' This implies when to use the tool (for testing rules pre-deployment) but doesn't specify when not to use it or mention alternatives like sibling tools. The guidance is implied rather than explicit, lacking detailed comparisons or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_codeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: the tool internally calls 'ast-grep run', explains output formats (text and JSON) with examples, clarifies that 'max_results' limits complete matches (not lines), and shows how limited results are displayed. It covers execution, output, and limitations well, though it doesn't mention error handling or performance aspects like rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized, with clear sections for purpose, usage guidelines, internal details, output formats, and examples. It uses bullet points and examples effectively, though it could be slightly more concise by integrating some details (e.g., internal call info) more seamlessly. Every sentence adds value, but minor trimming is possible.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, no annotations, no output schema), the description is largely complete. It covers purpose, usage, behavior, and outputs in detail, compensating for the lack of annotations and output schema. However, it doesn't explicitly address potential errors (e.g., invalid patterns or paths) or prerequisites (e.g., tool installation), leaving a small gap in contextual coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so the schema already documents all parameters thoroughly. The description adds minimal parameter semantics beyond the schema, such as noting that the pattern is for 'simple and single-AST node result' and providing example usage with 'pattern' and 'max_results'. However, it doesn't significantly enhance understanding of parameters like 'project_folder' or 'language' beyond what the schema provides, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find code in a project folder that matches the given ast-grep pattern.' It specifies the verb ('Find'), resource ('code'), and scope ('project folder'), and distinguishes it from sibling tools by noting that 'find_code_by_rule' is for more complex usage. This is specific and avoids tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool versus alternatives: 'Pattern is good for simple and single-AST node result. For more complex usage, please use YAML by `find_code_by_rule`.' It also mentions internal implementation details and includes example usage, clearly defining the context and exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_code_by_ruleA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: the internal implementation ('Internally calls: ast-grep scan'), output formats with detailed examples, how max_results works ('limits the number of complete matches'), and handling of limited results. It doesn't mention error conditions or performance characteristics, but covers most essential operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: purpose statement, comparison to sibling tool, usage tip, implementation details, output formats with examples, parameter behavior explanation, and example usage. While comprehensive, some sentences could be more concise, and the example usage section is quite detailed but necessary for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 parameters, no annotations, and no output schema, the description provides substantial context: purpose, differentiation from siblings, behavioral details, output format explanations with examples, and parameter usage guidance. It covers what the tool does, how to use it, and what to expect, though it doesn't document the exact structure of returned data beyond format descriptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal parameter-specific information beyond the schema - it mentions YAML rule requirements ('must have id, language, rule fields') and provides example usage with parameters, but doesn't significantly enhance understanding of individual parameters. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find code using ast-grep's YAML rule in a project folder.' It specifies the verb ('Find'), resource ('code'), and method ('using ast-grep's YAML rule'), and explicitly distinguishes it from the sibling tool 'find_code' by stating it's 'more powerful' and 'more advanced.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool versus alternatives. It states it's 'more powerful than simple pattern' and 'a more advanced search tool than the simple `find_code`,' directly comparing it to a sibling tool. It also includes a tip for using relational rules and example usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dump_syntax_treeA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of behavioral disclosure. It does well by explaining the tool's internal implementation ('Internally calls: ast-grep run...'), which reveals it's a wrapper around a command-line tool. However, it doesn't mention potential side effects, error conditions, or output format details beyond the format parameter options.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It starts with the core purpose, provides usage guidance, explains parameters with practical examples, and ends with implementation details. While slightly longer than minimal, every sentence adds value and the information is front-loaded with the most important details first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a debugging tool with 3 parameters and no output schema, the description provides good context about what the tool does, when to use it, and how parameters affect behavior. It could be more complete by describing the output format or potential error cases, but given the tool's relatively simple purpose and good parameter documentation, it's mostly sufficient for an agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3, but the description adds significant value by explaining the practical meaning of the format parameter options (cst for 'debug target code' and pattern for 'debug pattern rule'), which goes beyond the schema's technical enum values. It also clarifies that code and language are 'self-explanatory' and provides context about what the tool actually does with these parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('dump code's syntax structure' and 'dump a query's pattern structure') and distinguishes it from sibling tools by specifying it's for debugging syntax/pattern discovery rather than finding or testing code. It explicitly mentions this is for 'debugging a rule' which differentiates it from find_code, find_code_by_rule, and test_match_code_rule.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Call it when debugging a rule') and offers clear alternatives for different debugging scenarios with the format parameter (use format=cst for debugging target code, use format=pattern for debugging pattern rule). This gives the agent specific decision criteria for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.