Server Quality Checklist

Tool Scores

• scan_evtxA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, description discloses key behaviors: wraps hayabusa commands, runs non-interactively, temporary file handling for output_path, rule_filter copying, and result_detail controls. Could add error handling or permissions but is comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with purpose first, then parameter details, then return. Each sentence adds value, though slightly lengthy. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given complexity (10 params, no annotations, no output schema), description covers parameters, return shape, and behavioral nuances. Lacks error cases or performance notes but is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, description fully explains all 10 parameters with context, defaults, and relationships to underlying tool. Adds meaning beyond field names, e.g., rule_filter copies files, result_detail controls preview.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it runs a Hayabusa detection scan on .evtx files, distinguishing it from sibling tool 'get_hayabusa_rules' which retrieves rules. Verb 'scan' and resource 'Windows Event Log (.evtx) data' are specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explains when to use the tool (scanning evtx files) and contrasts with sibling tool. Provides context on interactive vs non-interactive, default behaviors, and output handling. Lacks explicit when-not-to-use or alternative tools beyond the sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_hayabusa_rulesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden. It discloses case-insensitive matching, default max_results (50), and the meaning of total_matched. It does not mention read-only nature or auth requirements, but for a listing tool, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured: purpose sentence, usage paragraph, then bulleted parameters (though not in markdown), and return format. Every sentence adds value with no repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers all aspects: inputs, outputs, relationship to sibling, example usage, default behaviors. Given 3 optional params and no output schema, the description is fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description compensates fully: explains keyword filtering (case-insensitive, examples), rules_dir default path, and max_results cap with response field clarification. Each parameter gets detailed, contextualized meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List available Hayabusa/Sigma detection rules, optionally filtered by keyword.' It identifies the specific verb (list) and resource (rules), and distinguishes from sibling tool scan_evtx by explaining the relationship.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states it's 'useful for understanding what rules exist before running scan_evtx' and provides an example with keyword='mimikatz'. It could be improved by mentioning when not to use (e.g., for scanning logs), but the guidance is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.