Server Quality Checklist

Tool Scores

• draft_ruleA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behavioral traits: deterministic, local, no network or LLM call, and describes the return envelope contents. No annotations are provided, so the description carries the transparency burden well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the core purpose. It efficiently communicates usage context and behavior without unnecessary words, though could be slightly more structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 8 parameters (1 required) and no schema descriptions, the description adequately covers the main input and optional MITRE hints but lacks detail on other parameters. The output schema is mentioned, which adds completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description only explains the 'description' and 'mitre_ttps' parameters. Other parameters like rule_type, references, target_platform, severity, title, and author are left unspecified, requiring inference from defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool drafts a sigma detection YAML rule from natural language, distinguishing it from sibling tools convert_rule and validate_rule.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: when a starting-point sigma rule is needed from a plain-English threat summary. Does not mention when not to use or discuss alternatives explicitly, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• convert_ruleA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the full burden. It discloses successful output (converted query + warnings) and failure modes (missing pySigma or backend packages yield actionable pip install commands). Does not discuss auth or rate limits, but these are irrelevant for a conversion tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main action and uses a clear structure: purpose, usage, details. It is concise with three sentences, no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 3 parameters, no annotations, and an output schema (not shown), the description covers purpose, usage, primary output, and error handling. It lacks parameter explanations but is adequate for a simple conversion tool when combined with the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description only implicitly associates yaml_content with the sigma rule and target with backend names, but does not explain the config parameter or enumerate target options. It adds limited semantic value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it converts a sigma YAML rule to a SIEM-native query string, specifying three target backends. It distinguishes from sibling tools 'draft_rule' and 'validate_rule' by focusing on conversion rather than drafting or validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use when the caller has a validated sigma rule' and lists target SIEMs. Does not explicitly exclude alternatives, but the context is clear. Mentions return of conversion warnings and error envelopes for missing dependencies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_ruleA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It describes specific checks (empty references, missing falsepositives, missing MITRE tag, vague condition). It also notes strict mode behavior and that target_backend is informational. No side effects mentioned but no destructive hints either.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise at ~80 words, front-loaded with purpose. Every sentence adds value. No repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given output schema exists (not shown), return values need not be explained. Description covers validation specifics, sibling differentiation, and parameter roles. Could briefly mention that it returns a list of issues, but output schema likely covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description compensates. It explains yaml_content is the rule to validate, target_backend is informational, and strict promotes warnings to errors. Adds meaningful context beyond bare parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates sigma YAML rules for schema correctness, pySigma compatibility, and best-practices linting. It distinguishes from siblings (convert_rule, draft_rule) by focusing on validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit use cases are given (rule drafted, pasted, or read from disk). It mentions when to use it: to check parseability, spec compliance, and quality. Does not explicitly state when not to use it but provides clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.