Server Quality Checklist

Tool Scores

• group_threat_actorC
  2.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description is the sole source of behavioral info. It states the tool returns structured assessment, regulation refs, and recommended actions, but does not disclose whether the operation is read-only, requires special permissions, or has side effects. The lack of transparency leaves the agent uncertain about impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, using three lines to cover purpose, args, and returns. It is efficiently structured, though the initial fragment is slightly awkward. Every sentence adds some information, but the first sentence could be clearer.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description lacks differentiation from sibling tools and fails to explain how this tool fits into a workflow. With two parameters, an output schema, and related tools, the description should provide more context on use cases and relationships, but it is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description adds some meaning: 'query: Optional query parameter (regulation ref, identifier, or input data)' and 'api_key: Optional MEOK API key for Pro+ tier features.' While this clarifies parameter purpose, the descriptions are still vague (e.g., 'input data' is broad). It partially compensates for the schema gap but not fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Threat actor groups (G-codes) using a technique' is a fragment that lacks a clear verb or action. It suggests the tool retrieves groups by technique, but does not definitively state what the tool does. The title and name are vague, and the purpose is not explicitly stated, making it difficult for an agent to understand the tool's primary function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings (list_tactics, map_incident, mitigation_lookup, query_technique). There are no explicit conditions, exclusions, or alternatives mentioned, leaving the agent without context to decide between tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• mitigation_lookupC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations, so description must disclose behavior. It mentions required API key for Pro+ features but does not specify if tool is read-only, has rate limits, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is short but includes training-style Args block with minimal extra value. Could be tightened to one sentence without losing clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists, so return details are not required. However, description lacks context on input-output mapping, default behavior, and edge cases like empty results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description adds meaning: query is 'regulation ref, identifier, or input data' and api_key is 'for Pro+ tier features'. Partially compensates but could be more precise.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states it returns mitigations (M-codes) for a technique, but 'technique' is not defined and does not differentiate from sibling tools like query_technique or map_incident.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The query parameter is described as optional but no context on when to provide it or omit it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_techniqueB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It mentions the tool returns JSON with structured assessment and mentions an optional API key for Pro+ features, but does not disclose whether the tool is read-only, has side effects, or any rate limits. The description is adequate but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of a single sentence followed by a brief Args/Returns section. It is front-loaded with the core purpose and avoids unnecessary detail, though the bullet-style list could be more integrated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  An output schema exists (not shown), so the return value description is optional but the description provides a high-level summary. However, the usage of 'query' is inconsistently described (ID/name vs. regulation ref), and there is no explanation of behavior when query is empty or malformed. The tool is simple, but completeness suffers from these ambiguities.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It states 'query' can be a regulation ref, identifier, or input data, and 'api_key' enables Pro+ features. However, it does not clarify the expected format for IDs or names beyond 'Txxxx', nor does it explain the meaning of 'input data'. This adds some value but leaves gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it queries ATT&CK techniques by ID or name, distinguishing it from sibling tools focusing on threats, tactics, incidents, and mitigations. However, the mention of 'regulation ref, identifier, or input data' as query options adds ambiguity beyond the stated purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for querying techniques but provides no explicit guidance on when to use this tool versus siblings. There is no mention of prerequisites, context, or exclusion criteria, leaving the agent to infer from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tacticsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states 'Returns: JSON with structured assessment...', lacking details on side effects (none expected), authentication requirements, rate limits, or whether it's read-only. The description is minimally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the key purpose. The Args and Returns sections are clearly structured. No redundant information is present. It earns high marks for efficiency, though the lack of a title is noted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has only two optional parameters and an output schema exists, the description covers the basics: purpose, parameters, and return type. However, it omits behavioral context (e.g., pagination, limits) and does not integrate with sibling tools. It is adequately complete for a simple list tool but not exceptional.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds valuable meaning: 'query' is explained as referencing regulation ref, identifier, or input data; 'api_key' is for Pro+ tier features. This goes beyond the schema's empty defaults, though some ambiguity remains (e.g., 'input data' is vague).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'All 14 enterprise tactics (TA0001-TA0040+)', providing a specific verb (list) and resource (enterprise tactics) with a clear scope. It distinguishes itself from siblings like query_technique or group_threat_actor by naming the exact set of tactics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions optional query and api_key parameters, implying usage for filtering or Pro+ features, but does not provide explicit guidance on when to use this tool versus alternatives (e.g., when to use query_technique instead). No when-not-to-use conditions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• map_incidentA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description partially covers behavioral aspects: it notes the optional api_key for 'Pro+ tier features' and specifies the return type. However, it does not disclose idempotence, rate limits, or side effects, leaving gaps for an agent to assess safety.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with a clear docstring format: a one-line purpose followed by Args and Returns sections. It contains no redundant information, though it could be slightly more compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (2 optional parameters) and the existence of an output schema (which covers return details), the description adequately explains inputs and mentions the JSON output structure. It does not need to elaborate further on outputs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Since schema description coverage is 0%, the description adds significant meaning: 'query' is explained as 'optional query parameter (regulation ref, identifier, or input data)' and 'api_key' as 'optional MEOK API key for Pro+ tier features', which are not present in the input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Map incident IOCs/behaviors to ATT&CK techniques', specifying both the action and the resource. It distinguishes from sibling tools like 'query_technique' (which directly queries techniques) and 'list_tactics' (which lists tactics).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives such as 'group_threat_actor' or 'mitigation_lookup'. It only implies usage for mapping incidents to techniques, but lacks context on prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.