Server Quality Checklist

Tool Scores

• extract_stringsC
  2.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden for behavioral disclosure. It only states the basic extraction function but omits critical details such as whether the operation is read-only, how results are returned, edge cases like missing files, or any performance implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (10 words, single sentence), but this conciseness sacrifices essential information. It lacks structure and reads as a fragment, failing to earn its place when more detail is needed.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, no schema descriptions, and an available but unused output schema, the description is woefully incomplete. It does not address parameter roles, return value format, or how to interpret results, leaving the agent with insufficient context to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no information about any of the 5 parameters (case_id, file_path, min_length, max_results, pattern_filter). Since schema description coverage is 0%, the description fails to compensate, leaving the agent without any semantic understanding of required or optional inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (extracts), the resource (printable ASCII & Unicode strings), and the source (evidence artifact). It effectively distinguishes from sibling tools like extract_metadata and scan_iocs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of appropriate contexts, prerequisites, or exclusions, which is critical for an agent to decide between extract_strings and siblings like extract_metadata.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_iocsC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It lists the types of IOCs scanned but does not disclose whether scanning is read-only, destructive, or has side effects. No mention of permissions, runtime characteristics, or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, 22 words, front-loads the action and lists IOC categories. No wasted text; every part contributes to understanding the tool's core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While an output schema exists, the description omits essential context: file format support, size limits, relationship to case_id (e.g., required if multiple cases), and how results are structured. Incomplete for an agent to use confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It adds zero explanation for either parameter (file_path, case_id). The agent cannot infer what case_id is for or what format file_path expects.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it scans an evidence artifact for Indicators of Compromise and lists specific IOC types (IPs, C2 URLs, emails, commands, Base64). This differentiates it from siblings like extract_metadata and check_threat_intel, though it does not explicitly contrast them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus the siblings. It does not specify prerequisites, contexts, or limitations (e.g., file types, size, or that it should be used before checking threat intel).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_threat_intelB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses a safe offline fixture fallback, implying reliability. However, it does not mention if the tool is read-only, whether it may hit external APIs, or any latency implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one clear sentence, front-loading the primary action and key detail about fallback. No wasted words. Could slightly improve by adding context about parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool invokes an external database query with fallback, has 2 parameters (one unexplained), and an output schema exists, the description is adequate but incomplete. It should clarify the return value (the output schema may help, but it's not referenced).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the schema provides no descriptions. The description only mentions 'file_hash' implicitly (says 'for a file hash') but fails to explain the purpose of 'case_id' or its default value. This is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool queries threat intelligence databases for a file hash, and mentions a safe offline fixture fallback. The verb 'queries' and resource 'threat intelligence databases' are specific. However, it does not differentiate from the sibling 'scan_iocs', which may also query threat intel.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'scan_iocs'. The description does not state prerequisites, context (e.g., only for certain hash types), or when to avoid it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• extract_metadataB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavioral traits. It correctly indicates that the tool computes multiple hash types and detects MIME-type masquerading—both are non-obvious and valuable. However, it does not mention whether the tool modifies the file, requires network access, or what happens if the file is missing. The absence of annotations means the bar is higher, and this description just meets the minimum by covering key side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, information-dense sentence that front-loads the core action ('Extracts file metadata') and then lists specific capabilities (hash types, masquerading detection). No extraneous words or redundancy exist. Every phrase adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple hash computations, masquerading detection) and that an output schema exists, the description provides an overview but omits important details like whether hashes are returned as hex strings, what MIME-type detection criteria are used, or the behavior for invalid paths. With no annotations, the description carries more burden and is only partially complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema coverage is 0%, meaning neither parameter has a description in the schema. The tool description mentions 'file_path' indirectly by specifying 'Extracts file metadata', but it does not explain the 'case_id' parameter at all. Two out of two parameters lack documentation, and the description adds no value for 'case_id'. The baseline expectation is higher given zero schema coverage, so this is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's purpose: extracting file metadata and computing cryptographic hashes. It specifies the resources (file metadata, hashes) and the action (extracts), and the mention of MIME-type masquerading detection adds a distinguishing feature. However, it lacks explicit differentiation from sibling tools like 'extract_strings', which might also work on files, leaving some ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'extract_strings' or 'scan_iocs'. It does not mention prerequisites (e.g., file must exist on disk), error scenarios (e.g., unsupported file types), or when it is inappropriate to use. The context signals include sibling names, but the description itself fails to help the agent choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• system_health_checkA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility. It explicitly says 'inspects' (not modifies or deletes), and outlines three specific areas (capability readiness, storage, chain-of-custody). This is transparent about the non-destructive, read-only nature of the tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with 13 words, front-loading the key verb 'inspects' and then listing the three areas. There is no waste; every word contributes to the purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose and scope sufficiently for a zero-parameter, read-only health check tool. The output schema exists but the description doesn't need to explain return values. It could mention that the output schema details the health status, but overall it's complete for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and there are zero parameters. The description doesn't need to elaborate on parameters since there are none, but it does explain what the tool inspects, adding value beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it inspects 'real-time host forensic capability readiness, storage, and chain-of-custody status'. This provides a specific verb ('inspects') and a noun ('host forensic capability') that distinguishes it from sibling tools like scan_iocs or extract_metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a diagnostic/readiness check for forensic hosts, distinct from sibling tools that deal with extracted data or IOCs. It doesn't explicitly state when not to use it, but given its clear purpose and the zero-parameter input, the context is well-understood.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.