Server Quality Checklist

Tool Scores

• redact_phiB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full burden but only states 'detect and redact with placeholders.' It does not disclose how detection works, what patterns are matched, whether the operation is destructive, or the format of the output.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence. It is front-loaded but could be slightly expanded to include behavioral details without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that modifies data with no annotations or output schema, the description lacks completeness. It does not explain the redaction process, return value, or constraints. With three siblings, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 100% of parameters. The description adds no information beyond the schema; the parameters 'text' and 'mode' are already described adequately in the input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool redacts PHI-like identifiers with placeholders. This distinguishes it from siblings: audit_deidentification (auditing), scan_phi (scanning), and validate_no_phi (validating absence).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description does not mention prerequisites, preferred scenarios, or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_deidentificationB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It states 'audit-oriented summary,' hinting at a read-only operation, but fails to detail whether it modifies data, requires special permissions, or returns paginated results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is concise. However, it could be slightly longer to cover key details without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of output schema, the description should explain what the summary contains more precisely. Currently, it vaguely mentions 'PHI-like identifiers and limitations,' which is insufficient for an agent to anticipate the output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the parameter 'text' is described in the schema as 'Plain text to inspect.' The tool description adds no additional meaning beyond this.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'return' and resource 'audit-oriented summary of PHI-like identifiers and limitations,' which differentiates it from sibling tools like 'redact_phi' (modification) and 'scan_phi' (detection). However, it does not explicitly contrast with 'validate_no_phi'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for auditing (e.g., compliance review) but does not specify when to prefer this tool over alternatives or provide exclusions. Sibling names offer some context, but no explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_phiB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only states the tool's function without disclosing behavioral traits. It does not explicitly state that the tool is read-only or non-destructive, nor does it describe any side effects, limitations, or security considerations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no superfluous words. It is appropriately concise and front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and no output schema, the description lacks critical context. It does not describe what 'structured findings' look like, the types of PHI identifiers detected, or any constraints on the input text length or format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% since the single parameter 'text' is described in the schema ('Plain text to inspect.'). The description adds no additional meaning beyond the schema, achieving the baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Detect PHI-like identifiers and return structured findings' clearly states the tool's action (detect) and resource (PHI-like identifiers), and distinguishes it from siblings like 'redact_phi' (redact) and 'validate_no_phi' (validate absence). The verb 'scan' in the name is supported.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidelines are provided. The description does not mention when to use this tool vs alternatives (audit_deidentification, redact_phi, validate_no_phi), nor does it specify prerequisites or context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_no_phiB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description adds some value by indicating the tool checks for PHI-like identifiers but does not specify the output format (e.g., boolean, report) or behavior upon detection, limiting transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no waste, but it could be slightly more informative without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description fails to explain what the call returns. It also does not address when to use this validation versus scanning (scan_phi) for finding specific PHI items.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage for parameters is 100% with clear descriptions; the tool's description does not add meaning beyond the schema, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates whether text has no detected PHI-like identifiers, using a specific verb and resource. This distinguishes it from siblings like audit_deidentification, redact_phi, and scan_phi.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description does not mention conditions for use, exclusions, or comparison to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.