Server Quality Checklist

Tool Scores

• promptshield.evaluate_policyC
  2.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and description only states output behavior. Does not disclose side effects, authorization needs, or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence is concise and front-loaded, but severely under-specifies the tool's functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 0% schema coverage, no output schema, and no annotations, the description fails to provide necessary context for proper invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description does not explain the required 'payload' object or its expected structure. The schema allows any properties, so the agent has no guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it returns a normalized action recommendation, but 'normalized action' and 'policy' are vague. It distinguishes from sibling scanning tools but lacks specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like scan_text or redact_text. No context on prerequisites or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• promptshield.redact_textC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only states the action, omitting details on what constitutes sensitive spans, redaction format, side effects (e.g., mutability), permission requirements, or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and front-loaded. However, it is too minimal to be maximally effective, lacking necessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one complex parameter, no schema descriptions, no output schema, and no annotations, the description is severely incomplete. It fails to explain how to use the tool, what inputs are expected, or what outputs occur.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'payload' is an object with additionalProperties true and no schema description. The description adds no meaning beyond 'from text', failing to specify expected structure or how to provide text for redaction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it scans and redacts sensitive spans from text, which is a clear verb+resource combination. It distinguishes from siblings: evaluate_policy, scan_messages, and scan_text do not mention redaction. However, 'sensitive spans' is somewhat vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus its siblings. The description provides no context for when redaction is appropriate or when to prefer alternative tools like scan_text for scanning only.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• promptshield.scan_messagesC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It states that role and message indexes are preserved, which is a positive behavioral trait. However, it does not disclose whether the tool modifies data, requires authentication, or has side effects, leaving significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise, but it sacrifices helpful detail. It is front-loaded with the key point, but may be too terse to be useful without additional context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one free-form parameter, no output schema, no annotations, and multiple siblings, the description is far from complete. It lacks details on the return format, how the payload should be structured, and how it differs from promptshield.scan_text.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero description coverage (0%), and the description adds no meaning to the 'payload' parameter. The payload is a free-form object with no defined properties, and the description fails to explain its expected structure or content.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies 'scan chat messages' and mentions preserving role and message indexes, which clearly indicates the resource (chat messages) and a key behavior. However, the verb 'scan' is somewhat ambiguous (could imply reading, checking, or analyzing), and it doesn't fully distinguish from sibling tools like 'scan_text'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. There is no mention of prerequisites, exclusions, or comparison to sibling tools like promptshield.evaluate_policy or promptshield.scan_text.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• promptshield.scan_textC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose behavior. It mentions what is scanned for but omits details on return value, side effects (read-only or destructive), or whether it modifies the input. The description is too minimal to fully inform an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence, but it sacrifices necessary detail for brevity. While concise, it is under-specified for the complexity of the tool (nested object parameter, no schema description).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, output schema, and 0% schema parameter coverage, the description is extremely incomplete. It fails to explain the parameter, return format, or how it compares to siblings. The tool is not adequately specified for reliable agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description does not explain the 'payload' parameter. It is an object with additionalProperties: true, but no guidance on expected structure (e.g., keys, format of text). Agents cannot infer how to correctly invoke the tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans text for PII, secrets, and harmful content. The verb 'scan' and target 'text' are explicit, and it distinguishes from siblings like 'redact_text' (modification) and 'evaluate_policy' (policy evaluation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like promptshield.scan_messages or promptshield.redact_text. No context about prerequisites or typical use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.