Server Quality Checklist

Tool Scores

• is_sensitiveA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and openWorldHint. The description adds valuable behavioral context: the tool uses an LLM (requires WARDCAT_LLM_MODEL), returns a boolean, and serves as a holistic gate. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences with front-loaded core function. No wasted words. Every sentence earns its place: purpose, prerequisite, usage advice.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one parameter and an output schema (implied boolean). The description covers the core purpose, prerequisite, and usage context. It could be improved by contrasting with sibling tools like 'scan' to avoid confusion, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'text' is a string with 0% schema coverage. The description adds purpose (checks for sensitive info) but no additional constraints or format details. Score baseline 3 for a simple parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns True if text contains sensitive information, acting as a holistic LLM gate. However, it does not explicitly differentiate from siblings like 'scan' or 'redact', which could serve similar purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use: as a guardrail before forwarding to an external service. It also mentions a prerequisite (requires on-prem LLM layer). However, it lacks guidance on when not to use or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scanA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes the output structure in detail: 'violations' summary never contains raw values, 'sanitized_text' behavior varies by action (warn vs redact/mask/hash). No contradiction with readOnlyHint annotation since the tool appears to be a pure computation returning results without side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three paragraphs and somewhat verbose. Some sentences (e.g., about server actions) could be condensed. It contains necessary detail but would benefit from tighter phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and annotations, the description covers all essential aspects: purpose, parameters, output details, and safe-to-log summary. It is sufficiently complete for the agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description fully explains both parameters: 'text' is the input, and 'entities' is an optional array to specify PII types, with clear behavior when omitted vs when a disabled type is requested. This adds essential meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it detects PII in 'text' and returns sanitized text plus a PII-free summary. Verb and resource are specific, and it distinguishes from sibling tools like 'redact' (which likely just redacts without summary) and 'is_sensitive' (which probably returns a boolean).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear guidance on when to use the 'entities' parameter and that omitting it applies all enabled filters. Also warns that requesting an unenabled type is an error. However, it does not explicitly contrast with sibling tools to help decide between scan, redact, or is_sensitive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redactA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint annotation, the description discloses behavior of each action (redact, mask, hash, warn), states that violations summary never contains raw values, and notes that under 'warn' the sanitized_text holds original text. This adds significant context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: opening sentence states purpose, then a paragraph clarifying differences from 'scan' and action details, followed by entities usage. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has three parameters (one required), an output schema exists, and the description covers key behavioral aspects and parameter usage, it is fairly complete. Minor gaps: no mention of output fields or error handling for disabled entity types, but overall sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by detailing the 'action' parameter (including four options and default), the 'entities' parameter (subset of types), and implying the 'text' parameter's role. This adds comprehensive meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool detects PII in text and anonymizes matches. It distinguishes itself from sibling 'scan' by noting that with 'redact' you pick the action per call, and it lists specific actions. This provides a specific verb-resource pair with clear differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly contrasts with 'scan' to guide usage, and explains when to use the 'entities' parameter for subset selection. However, it does not provide explicit when-not-to-use conditions beyond the scan comparison.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• server_infoA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, and the description adds detail on what is reported (entity types, default action, NER/LLM layers). No contradiction; description enhances transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. First sentence states purpose, second adds benefit. Highly concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no parameters and an output schema, the description sufficiently explains the tool's purpose and output. No missing context for a simple info tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so baseline 4 applies. Schema coverage is 100%, so the description does not need to compensate for missing param info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it reports server detection capabilities and anonymization defaults, listing specific items like entity types and NER/LLM status. The verb 'report' and resource 'server info' are specific, and the tool distinguishes itself from siblings (is_sensitive, redact, scan) which are data processing tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  States the tool lets an agent discover capabilities without trial and error, implying it should be used before other tools. While not explicitly naming alternatives, the context with sibling tools makes the usage clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.