Server Quality Checklist

Tool Scores

• package_release_toolA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and excels. It discloses mutating behavior (writes to output, overwrites redacted source), guarantees nothing under the original path is modified, notes local execution/no network calls, admits the tool doesn't detect PII itself, and explicitly states the EU AI Act summary is a draft template, not a compliance guarantee. Error handling is also described. This is comprehensive behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though long, the description is well-structured and every sentence earns its place. It starts with a concise summary, then covers usage guidance, behavioral details, parameter semantics, return format, and error behavior in clear paragraphs. The inclusion of example calls is practical and not redundant. No fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, 0% schema coverage, no annotations, and a non-trivial side-effect profile, the description is remarkably complete. It explains all parameters, return values, error behavior, side effects, limitations, and even points to a CLI reference. The output schema may exist but the description still provides a clear explanation of the JSON return shape, making it self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does fully. It explains the meaning of `path`, `output`, `kind` (with allowed values), `redact_first`, and `strategy` (with allowed values and applicability condition). It also provides three example invocations that clarify parameter combinations. All five parameters are meaningfully described.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource statement: 'Scan a dataset/model directory, redact it, and generate the paperwork needed to publish it, in one call.' It immediately differentiates from siblings by naming exact alternatives: 'Prefer `scan_directory_tool` alone for a read-only report, or `redact_directory_tool` alone when you don't need the generated cards.' This fully clarifies what the tool does and how it is distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'use it when the goal is "make this directory publishable" rather than just inspecting or redacting it.' It also gives concrete conditions for alternatives and even includes example calls to illustrate common use cases. This goes beyond simple context and provides clear decision criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redact_directory_toolA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses mutation scope: 'never edits, moves, or deletes anything under path', overwrite semantics, silent overwrite risk, local-only execution, non-idempotency, and error return behavior. This goes well beyond what annotations would provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but every section earns its place: purpose, usage, safety, parameter details, examples, and return format. It is well-structured and front-loaded with the core action, then branches into necessary caveats rather than burying the main point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 4-parameter tool with no annotations and no schema descriptions, this description is remarkably complete. It covers sibling alternatives, operation details, failure modes, return value structure, and example calls, leaving almost no ambiguity for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It explains path must exist/readable, output collision/overwrite behavior, strategy values ('mask', 'hash', 'remove') with concrete examples, and overwrite semantics. Example calls demonstrate correct usage for all parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Scan a directory for PII/secrets with Presidio, then write a redacted copy to a new location.' It clearly distinguishes itself from siblings by explicitly naming scan_directory_tool for reports and package_release_tool for the full bundle.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance ('Call this once you already know...') and when-to-avoid ('Skip it if you only need a report...' or need the Hugging Face/EU AI Act bundle). It also clarifies that a prior scan step is unnecessary, which directly addresses an alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_directory_toolA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It thoroughly discloses behavior: read-only, no network requests, safe to call repeatedly, deterministic results, returns an error object instead of raising on missing paths, and explains limitations (no custom regex). It also details file types scanned and running entirely locally.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  While long, every sentence adds value. The structure logically moves from purpose to usage guidelines, parameter semantics, examples, and return format. It is front-loaded with the essential purpose and then provides comprehensive yet non-redundant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers not only the three parameters but also error handling, return schema, and behavioral guarantees. It references the sibling tools and provides additional CLI help for deeper detail. Given no annotations and a simple output schema, this is fully complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates fully. Each parameter is explained in detail: path conditions and file types, spacy_model defaults and installation requirement, and score_threshold range and effect on false positives/recall. Example calls illustrate usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Recursively scan a local dataset or model directory for PII and secrets before you publish it.' It clearly distinguishes from siblings by positioning itself as the read-only first step ahead of redact_directory_tool or package_release_tool, and explicitly states when it is the right standalone choice (for a report only).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit when-to-use guidance: 'Call this before releasing, uploading, or sharing a dataset/model directory' and contrasts with siblings. It also includes exclusions: 'Do not call it on directories you do not have read access to, or expect it to catch anything beyond what Presidio's own recognizers detect.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.