Server Quality Checklist

Tool Scores

• known_bad_lookupA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: it is free, read-only, exact hash match only, no analysis, null results possible. It explains how to interpret matched, false, and null outcomes, and warns that a non-match is not a clean verdict.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections and headings, making it easy to scan. While slightly lengthy, every sentence adds necessary detail without redundancy. It could be trimmed slightly, but remains efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter, lack of output schema, and the description's thorough coverage of input, behavior, cost, limits, and result interpretation, it is contextually complete. No important details are missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds significant value beyond the schema: it explains that the hash is of artifact bytes (not URL or package name) and provides an example of how to compute it (shasum -a 256). This extra context justifies a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'checks a single sha256 content hash against Lazaretto's known-bad indicator store', specifying the verb, resource, and scope. It distinguishes itself from siblings check_lockfile and scan_artifact by contrasting use cases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides an explicit 'WHEN TO USE' section that tells when to use this tool (when you already have a hash) and when to use alternatives (check_lockfile for dependency trees, scan_artifact for behavioral analysis). It also advises on result interpretation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_artifactA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It discloses that the artifact is fetched without execution, analyzed deterministically with no LLM, returns verdict with evidence, costs credits, and has limitations regarding heuristics, minified code, and size. It also explains how to interpret results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-structured with clear sections (WHEN TO USE, DETECTS, COST, LIMITS, READING RESULT). It front-loads the core purpose and each section adds necessary detail. Some redundancy in the reading section could be trimmed, but overall it is effectively organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains how to interpret results (gate on risk not verdict) and provides comprehensive input, behavior, cost, and limitations. It covers all necessary context for a complex scanning tool, making it self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, baseline 3. However, the description adds extra context for each parameter: for 'ref' it gives example formats and warns to pin versions, for 'depth' it explains the difference between full and lookup, for 'content' it clarifies when used, and for 'target_type' it describes each enum value. This added value justifies a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it scans artifacts for malicious behavior and differentiates from siblings check_lockfile and known_bad_lookup by explaining when to use each. It specifies verb 'scan' and resource 'artifact' with detailed scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description has an explicit 'WHEN TO USE' section that tells when to use this tool and when to use alternatives like check_lockfile for dependency trees or known_bad_lookup for sha256. It also provides context on prerequisites and cost.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_lockfileA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but the description thoroughly discloses behavior: it's free, read-only, reads from working directory, explains limits on version ranges and non-registry entries, and how to interpret results (malicious vs unverified).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with clear sections (WHEN TO USE, COST AND EFFECTS, LIMITS, READING THE RESULT). Every sentence adds value, no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains result format (malicious, unverified) and interpretation. Covers limitations, cost, and usage comprehensively for a two-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% description coverage for both parameters, but the description adds extra context (e.g., auto-detection for path, when to use lockfile parameter) beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool checks a lockfile for known-malicious dependencies, specifies it works on exactly-pinned versions, and distinguishes from siblings (scan_artifact, known_bad_lookup).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance on when to use (before installing, auditing), when to prefer alternatives (scan_artifact for package behavior, known_bad_lookup for sha256), and limitations (only exactly-pinned).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.