Server Quality Checklist

Tool Scores

• verify_lockfileB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description mentions scanning but not output format or behavior upon finding suspicious packages (e.g., returns list or errors). Lacks disclosure of read-only nature or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with front-loaded verb 'Scan'. Efficient but omits expected output details; still earns high score for brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Lacks output schema and doesn't describe return values. For a scan tool, agent needs to know if result is a list of issues or a boolean. Incomplete for making informed invocation decisions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67% (2 of 3 params described), but description adds no extra meaning beyond schema (e.g., no hint on how to set format enum or optional ecosystem). Baseline for partial coverage is 3, but description fails to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states action (scan lockfile), target (direct + transitive deps), and purpose (detect hallucinated/typosquatted/suspicious packages). Differentiates from siblings by specifying scanning entire lockfile vs. individual package checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises to call this before running install and to use instead of trusting LLM-generated lockfiles. Lacks explicit when-not-to-use or alternative tool references, but context from sibling tools implies differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_packageA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It adequately describes the tool's behavior (check existence and risk, return status) and gives interpretation clues, but does not disclose potential side effects (e.g., network calls, rate limits) or permissions required. Since it's a read-only check, these gaps are minor but present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no fluff. The first sentence states the core purpose, the second summarizes return values, and the third provides actionable interpretation. Information is front-loaded and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 2 parameters (name required, ecosystem optional), no output schema, and no annotations. The description covers the purpose, return categories, and interpretation, but omits mention of the 'ecosystem' parameter and does not fully specify the return structure (e.g., exact risk details or flags format). For a security-focused tool, this is adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50%, with only the 'name' parameter having a description (including an example). The description adds no additional parameter-specific details beyond the schema; it does not explain the 'ecosystem' parameter or its defaults/values, relying on the schema's enum and default value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (check) and resource (a single package), and explicitly distinguishes from sibling tools by specifying it assesses typosquat risk before installation, which other tools (check_workflow, score_manifest, verify_lockfile) do not do.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by stating 'BEFORE installing it', implying the tool is intended for pre-installation safety checks. It also gives interpretation hints for non-existent names and names close to popular packages, but does not explicitly mention when not to use it or compare to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_workflowA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It explicitly lists the security checks performed (mutable action pins, compromised actions, untrusted owners, etc.) and states the return values (risk score and merge recommendation). It does not explicitly state it is read-only or has no side effects, but the nature of validation implies no modifications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the key purpose ('Validate a CI workflow before merging') and then lists specific checks and return value. It is concise with no wasted words, though it could be slightly more structured with bullet points for readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 parameters, no output schema), the description provides sufficient context: it specifies the input (workflow YAML), the checks performed, and the output (risk score and recommendation). It does not detail the return format beyond that, but it is complete enough for an agent to understand its function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 2 parameters (platform and workflow_content) with 50% description coverage (only workflow_content has a schema description). The tool description does not add any additional meaning or syntax details beyond the schema. For low schema coverage, the description should compensate but fails to provide extra guidance on parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'validate', the resource 'CI workflow (GitHub Actions / GitLab CI YAML)', and the context 'BEFORE merging a PR that touches it', effectively differentiating it from sibling tools like check_package, score_manifest, and verify_lockfile which operate on different artifacts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'before merging a PR' and lists specific security checks being performed, providing clear context for when to use the tool. It does not, however, explicitly state when not to use it or mention alternative tools for similar tasks, though the sibling tools are clearly different in domain.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_manifestA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the full burden. It clearly explains what the tool does (scores for poison/backdoor, credential over-reach, drift) and its output format. Since it's a read-only scoring tool, no side effects are expected, and the description does not contradict any annotations (none exist).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and context, followed by output specification. Every part earns its place with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description comprehensively defines the tool's purpose, input types, output format (risk score plus recommendation), and the optional drift scoring feature. It is complete for a security scoring tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75% (one parameter missing description). The description adds context beyond the schema by explaining the overall scoring purpose and output. It does not detail each parameter, but the enum values for manifest_type are self-explanatory, and the other parameters have descriptions in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scores manifests for security issues (poison/backdoor, credential over-reach, drift) and specifies the output (risk 0-100 + install recommendation). It distinguishes from siblings like check_package by focusing on manifest files before installation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'BEFORE installing a third-party agent extension', giving a clear when-to-use context. It does not explicitly state when not to use or list alternatives, but the sibling names imply different purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.