Server Quality Checklist

Tool Scores

• configsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description should disclose behavioral traits. It simply says 'generate,' which implies creation but does not specify idempotency, overwrite behavior, side effects, or other important characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no filler. However, it may be too terse, missing opportunities to add useful context without bloating.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description should explain return values or effects. It does not. Also, details about how configs are generated or any prerequisites are absent, leaving the agent with incomplete information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters with descriptions, so the baseline is 3. The description adds no extra meaning beyond the schema, but does not contradict it either.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates configuration files using 'enhanced standards,' which is specific and provides a verb-resource pair. However, it does not differentiate from sibling tools like 'audit-plugin' or 'diff-template,' which may have overlapping purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, or what triggers its use. The description lacks context about prerequisites, scenarios, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update-depsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must fully disclose behavior. It states it 'updates dependencies' but does not note whether it modifies files, side effects, permissions needed, or the impact of parameters like 'dryRun' or 'interactive'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that immediately conveys the tool's purpose. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and four parameters, the description is insufficient. It lacks details on return values, error handling, or behavioral nuances, making it hard for an agent to predict the outcome of invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds no additional meaning beyond the schema, e.g., it does not explain how parameters interact or what 'npm-check-updates' specifically does with them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('update dependencies'), the target ('Metalsmith plugins'), and the method ('npm-check-updates'), distinguishing it from sibling tools like 'audit-plugin' or 'validate'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., 'audit-plugin' for auditing or 'validate' for validation). There is no mention of prerequisites or context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validateB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description only says 'Check', which implies a read-only operation. It does not disclose potential side effects, permissions, or return value behavior. More detail needed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with no wasted words. The purpose is front-loaded and efficiently communicated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema and description lacks any indication of return values or what the validation produces. Given 3 parameters and no output documentation, completeness is low.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for all parameters. The description adds no extra meaning beyond what schema provides, resulting in baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a Metalsmith plugin against quality standards, with a specific verb and resource. It distinguishes from siblings like 'audit-plugin' which likely has different purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context (checking an existing plugin) but does not explicitly state when to use vs alternatives or provide exclusions. No guidance on when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diff-templateA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the tool's behavior: it diffs and reports match, missing, or drift with unified diff snippets, which covers its core operation. Since no annotations are provided, the description carries the full burden, but it omits side effects (e.g., read-only nature), authentication requirements, or limits, resulting in moderate transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise: two sentences clearly state the tool's action, output, and recommended use case. Every sentence adds value without redundancy or unnecessary detail, achieving perfect front-loading of essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two optional parameters, no output schema, and no nested objects, the description covers the key aspects: what the tool does, what the output contains (match, missing, drifted with diffs), and when to use it. It lacks details about potential mutation (likely read-only) or error conditions, but overall it is sufficiently complete for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, both parameters already have basic explanations. The tool description adds extra context: for 'templates', it explains that the filter can use template paths or target paths, and omitting diffs all tracked templates. This enriches the schema information, justifying a higher score above the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Diff') and clearly identifies the resource ('plugin against the current scaffold templates'). It details the output (reports match, missing, drifted with unified diffs) and distinguishes its purpose from sibling tools like audit-plugin or get-template by focusing on syncing aging plugins with the latest scaffold.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises using the tool 'to keep aging plugins in sync with the latest scaffold', which provides a clear use case. However, it does not explicitly state when not to use it or mention alternatives like get-template or list-templates for merely viewing templates, leaving room for ambiguity in tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• install-claude-mdA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool uses smart-merge by default (preserving customizations) and mentions replace and dryRun options. However, it does not cover all edge cases like error handling or directory creation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, under 200 characters, and front-loaded with the purpose. Every part is necessary and clearly written.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and simple parameters, the description covers the core functionality well. It explains the merge behavior, but lacks detail on error conditions or prerequisites (e.g., plugin must exist). Still, it is largely sufficient for a straightforward tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema coverage is 100%, so baseline is 3. The description adds value by explaining the behavior of the 'replace' parameter (smart merge vs. full replace) and 'dryRun' as preview mode. This goes beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it installs a CLAUDE.md in an existing Metalsmith plugin with smart-merging to preserve user customizations. This distinguishes it from sibling tools like 'audit-plugin' or 'configs' which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (install CLAUDE.md), but does not explicitly state when to use alternatives or provide exclusionary guidance. Usage is implied by the purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plugin-scaffoldA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral disclosure. It states the tool creates files and directories (src/, test/, package.json, README.md, CLAUDE.md, GitHub workflows) and warns against existing projects, implying potential overwrites. However, it lacks explicit details on prerequisites (e.g., Node.js), whether it runs npm install, or any network access for workflows, which would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at two sentences, front-loading the purpose and output files, then immediately adding usage constraints. Every sentence serves a purpose with zero redundancy or fluff, making it efficient for an AI agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, no output schema, no annotations), the description covers the core aspects: what it does, what it creates, and when to use it. It lacks guidance on post-scaffold steps (e.g., 'run npm install') or behavior with optional features, but it is sufficient for the agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing descriptions for all 5 parameters. The description adds no new information beyond the schema; it reiterates the name and description requirements and the 'no metalsmith- prefix' rule already present in the schema. For features and outputPath, it offers no additional context, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it scaffolds a complete Metalsmith plugin, listing specific files (src/, test/, package.json, etc.). It explicitly distinguishes from sibling tools by specifying 'for NEW plugins in an empty directory' and warns against using on existing projects, ensuring no confusion with audit, config, or template tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidelines: 'Use only for NEW plugins in an empty directory — never against an existing project.' It also instructs to pass the exact plugin name without adding a 'metalsmith-' prefix and notes that both name and description are required, leaving no ambiguity about when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-templatesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It clearly says 'list' and 'hand back', implying a read-only operation, but could be more explicit about side effects (none expected). Still, it's straightforward.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words. It conveys purpose and usage guidelines efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no parameters and no output schema, the description fully covers its purpose and usage context. It tells the agent exactly what to expect and how to use it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, and schema description coverage is 100%. The description adds no parameter details because none exist, meeting the baseline of 4 for no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists every template available on the server, and explicitly relates it to get-template, distinguishing it from siblings like get-template (which fetches a specific one) and show-template.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'Call this before guessing a template name.' This tells the agent when to use it, though it doesn't explicitly state when not to use it, which is acceptable given the simplicity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.