Server Quality Checklist

Tool Scores

• get_billing_usageA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry full weight. It discloses that it retrieves statistics, but does not clarify whether this is a read-only operation (likely safe), what data is returned (cache stats, recent run timing), or any potential side effects. It doesn't mention rate limits, authorization needs, or whether it only works for paid plans. This is a basic functional description with no behavioral depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the verb and resource. It is concise and avoids redundancy, but it could potentially be split into two sentences for clarity (e.g., separating the scoping condition). Still, it is efficient with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple with only 2 params and no output schema. The description covers the main purpose and scoping, which is adequate for a read-only retrieval tool. However, given no annotations and no output schema, it would benefit from mentioning whether the returned data is summarized or detailed, or noting that repo-scoped usage may not include owner-level billing aggregates. Completeness is acceptable but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both parameters having descriptions. The description adds that 'repo' returns 'repo-level cache stats and recent run timing,' which supplements the schema by specifying the effect of providing the optional repo. However, it doesn't detail the exact format of 'owner' (e.g., case sensitivity) or the structure of returned data, but it adds meaningful scoping context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Retrieve' and the specific resource 'GitHub Actions billing and cache usage statistics', and it distinguishes scope options (owner vs. repo). It differentiates from siblings like 'list_workflow_performance' and 'analyze_workflow_config' by targeting billing/cache, while siblings focus on performance/config.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying optional scoping to a repository, but it does not explicitly state when to use this vs. siblings, nor provide exclusions. There is no mention of prerequisites (e.g., required permissions) or when owner-only vs. repo-scoped is appropriate. Context like billing queries is implied but not elaborated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_workflow_configA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of disclosure. It says the tool 'parses' and 'identifies' optimization opportunities, implying read-only analysis. However, it doesn't state whether it modifies anything, whether it requires valid YAML, or what happens on invalid input. The description doesn't contradict annotations, but it adds minimal behavioral detail beyond the action itself.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that packs substantial information: the input format (YAML string), the action (parse and identify optimization opportunities), and specific examples of what it looks for. Every element earns its place, and it's front-loaded with the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given it's a single-parameter tool with a descriptive schema, no output schema, and no annotations, the description covers the essential aspects: input format, purpose, and analysis areas. It's quite complete for its simplicity. The only gap is specifying expected output format, but since no output schema exists, the description should ideally mention what the analysis returns, though it's not critical for invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'workflow_content' has a clear schema description 'Full YAML content of the GitHub Actions workflow file', and schema coverage is 100%. The description adds the context of what the tool does with it (parse and analyze) but doesn't add format expectations beyond what the schema says. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool parses a GitHub Actions workflow YAML string and identifies optimization opportunities, listing specific areas (caches, matrix strategy, concurrency, dependency installs, artifacts). This is a specific verb 'analyze' with a clear resource 'workflow config' and distinct purpose from siblings like list_workflow_performance and get_billing_usage, which focus on performance listing and billing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use: when you have a workflow YAML string to analyze for optimization. It does not explicitly contrast with siblings or mention when not to use. While the purpose is clear, there is no explicit guidance on alternatives or exclusions, so it's adequate but lacks depth.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_workflow_performanceA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses that it fetches runs and computes statistics, but does not mention API rate limits, authentication needs, or what happens with insufficient data. The description adds some behavioral context (computing stats) but omits other factors like data retention or pagination.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first states the function, the second states the use case. No waste, fully front-loaded, concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given it's a computation tool with no output schema and no annotations, the description effectively conveys purpose and usage. Missing details like output format or edge cases (e.g., no runs found) are not critical given the tool's simplicity. It covers essential aspects for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all four parameters. The description adds no additional parameter-level details beyond what the schema provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches the last N workflow runs and computes job-level timing statistics, which is a specific verb+resource combination. It also distinguishes itself from siblings like analyze_workflow_config and get_billing_usage by focusing on performance statistics derived from runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for performance analysis ('Useful for identifying slow jobs and trends') but does not explicitly say when not to use it or mention alternatives. It provides clear context for when this tool is appropriate, but lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.