Server Quality Checklist

Tool Scores

• cancel_runC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool cancels runs 'in progress or queued', which implies mutation but lacks details on permissions, side effects, error handling, or what happens post-cancellation. This is inadequate for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and appropriately sized, with every word earning its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity as a mutation operation, lack of annotations, and no output schema, the description is incomplete. It does not cover behavioral aspects like permissions, side effects, or response format, which are crucial for an agent to use this tool correctly in context with siblings.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all three parameters (owner, repo, run_id) with descriptions. The description does not add any meaning beyond what the schema provides, such as explaining how to obtain a run_id or format the owner/repo. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('cancel') and target ('a workflow run that is in progress or queued'), providing a specific verb+resource. However, it does not explicitly distinguish this tool from sibling tools like 'rerun_failed_jobs' or 'rerun_workflow', which might involve similar contexts but different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, such as 'rerun_failed_jobs' or 'rerun_workflow', nor does it mention prerequisites like needing a run to be in a cancellable state. It only states what the tool does, not when it should be used.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_runC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While 'Get details' implies a read operation, it doesn't specify whether this requires authentication, has rate limits, returns structured data, or what happens with invalid run IDs. This leaves significant behavioral gaps for a tool with 3 required parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise - a single sentence that states exactly what the tool does with zero wasted words. It's front-loaded with the core purpose and contains no unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 required parameters, no annotations, and no output schema, the description is insufficient. It doesn't explain what 'details' are returned, whether authentication is needed, error conditions, or how this differs from sibling tools. The agent would need to guess about important behavioral aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no parameter information beyond what's already in the schema (which has 100% coverage). It doesn't explain relationships between parameters or provide context about what constitutes a valid 'run_id'. Since schema coverage is complete, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get details') and resource ('specific workflow run'), making the purpose immediately understandable. However, it doesn't differentiate this tool from potential siblings like 'get_run_logs' or 'list_runs' that also retrieve run information, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'list_runs' (for multiple runs) and 'get_run_logs' (for logs rather than details), the agent receives no help in selecting the appropriate tool for different scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_artifactsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic action without behavioral details. It doesn't disclose whether this is a read-only operation, if it requires authentication, rate limits, pagination behavior, error conditions, or what the output format looks like (e.g., list of artifact names with metadata).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with zero wasted words. It's front-loaded with the core purpose and appropriately sized for a simple list operation, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations and no output schema, the description is incomplete. It doesn't explain what artifacts are (e.g., build outputs, logs), the return format, or behavioral aspects like pagination or error handling. For a tool with 3 required parameters and no structured output documentation, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all three parameters (owner, repo, run_id). The description adds no additional parameter semantics beyond implying these are needed to identify a workflow run, which is already clear from the schema. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and target resource ('artifacts produced by a workflow run'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'list_runs' or 'get_run_logs', which would require mentioning it specifically returns artifacts rather than run metadata or logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description doesn't mention prerequisites (e.g., needing a valid run_id), exclusions, or comparisons to siblings like 'get_run' (which might include artifact info) or 'list_runs' (which lists runs rather than artifacts).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_workflowsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states it lists files but doesn't cover key traits like whether it requires authentication, includes pagination, returns specific formats, or has rate limits, which are critical for a GitHub API tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with zero waste—it directly states the tool's function without unnecessary words. It's appropriately sized and front-loaded, making it efficient for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a tool with 2 parameters. It doesn't explain what the output looks like (e.g., list format, fields), authentication needs, or error handling, leaving gaps in understanding how to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('owner' and 'repo') fully. The description adds no additional meaning beyond what the schema provides, such as explaining how to format inputs or handle edge cases, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('workflow files in a GitHub repository'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'list_runs' or 'list_artifacts', which also list GitHub-related items, so it misses full sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing repository access, or compare it to siblings like 'list_runs' for listing workflow runs instead of files, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rerun_workflowC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action is a 'Re-run' but doesn't clarify if this requires specific permissions, whether it's idempotent, what happens to the original run, or any rate limits. This is inadequate for a mutation tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core action and resource, making it highly concise and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a workflow rerun tool with no annotations and no output schema, the description is insufficient. It doesn't explain behavioral aspects like side effects, return values, or error handling, leaving significant gaps for the agent to operate safely and effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, fully documenting the three parameters (owner, repo, run_id). The description adds no additional semantic context beyond what the schema provides, such as format examples or constraints, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Re-run') and the resource ('a complete workflow run'), which is specific and unambiguous. However, it doesn't distinguish this tool from its sibling 'rerun_failed_jobs', which handles partial reruns versus complete reruns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'rerun_failed_jobs' for partial failures or 'trigger_workflow' for new runs. It lacks context about prerequisites or exclusions, leaving the agent to infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_run_logsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that 'GitHub returns a redirect to a zip file,' which adds useful context about the output format (a redirect URL to a zip). However, it doesn't cover critical aspects like authentication needs, rate limits, error conditions, or whether this is a read-only operation. For a tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and front-loaded: the first sentence states the core purpose, and the second adds crucial behavioral context about the redirect. Every sentence earns its place with no wasted words, making it efficient for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no output schema, no annotations), the description is partially complete. It explains the purpose and output format but lacks usage guidelines, behavioral details like auth or errors, and doesn't leverage sibling context. It's adequate as a minimum viable description but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all three parameters (owner, repo, run_id) with clear descriptions. The description doesn't add any meaning beyond what the schema provides, such as explaining parameter relationships or constraints. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get logs URL for a workflow run.' It specifies the verb ('Get'), resource ('logs URL'), and target ('workflow run'), making the action explicit. However, it doesn't differentiate from siblings like 'get_run' or 'list_runs' beyond mentioning logs, which keeps it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, exclusions, or compare to sibling tools such as 'get_run' (which might include logs) or 'list_artifacts' (which could contain log files). This lack of context leaves the agent to infer usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trigger_workflowB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the requirement for 'workflow_dispatch enabled' but fails to describe key traits such as whether this is a write operation (implied by 'trigger'), potential side effects, authentication needs, rate limits, or what happens upon execution. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that directly state the purpose and a key requirement. It is front-loaded with the main action and avoids unnecessary elaboration. While efficient, it could be slightly improved by integrating the requirement more seamlessly, but overall it earns its place without waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of triggering a workflow (a write operation with potential side effects), no annotations, and no output schema, the description is incomplete. It lacks details on behavioral traits, error handling, return values, and how it differs from sibling tools. This makes it inadequate for safe and effective use by an AI agent in a real-world context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, meaning all parameters are documented in the schema itself. The description adds no additional parameter details beyond what the schema provides, such as examples or constraints. However, since the schema is comprehensive, a baseline score of 3 is appropriate as the description doesn't need to compensate for gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('trigger a workflow run') and mechanism ('via workflow_dispatch'), which is specific and actionable. It distinguishes from siblings like 'list_workflows' or 'get_run' by focusing on execution rather than querying. However, it doesn't explicitly differentiate from 'rerun_workflow' or 'rerun_failed_jobs', which are also execution-related siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some context by mentioning the prerequisite ('Requires a workflow with workflow_dispatch enabled'), which helps determine when it can be used. However, it offers no guidance on when to use this tool versus alternatives like 'rerun_workflow' or 'rerun_failed_jobs', nor does it specify exclusions or complementary tools, leaving usage decisions ambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_runsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions filtering capabilities but doesn't describe critical behaviors such as pagination handling (implied by 'per_page' in schema), rate limits, authentication requirements, or what the output format looks like. This leaves significant gaps for an agent to use the tool effectively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes key optional features. There is no wasted text, and it's appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't address behavioral aspects like pagination, error handling, or return format, which are crucial for a list operation with multiple parameters. The schema covers inputs well, but the description fails to compensate for missing context on how the tool behaves.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds minimal value by mentioning optional filtering by workflow or status, which aligns with the schema but doesn't provide additional context or examples beyond what's in the structured data.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'workflow runs for a repository', which is specific and actionable. However, it doesn't explicitly distinguish this tool from sibling tools like 'get_run' or 'list_workflows', which might cause confusion about when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by mentioning optional filtering by workflow or status, but it doesn't provide explicit guidance on when to use this tool versus alternatives like 'get_run' for a single run or 'list_workflows' for workflows themselves. No exclusions or prerequisites are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• rerun_failed_jobsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the action ('Re-run') but lacks details on permissions required, rate limits, whether it's idempotent, or what happens to successful jobs. For a mutation tool with zero annotation coverage, this is a significant gap in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every part of the sentence earns its place by specifying the action and scope clearly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that this is a mutation tool (implied by 'Re-run') with no annotations, no output schema, and incomplete behavioral transparency, the description is insufficient. It does not cover what the tool returns, error conditions, or side effects, leaving the agent with critical gaps in understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (owner, repo, run_id). The description does not add any meaning beyond what the schema provides, such as explaining how to identify a 'workflow run' or the format of 'run_id'. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Re-run') and the target ('only the failed jobs from a workflow run'), distinguishing it from siblings like 'rerun_workflow' (which likely re-runs all jobs) and 'cancel_run' (which stops execution). It uses precise terminology that aligns with the tool's name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when there are failed jobs in a workflow run, but it does not explicitly state when to use this tool versus alternatives like 'rerun_workflow' or 'trigger_workflow'. No exclusions or prerequisites are mentioned, leaving some ambiguity for the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.