Server Quality Checklist

Tool Scores

• get_pr_checksA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It states it returns an aggregated check rollup, but lacks details on side effects, permissions, or pagination. Adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences efficiently state purpose and input format. Could be slightly more concise by omitting the second sentence since schema is clear, but still effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, so description should explain return structure. 'Returns aggregated check rollup' is vague; does not specify fields or format. Context with siblings is missing. Adequate but incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema already has full descriptions for both parameters. Description repeats the input format without adding new meaning. Baseline 3 for 100% coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists CI checks for a GitHub PR with conclusion and URL, using specific verb and resource. It distinguishes from siblings like get_failed_jobs and get_job_logs_tail which are more specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool vs alternatives. Sibling tool names are given but no context for choosing this over get_failed_jobs or summarize_pr_failure.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_failed_jobsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It states that the tool returns jobs with 'conclusion=failure' and lists the fields returned (name, databaseId, url). However, it does not mention whether it's read-only, auth requirements, rate limits, or behavior when no failed jobs exist. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The purpose is stated first, followed by input format and output summary. Every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 parameters, no output schema), the description adequately covers what the tool does, inputs, and outputs. It lacks error handling or pagination details, but for a list-failed-jobs tool, this is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds value beyond the input schema by showing the expected format for 'repo' ('owner/repo') and clarifying 'runId' as a number from the check URL. Schema coverage is only 50% (runId has a description, repo does not), so the description compensates partially.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the specific action: 'List only failed jobs for a workflow run.' It clearly identifies the resource (failed jobs) and distinguishes it from sibling tools like 'get_job_logs_tail' or 'summarize_pr_failure' that have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides the expected input format but does not explicitly guide the agent on when to use this tool versus alternatives. No mention of prerequisites, when-not-to-use, or comparisons with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_logs_tailA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the key behavioral trait: 'cleaned of ANSI + timestamp prefixes', which adds value beyond the schema. It does not mention side effects (none expected) or auth, but the cleaning detail is helpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence plus inline input format example. It is front-loaded with the purpose, then provides input format concisely. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 3-parameter read tool with no output schema, the description covers purpose, input format, and return value behavior (last N lines cleaned). It could mention error cases or that it only works for failed jobs (already implied), but overall it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67% (2 of 3 params have descriptions). The description repeats the schema structure without adding new constraints or examples. The default for lines is already in the schema. No extra semantic value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Fetch tail of a failed job log' uses a specific verb and resource, clearly distinguishing it from siblings like get_failed_jobs (list jobs) or grep_job_logs (search logs). The input format and cleaned output are also stated, leaving no ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after get_failed_jobs by specifying 'failed job log' and referencing jobId from get_failed_jobs, but does not explicitly compare with siblings or provide when-not guidance. Usage context is implied but not fully explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_pr_failureA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, but description discloses the tool's actions (read-only operations on PR checks and logs) and its output (summary with root cause). Could mention permissions or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single, clear sentence with no unnecessary words. Front-loaded with 'End-to-end triage'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately describes return value as summary with root cause, but lacks details on output structure. Given only 2 simple params, completeness is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and description only mentions parameter names (repo, prNumber) without explaining their format or constraints (e.g., repo name format). Adds minimal value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it performs end-to-end triage by findining failed checks, tailing logs, extracting errors, and returning a summary with root cause. It distinguishes from sibling tools that perform individual steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies use case for complete triage vs individual sibling tools, but does not explicitly state when to use alternatives or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• grep_job_logsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses key behaviors: regex matching (case-insensitive), returns matching lines with 2 lines of context before/after, and default maxMatches of 50. However, it does not mention error handling, permissions, or whether the operation is read-only, but the core behavior is well-covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences front-loading the main action, then output format, usage guidance, and input format. Every sentence is necessary and there is no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (regex search with context) and lack of output schema, the description is largely complete. It covers purpose, usage, input, and output format. It could add error handling or performance notes, but for the intended use case it suffices.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50% (pattern and maxMatches have descriptions, repo and jobId do not). The description compensates by listing all parameters in a compact format: 'Input: {repo, jobId, pattern: string, maxMatches?: number=50}', clarifying types and providing default value for maxMatches.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Grep a job log for lines matching a regex (case-insensitive)'. It specifies a specific verb (grep) and resource (job log), and distinguishes itself from sibling tools like get_job_logs_tail by focusing on pattern matching with context lines.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when to use: 'Use when failure is mid-log (E2E tests, build errors) rather than at the tail.' This implies when not to use (for tail failures) and gives context signals, differentiating from the sibling tool get_job_logs_tail.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.