Server Quality Checklist

Tool Scores

• extract_failure_signalA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the operation reads a log or artifact and uses 'real parsers (JUnit/Surefire XML, ESLint stylish, tsc, Prettier, Maven console), not an LLM guessing from raw text.' This adds valuable behavioral context beyond a simple 'extract' verb, implying deterministic, read-only behavior. It does not discuss edge cases or errors, but the read-only nature is clear through 'reads' and 'extract.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences that are information-dense: the first defines the transformation and output, the second provides usage guidance. There is no filler or repetition of schema field names. It is well-structured and front-loaded with the primary purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema and no annotations, so the description must cover return semantics. It does list the output fields (test name, class, message, stack trace, signature), which is helpful. However, it does not explain behaviors when no failure signal is found, how errors are handled, or whether the output is always a list. It also doesn't address potential edge cases like missing artifacts or malformed logs, leaving some ambiguity for a parser tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 80% (4 of 5 parameters have descriptions). The description adds meaning for artifactName by explaining it selects structured XML instead of the job log. However, runId lacks any description in either the schema or the description, and the description does not compensate for that gap. Overall, it meets the baseline but doesn't significantly elevate parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Turns a noisy raw CI log or a downloaded JUnit/Surefire XML artifact into structured FailureSignal entries') with a defined resource (the log or artifact) and output (test name, class, message, stack trace, signature). It distinguishes from siblings by focusing on extraction/parsing rather than fetching, searching, recording, or triaging.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage guidance: 'Prefer passing artifactName when fetch_pipeline_run listed one matching /junit|surefire/i for this run; otherwise this reads the job's own console log.' This provides a clear conditional for when to use the artifact vs the log. It does not explicitly name alternative sibling tools for the same task, but the conditional context is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_pipeline_runA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden of behavioral disclosure. It discloses key behaviors: the resolution logic (runId vs. latest failed run), the output (jobs and artifacts), and a significant restriction (allowedRepos config requirement, referencing guardrails.ts). It does not mention whether the operation is read-only, but 'lists' implies no side effects. The description adds context beyond a bare schema, covering important operational constraints, though it could go further (e.g., error handling or rate limits).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the main purpose in the first sentence and the key prerequisite in the second. Every phrase earns its place—no fluff or redundancy. It is appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential aspects for an agent to use this tool: what it does (fetch jobs/artifacts), how it selects the run (runId or latest failed on branch), and a critical prerequisite (allowedRepos config). Since there is no output schema, the description directly states the return content ('jobs and artifacts'), which is sufficient. It lacks some detail like error cases or return format specifics, but given the moderate complexity and clear title, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage, describing all four parameters (owner, repo, runId, branch) with examples and defaults. The tool description itself adds no additional parameter-specific information; it only mentions runId and branch conceptually, but the schema already explains them. Per the rubric, this is a baseline 3 since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Resolves a GitHub Actions run (by runId, or the most recent failed run on a branch) and lists its jobs and artifacts.' This is specific with a verb ('Resolves', 'lists'), a resource (GitHub Actions run), and outputs (jobs/artifacts). It also distinguishes itself from sibling tools (e.g., extract_failure_signal, find_similar_past_failures) which focus on analyzing or recording triage, not fetching raw run data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear functional context: it fetches a run's jobs and artifacts, with an explicit resolution mode ('by runId, or the most recent failed run on a branch'). It also mentions a critical prerequisite: 'Only works for repos listed in allowedRepos in ~/.ci-triage-mcp/config.json -- add a repo there yourself first.' While it does not explicitly name alternative tools or say 'use this when...', the purpose is clear enough for an agent to infer when to use it, and the prerequisite offers important usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• publish_triageA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully bears the transparency burden. It discloses that publishing 'may make the triage readable by others,' warns that the tool will currently fail with a 'not configured' error, and instructs to ask the user first. This reveals side effects, failure behavior, and an important usage constraint beyond what the schema or annotations could convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately front-loaded with the primary action, then prerequisites, current status, and a final warning. It is somewhat verbose, but each sentence contributes necessary operational context. A tighter version could combine the configuration prerequisites, but the structure is logical and not wasteful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 params, no output schema) and lack of annotations, the description covers the key operational context: what it does, what's needed to run, current deployment reality, and the need for user consent. It mentions the return value (shareable URL) but does not detail response structures or additional error cases beyond configuration. This is near-complete for the tool's purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers only 29% of parameters (only repo and owner have descriptions), so the description must compensate. It mentions 'explanation and signals,' which loosely maps to summary/category and the signals array, but it does not elaborate on individual parameter meanings like runId or suggestedFix. This partial coverage merits a 3, as it adds some context but leaves many parameters to inference.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Sends a triage's explanation and signals to a deployed ci-triage-dashboard instance, returning a shareable URL.' This uses a specific verb ('sends') and resource ('dashboard instance'), and differentiates it from siblings like record_triage_note by focusing on external publishing rather than local recording.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context: prerequisites (dashboardUrl and CI_TRIAGE_DASHBOARD_TOKEN), current deployment status, and a clear instruction to 'Ask the user before calling this once it is configured.' It does not name alternatives, but the warning about failure and permission-seeking make when-to-use clear. Slightly short of a 5 because it doesn't explicitly contrast with sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• record_triage_noteA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It reveals the tool performs a local file write only, is safe to call without asking the user, and appends/increments occurrence count. It does not cover edge cases like failure handling or file locking, but provides strong transparency for a simple local operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: first states the action and effect, second clarifies safety and contrast with publish_triage, third gives sequential context. Every sentence is information-dense with no filler. Front-loaded with the primary purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple local-write tool with no output schema, the description covers purpose, safety, usage context, and workflow integration with sibling tools. The main gap is insufficient parameter semantics for two of the five fields, but overall the tool is adequately contextualized within its triage pipeline.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is low (40%), and the description does not compensate. It references 'failure signature' and 'explanation' conceptually but does not add meaning for the signature or suggestedFix parameters, nor clarify the relationship between repo/signature/category beyond what the schema already states. The description adds little value beyond the schema's sparse field descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('appends') and resource ('explanation for a failure signature to the local history store'), and clearly states it also bumps occurrence count. It distinguishes itself from the sibling publish_triage by explicitly contrasting local write vs. shared dashboard.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance ('Do this after explaining a failure') and names the alternative (publish_triage) with a clear differentiator (local vs. shared dashboard). It also ties the action to the purpose of enabling future find_similar_past_failures calls, making the usage context unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• triage_pipeline_failureA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the entire pipeline chain, including downloading artifacts/logs and running parsers/search, and mentions the dashboardConfigured flag behavior. It doesn't mention potential performance costs of downloading artifacts or any side effects, but overall it's quite transparent about how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two dense sentences that front-load the purpose and chain, then add a usage alternative and a critical publishing caution. It is efficient but slightly long; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex chained tool with no output schema or annotations, the description covers the pipeline, what it returns at a high level, and a key behavioral caveat. It lacks a detailed return structure, but for the purpose of selecting and invoking the tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 75% (branch lacks a description), so baseline is 3. The description does not add parameter-level details beyond what the schema already provides, but the composite nature implies the parameters align with fetch_pipeline_run. No extra meaning is added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states this is a convenience tool that chains fetch_pipeline_run, artifact/log downloads, extract_failure_signal's parsers, and find_similar_past_failures, returning everything needed to write an explanation. It explicitly distinguishes itself from granular siblings by advising to use those when targeting a specific job.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use (want a full triage in one call) and when not to (when targeting one specific job, use granular tools). Also provides a post-call guideline about asking the user before publishing if dashboardConfigured is true.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_similar_past_failuresA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It explicitly states the tool is read-only, 'Doesn't modify history', and identifies the local store path. It doesn't cover edge cases like missing history file, but the key behavioral guarantees are clearly disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with no fluff. The first sentence front-loads the action and target store; the second adds usage context and an explicit pointer to the write alternative. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter lookup tool with no output schema, the description covers purpose, usage, storage location, safety, and alternatives. It doesn't describe return format or edge cases, but the core context is sufficient for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for both parameters, so the baseline is 3. The description adds a small connection between signatures and extract_failure_signal's FailureSignal entries, but mostly repeats what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Look up') and a clear resource: a local history store of failure signatures for the repo. It clearly distinguishes itself from siblings by contrasting with record_triage_note, making the tool's unique role obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use it: to say 'this has happened N times before' instead of treating failures as novel. It also points to record_triage_note for persisting explanations, providing a clear alternative and exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.