Server Quality Checklist

Tool Scores

• review_open_ticketsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish read-only/open-world safety, so the description adds useful behavior by disclosing the sort order and compact one-line output. However, 'every open ticket' is not reconciled with the limit parameter default of 50, and no pagination behavior is described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences with no filler; the main action is front-loaded, and both following sentences add behavioral or routing value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, alternative routing, and annotation-protected safety, and an output schema exists. It is not complete for parameter usage, especially limit and filters, and the 'every' vs. default-limit inconsistency leaves a real gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description does not explain any of the four optional parameters (label, limit, project, tracker) or their filtering semantics. The agent must infer meaning entirely from parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a concrete action ('Review every open ticket') and a distinctive output ordering ('least-aligned first'), which is enough to separate it from sibling review_ticket. The phrase 'One line per ticket' further specifies the granularity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit planning context: it is 'the answer to what needs tidying before we can estimate any of this.' It names the alternative tool, review_ticket, for when detail is needed after the review.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• house_styleA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark it readOnlyHint/openWorldHint/destructiveHint false; the description adds that it reads a cached profile and, importantly, will explicitly say when no profile exists instead of returning misleading empty data. That is meaningful behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short, front-loaded sentences: the first states the resource and content, the second adds honesty about the unlearned state. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only tool with an output schema and safety annotations, the description covers the main purpose and the no-profile edge case. It would be stronger if it explained the project/tracker parameters, but they are optional and not blocking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are two optional parameters (project, tracker) with 0% schema description coverage, and the description does not mention them or clarify whether they filter the cached profile. It therefore does not compensate for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a concrete object ('What this team's tickets look like: template, length, labels, habits') and clarifies the mechanism with 'Reads the cached profile.' This clearly differentiates it from sibling learn_conventions, which implies the profile must be learned elsewhere.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The tool is positioned as a cached read, and the phrase 'if none has been learned yet' implies the agent should call it when house style is needed after learning conventions. However, it never names alternatives like learn_conventions or states when not to use it, so guidance is mostly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• review_ticketA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish readOnlyHint, openWorldHint, and destructiveHint, so the safety profile is covered. The description adds genuinely useful non-obvious behavior: that every finding cites a count over the exemplar sample and that this numerical grounding is the difference between this advice and generic advice. This tells the agent something about output provenance that annotations cannot convey. No contradiction with annotations; 'measure' is consistent with readOnlyHint.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with zero filler. The purpose sentence is front-loaded, and the second sentence earns its place by adding a critical presentation instruction (repeat the counts to the user). No redundant restatement of the tool name or schema fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-ticket comparison tool with an output schema present, the description is largely sufficient: it explains the measurement target, the grounding mechanism (counts over exemplar sample), and how to relay results. The main omission is the role of the two optional parameters, project and tracker, which are used in nearly every call context and deserve a sentence. Output schema covers return values, so that burden is already lifted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it only weakly does. The phrase 'one ticket' clarifies that 'ticket' is the object being measured, but 'project' and 'tracker' are entirely unexplained — the agent cannot tell whether they filter the exemplar sample, scope the house style, or constrain the comparison. With two of three parameters undocumented in both schema and description, this is a real gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence states a specific verb ('Measure'), a clear resource ('one ticket'), and the comparative frame ('against the learned house style'). It distinguishes itself from siblings by scoping to a single ticket, which contrasts with review_open_tickets (bulk) and review_draft (drafts, likely pre-submission). The scope word 'one' plus 'house style' anchors exactly what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied through the comparative frame (measure a ticket against house style), but no explicit when-to-use or when-not-to-use guidance is given, and no sibling alternatives are named. It doesn't tell the agent when to prefer this over review_draft or house_style. The only concrete usage instruction is 'Repeat those numbers to the user,' which is a presentation directive rather than a selection rule.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• learn_conventionsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, and the description adds genuine value beyond them: it discloses that the operation is slow (a few hundred API calls) and that it caches the result. This latency and persistence context is not conveyed by the annotations and helps an agent budget for cost.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three front-loaded paragraphs where each earns its place: purpose first, then cost/frequency, then the two parameter modes. Slightly long but well organized with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 5 parameters, no schema descriptions, and two distinct operational modes, the description explains the modes well but leaves keep, project, and tracker unexplained. An output schema exists so return values need no coverage, but the unexplained parameters leave a real gap for a tool this complex.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description carries the burden and does explain sample (the cap on tickets read) and from_tickets (take examples as given vs mine the tracker). But keep, project, and tracker are left entirely unexplained, so the description compensates for only 2 of the 5 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb+resource ('Learn how this team writes tickets, and cache the result') that clearly distinguishes it from sibling tools that review slips (review_draft, review_ticket) or provide templates (ticket_template, template_gaps). The learning/caching purpose is unmistakable and differentiates it from the surrounding review-and-draft sibling set.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Offers strong operational guidance: 'Call it once per project, not once per question' plus when to use from_tickets vs mining the tracker. However, it never explicitly compares against alternatives or states when-not-to-use — the differentiation from siblings is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ticket_templateA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety profile is covered. The description adds behavioral context by stating the tool 'returns the sections' and that 'the server has no opinion about the content,' clarifying that it only provides a structure and does not validate content. This adds value beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and front-loads the primary purpose. It is concise, but the wording is slightly informal and could be tightened (e.g., 'the skeleton to fill in' is a bit vague). Still, it avoids fluff and each sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple optional parameters, the presence of an output schema (which presumably describes the returned template), and annotations covering read-only behavior, the description is sufficient for an agent to know when to call it and what to expect. The only minor gap is the lack of parameter explanation, but that is mitigated by their optionality and defaults.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description makes no mention of the 'project' and 'tracker' parameters. While both are optional and have defaults, an agent cannot infer whether they filter the template or influence the returned sections. The description does not compensate for the lack of schema descriptions, leaving parameter semantics unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: it returns an empty ticket skeleton with the sections and lengths the team uses, to be filled in before drafting. It distinguishes itself from content generation ('write the ticket yourself') and from other workflow tools by specifying it's for the structure, not the content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use it 'before drafting, not after' and instructs the user to write the ticket themselves afterward, establishing a clear workflow position. It does not explicitly mention sibling alternatives or when not to use it, but the context is sufficient for an agent to understand this is an early-stage template retrieval step.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• review_draftA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the read-only nature is covered. The description adds the equivalence to review_ticket and the rationale for the pre-creation check, which is valuable behavioral context. It does not contradict annotations. The extra instruction to fix rather than report is an agent directive, not a tool trait, so it doesn't inflate the score further.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short paragraphs, front-loaded with the core instruction. The first sentence is direct and actionable. The subsequent paragraphs justify the tool's existence, which is relevant for usage but slightly verbose. Every sentence earns its place, though the rationale could be tightened. Overall well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, so return format is covered. The description explains the tool's purpose, when to call it, and its relationship to review_ticket. It does not mention edge cases or error handling, but for a read-only check tool this is adequate. The key context—why this tool exists and what it guarantees—is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameter semantics. It does not. The parameter names (title, description, labels, project, tracker) are self-explanatory in context, but no format, constraints, or relationship is described. For a tool that validates a draft, the description gives no insight into what fields matter or how they are validated, leaving the agent to guess.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Check a ticket you have written but not created yet.' It explicitly ties the tool to review_ticket, stating it is the same measurement, which distinguishes it from sibling tools like review_ticket and review_open_tickets. No ambiguity remains about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Call this on your own draft before showing it to the user.' It explains why (point of no return, notifications, edit history) and implies when not to use it (after creation, use review_ticket). This is unambiguous routing to the correct workflow step.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• template_gapsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false, so the safety profile is known. The description adds behavioral detail beyond that: it reads `.github/ISSUE_TEMPLATE` or `.gitlab/issue_templates` from the checkout, performs a bidirectional gap analysis, and explains what each type of gap signifies. This transparency helps the agent anticipate the tool's findings without contradicting the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by the underlying mechanism and the two types of findings. It uses a clean bullet-like presentation for the gap directions. While it is slightly verbose, each sentence contributes meaningful information, and nothing is redundant or wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity: it reads template files and ticket data, performs comparative analysis, and has an output schema to handle results. The description covers the input behavior, the analysis logic, and the significance of the two gap directions, which is sufficient for an agent to understand what the tool does and how to interpret its output. The only notable omission is explicit parameter meaning, but given the output schema exists and the overall logic is well explained, this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has three optional parameters (repo, project, tracker) with 0% schema description coverage, so the description must compensate by explaining what these parameters mean. However, the description never mentions them explicitly; it only references 'the checkout,' leaving it unclear how the parameters select which repository or project to examine. The agent would have to infer the parameter semantics from the tool name and context, which is insufficient given the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: comparing the issue template a repository declares with the tickets it actually receives. It uses specific verbs ('Compare', 'Reads', 'lines up') and explicitly differentiates itself from siblings by noting it is the only tool that reads what the project said it wanted, rather than only what it does. This leaves no ambiguity about the tool's function or how it differs from nearby alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly gives usage context: 'Use this when asked how to improve a board rather than one ticket.' It also distinguishes the tool from siblings by explaining it reads declared intent versus observed behavior. This gives clear when-to-use guidance and effectively excludes the wrong scenario (improving a single ticket), making it easy for an agent to choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ticket_contextA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false. The description adds meaningful behavioral detail beyond these annotations: it states the output is 'evidence, not prose', highlights that the file list is 'a search result, not an understanding of the code', and even notes it may include irrelevant matches ('things that merely share a word'). This transparently sets expectations about the tool's limitations and nature, which is exactly what the dimension asks for.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than average but well-structured with a clear instruction up front, followed by bullet points that add value. Every sentence serves a purpose: defining scope, explaining the return type, listing evidence types, and giving a caution. It is front-loaded with the core directive and does not waste words, so the length is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that searches across tickets, MRs, and checkout files, the description covers what the tool returns, the caveats, and the recommended usage flow. An output schema exists, so the return structure is likely defined there; the description doesn't need to repeat it. The only missing element is a concrete note on parameter syntax or edge cases, but given the output schema and the detailed behavioral notes, the description is sufficiently complete for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides no descriptions for its parameters and the schema_description_coverage is 0%. The description mentions 'subject' implicitly in the purpose line ('a subject') but does not explain how to use `repo`, `project`, or `tracker`. With zero schema coverage, the description needed to compensate by clarifying these parameters, but it does not. The agent is left to guess what each parameter means, making this a notable gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear, specific directive: 'Gather what is known about a subject before you write the ticket for it.' It explicitly lists what it returns (evidence, past tickets, files changed, file mentions) and contrasts it with prose. It also names the sibling `ticket_template` and positions itself as complementary, making the tool's distinct role immediately apparent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit, actionable workflow: 'Call this together with `ticket_template`, then write the ticket yourself.' It further instructs the agent to 'Read the files it points at before drafting' and warns against treating the list as understanding. This leaves no ambiguity about when and how to invoke the tool relative to the ticket-writing pipeline.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.