Server Quality Checklist

Tool Scores

• changes_by_entityA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The readOnlyHint annotation already signals safety, and the description adds useful behavioral detail beyond it: exact match, case-insensitivity, and the reverse-index nature of the lookup. It does not mention output ordering or pagination, but for a straightforward read-only lookup this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the core concept, and every sentence contributes meaning: what it does, matching behavior, and when to use it. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with two parameters and no output schema, the description gives enough context: purpose, identifier types, matching behavior, and usage trigger. It could mention pagination or result contents, but it is not a critical gap for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both 'kind' and 'name' well. The description reinforces exact-identifier semantics and case-insensitivity but adds little beyond what the parameter descriptions already provide.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb/resource framing: 'Reverse index: every change touching one exact identifier' and enumerates supported identifiers (CVE, CRD, feature gate, etc.). This clearly distinguishes it from siblings like list_changes or get_release, which are broader or differently scoped.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Call this when you have a specific identifier... and want to know what changed around it,' giving a clear use case. It does not name alternatives or exclusion cases, but the context is sufficiently clear versus the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_matterA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the description only needs to add behavioral context. It does so by specifying ordering (oldest first), the effect of include_all (routine bot-bump records), and a concrete rationale for returning all releases. It does not contradict annotations, and provides useful context beyond the safe-read signal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a clear purpose and each sentence adds value. However, the containerd roll-up rationale is fairly long for a two-parameter read tool, making the description more verbose than necessary, though the detail is meaningful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool shape and absent output schema, the description sufficiently covers what is returned (every release, oldest first), when to use it, and how the include_all flag changes the result. No significant gaps remain for an AI agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description largely restates what the schema already says about matter_key (verbatim, case-sensitive, contains '/' and ':') and include_all (routine records, off by default), adding no new parameter-level meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states the operation: list every release in which a matter appeared, oldest first. It also distinguishes this from sibling tools like list_releases or get_release by focusing on a single matter across releases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly identifies the questions this tool answers: 'which version fixes this for MY branch' and 'have I already handled this'. It also explains why returning every occurrence matters, which guides when to use this tool, though it does not name alternative tools explicitly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_releasesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses key behavioral traits such as the newest-first ordering, the meaning of changes_total=0 as 'auditable silence' (routine read), and the limitation to N newest releases. It also mentions the light summary format. While annotations already indicate read-only, it adds meaningful context beyond that, though it could also note potential pagination or lack thereof.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the core purpose, and every sentence adds value: it defines the output fields, contrasts with list_changes, clarifies changes_total semantics, and provides a pointer to get_release for full details. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, the description is comprehensive: it specifies the output fields, ordering, the meaning of a zero changes_total, and a pointer to get_release for full details. Annotation provides additional safety context (read-only), and the schema covers parameters fully. No gaps are evident.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents both parameters (project and limit) with examples and defaults, achieving 100% coverage. The description does not add additional parameter semantics beyond what is in the schema, so the baseline of 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as returning the newest N reviewed releases with a specific set of summary fields, and it is explicitly contrasted with list_changes, which walks oldest-first. It states the verb 'list' and the resource 'releases', making its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It specifies when to use this tool ('recent releases of X' / 'what changed in X lately') and explicitly differentiates it from the sibling tool list_changes, which is oldest-first by seq, providing a clear alternative and exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_projectsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, and the description adds behavioral context: it explains how cluster_core projects are included, how visibility hints work, and that unreadable components are reported as unchecked, never guessed. This goes beyond the annotation by detailing edge-case behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but information-rich, covering all key aspects in a single paragraph. It is front-loaded with the core purpose and then details edge cases. Slightly long but every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description fully compensates by explaining the return fields, the meaning of cluster_core, visibility hints, and the recommended usage pattern. It is complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema coverage is 100% (no params). The description explains the output fields in detail, which is valuable since there is no output schema. It compensates for the lack of structured output documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all projects tracked by ratatosk, specifying the fields returned (slug, name, tier, category, analyzed_releases, image_aliases, cluster_core) and the purpose of each. It distinguishes itself from siblings by emphasizing it is the canonical source for slugs and should be called first when unsure.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call this tool FIRST when unsure of a slug, and warns against guessing (a wrong slug shows up as tracked:false in check_stack). This provides clear when-to-use guidance and differentiates from sibling tools like check_stack.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_stackA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes well beyond the readOnlyHint annotation: it states that versions are compared inside the server process, that only project slugs are sent upstream, that /v1/upgrade is never called, and that running versions are not logged. It also discloses branch-aware and line-aware comparison behavior, the tracked:false no-coverage meaning, and the 50-item cap with relevant_changes_omitted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but information-dense; every sentence carries behavior an agent must know, and the core purpose is front-loaded. It is not formatted with bullets or section breaks, which makes it harder to scan, but there is little to no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description carries the full burden of explaining return values, and it does so thoroughly: summary fields, bucket split, action_required vs check_config, matter_key collapsing, note flags, tracked semantics, and follow-up drill-down tools. It also anticipates common pitfalls like wrong line prefixes and target versions at or below the running version.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100%, the description adds critical parameter semantics: version must be passed exactly as published with its line prefix, target_version only applies when strictly above the running version, version_source is echoed as an audit trail, and severity_min filters by severity. It also explains the detail:'full' behavior and the omission cap, which the schema alone does not convey.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Check the user's running component versions against known changes.' It then states exactly what is returned — per-component changes from releases newer than the running version — which clearly distinguishes this stack-level check from sibling drill-down tools like get_release and changes_by_entity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context: use it when you have running component versions and need the upgrade path, and it explains how to shape results with detail, target_version, and severity_min. It explicitly points to alternatives at the end ('Drill down with get_release or changes_by_entity'), though it does not enumerate when list_changes or list_projects would be the better choice.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_releaseA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as read-only, and the description adds valuable behavioral details: changes=[] represents auditable silence rather than a gap, each change includes family/actionability/bucket/applies_if/advisories/verbatim quote, and summaries are derived from the same set. It also discloses error behavior for an invalid version, exceeding what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Though dense, every sentence earns its place: core purpose, edge-case meaning of changes=[], field enumeration, version semantics, and include_raw. There is no filler or redundancy, and the most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description fully documents return values: envelope fields, change fields, summary fields, notes_total, and raw_notes. It also explains error handling and parameter behavior, making the tool self-sufficient for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description enriches every parameter: project slug example, version can be omitted for latest, 'v' prefix tolerance, wrong-tag error behavior, and include_raw returning raw_notes. This goes well beyond the schema's per-parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'One reviewed release' and immediately specifies the return shape: an envelope (summary, source URL, release URL) plus all changes. This clearly identifies the tool as fetching a single release's full detail, distinguishing it from siblings like list_releases, with a specific verb and resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides concrete usage guidance: omit version for the latest reviewed release, version accepts with or without a leading 'v', and a wrong tag returns an error listing recent reviewed tags for retry. It does not explicitly name alternatives (e.g., 'use list_releases to see all'), so it falls short of full when/when-not coverage, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_changesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already mark readOnlyHint=true, and the description adds substantial behavior beyond that: ascending seq order, oldest-first semantics, next_since-based pagination until null, default exclusion of routine records, and the roles of applies_if and matter_key. This gives an agent a clear model of how the feed behaves, not just what it does.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence carries information: purpose, ordering, pagination, alternatives, filter semantics, and key response fields. It is front-loaded with the core sync behavior. It could be slightly more scannable, but it is not padded or redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-parameter tool with no output schema and no enums, the description covers the essential context: how to page, when to stop, which filters to use, what the key response fields mean, and how to route to sibling tools. An agent has enough to invoke it correctly and interpret the response meaningfully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so a baseline of 3 applies. The description adds meaningful value by explaining bucket as 'how to act now' with per-bucket guidance, family as 'what kind of thing it is', and limit as something to raise for syncs but not one-off questions. This goes beyond the schema's field labels.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise statement: 'Incremental SYNC feed of release changes for CNCF/cloud-native projects.' It names the verb, resource, and ordering semantics, and explicitly distinguishes itself from list_releases/get_release for 'latest release' queries. This makes the tool's purpose unmistakable and clearly separate from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says when to use the tool ('Built for keeping a local copy up to date') and when not to ('For what is the latest release... use list_releases or get_release'). It also explains pagination with since and when to stop, and clarifies filters and the default exclusion of routine records.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.