Server Quality Checklist

Tool Scores

• facts_by_entityA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses case-insensitivity, a useful behavioral trait. However, it does not mention read-only nature, result limits, or data freshness, which would enhance transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no wasted words. Front-loaded with core purpose, followed by usage context. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple query tool with 2 parameters and no output schema, the description covers purpose, usage, and parameter hints. It does not describe output format, but given no output schema, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3. Description adds value by explaining 'name' is an exact identifier and clarifying 'kind' is optional with example values, going beyond the schema's minimal description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool is a reverse index for facts touching an exact identifier, listing specific types like CVE, CRD, etc. It distinguishes itself from sibling tools like list_facts by emphasizing exact identifier lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance: 'Call this when you have a specific identifier... and want to know what changed around it.' Implicitly tells when not to use (if no exact identifier). Sibling tools listed in context but not explicitly contrasted.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_stackA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries full burden. It is exceptionally transparent: explains that versions are compared in-process, only project slugs are sent upstream, never calls the /v1/upgrade endpoint, does not log on hosted endpoint, and cannot verify versions. It also discloses edge-case behavior like tracked:false, ignored target_version, and applies_if precondition logic.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph lacking any formatting (bullets, sections). While every sentence carries information, the wall of text is difficult to parse and repeats some schema details (e.g., target_version behavior). It would benefit from structured breaks to improve scannability, especially for an AI agent parsing the text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multi-component checks, modes, filters, no output schema), the description is remarkably complete. It explains return behavior, default briefing, how facts are grouped, severity filtering, and the privacy model. There are no significant gaps that would leave an agent uncertain about what the tool does or returns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (baseline 3), but the description adds meaningful semantics beyond the schema. For example, it elaborates the briefing logic (action_required vs check_config, applies_if resolution, group-max severity merging) and clarifies the audit purpose of version_source. It does not cover every parameter interaction, but it substantially enriches understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Check the user's running component versions against known facts.' It clearly distinguishes itself from siblings by explaining its unique role (version comparison, upgrade path) and explicitly references 'get_release' and 'facts_by_entity' for drill-down, making its niche clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides strong context on when to use the tool (checking a running stack) and how to tailor it via detail, target_version, and severity_min. It references alternatives at the end ('Drill down with get_release or facts_by_entity'). It does not explicitly state exclusions (e.g., 'use list_facts instead when...') but the guidance is sufficient for an agent to choose it appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_releaseA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description comprehensively discloses behavioral traits: envelope structure, facts listing, the auditable silence case (facts=[] with coverage=full_reviewed), automatic inclusion of raw_notes when coverage insufficient or zero facts, and error behavior on wrong tags. This exceeds typical transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is effectively concise, packing all key information into a single dense paragraph. It front-loads the core result and covers edge cases. Minor room for improvement: could be broken into bullet points for readability, but still efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description fully explains the return structure (envelope with coverage, assessment, source URL, plus facts) and special cases (auditable silence, automatic raw_notes). It covers all parameters and error handling, making the tool entirely understandable without external context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, but the description adds significant value: explaining version omission for latest, tolerance for leading 'v', and that wrong tags return recent tags. It also clarifies include_raw's purpose and automatic raw_notes inclusion.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns 'One reviewed release: envelope (coverage, assessment, source URL) plus all its facts.' This verb+resource structure precisely defines what the tool does and distinguishes it from siblings like list_releases which lists releases, or fact_by_entity which filters facts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on omitting version for latest release, version format flexibility ('with or without leading v'), error handling (wrong tag returns recent tags for retry), and when to set include_raw. It lacks explicit when-not-to-use or direct sibling comparisons, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_releasesA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It clearly states this is a read-like operation (listing releases), describes output fields (version, date, coverage, fact counts, severity), and even explains the meaning of 'facts_total=0 with coverage=full_reviewed' as routine auditable silence. The only minor omission is mention of authentication or rate limits, but for most contexts this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence serves a purpose: overview, usage guidance, ordering clarification, edge-case explanation, and drill-down suggestion. No filler. Well-structured and appropriately sized for the complexity of the tool (two parameters).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description covers return values (light summaries with key fields) and includes an important edge case (zero facts with full reviewed). It also references sibling tools (list_facts, get_release) to provide integration context. This is complete for an agent to understand when and how to use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by explaining the default (5) and max (20) for 'limit', and notes that 'project' is a slug like 'istio'. It also implies that the parameters control the scope of releases returned. This goes beyond the schema's bare descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Purpose is crystal clear: 'list_releases' lists recent releases with light summaries. The description specifies verb+resource ('list releases of one project') and distinguishes it from sibling 'list_facts' by noting the different ordering (newest first vs oldest-analyzed first). This is a specific, actionable description.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'THE tool for recent releases of X / what changed in X lately'. Contrasts with list_facts (oldest-first ordering) and points to get_release for drilling into full details. No ambiguity about which tool to choose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_projectsA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully explains the response contents, the semantics of cluster_core and image_aliases, and the visibility hint behavior (unreadable components reported as unchecked, never guessed). It also notes the response is small and takes no arguments.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense run-on sentence that packs each field explanation and usage guidance together without clear structural breaks. The key usage instruction ('call this FIRST') appears only at the end, limiting front-loading.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description enumerates all response fields and their meanings, including edge cases for cluster_core visibility. It also provides context for how this tool fits with siblings, making the description complete for a zero-parameter listing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the description explicitly states 'no arguments,' confirming the schema and adding clarity that no input is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool returns: every project with its slug, name, tier, category, and other attributes. It also distinguishes this tool from siblings by noting that slug is the canonical ID other tools accept and that this should be called first when unsure of a slug.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call this tool first when unsure of a slug, and references check_stack as the place where a wrong slug shows up as tracked:false. This provides clear when-to-use and alternative context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_factsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so richly. It discloses the ordering ('ordered by fact_id ascending — OLDEST analyzed first'), the pagination contract ('page through with since=<returned next_since> until next_since comes back null'), and the nuance that a single page is not the newest data. It also explains the meaning of advisory_group_key and group_severity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a tight set of four sentences, each earning its place: purpose, ordering/pagination, alternative, and advisory-key semantics. It is front-loaded with the core purpose and avoids filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema and annotations, the description covers the essential behavioral contract: pagination, ordering, alternative usage, filter options, and the meaning of advisory fields. This is sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaningful usage context beyond the schema: it clarifies that `since` is a cursor driven by the returned `next_since`, and it explains filter semantics like severity being group-maximum. This elevates the score to 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Incremental SYNC feed of release facts' for CNCF/cloud-native projects, and details the types of changes covered. It explicitly distinguishes itself from the get_release sibling for latest-release queries, satisfying the sibling-differentiation requirement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly states the intended use case: 'Built for keeping a local copy up to date.' It also provides an explicit alternative: 'For "what is the latest release of X" or "recent releases of X", use get_release (omit version for the newest) instead.' This is a clear when-to-use and when-not-to-use directive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.