Server Quality Checklist

Tool Scores

• openshift_get_projectC
  2.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose any behavioral traits (e.g., read-only, authentication needs, error handling). The description carries the full burden but fails to address this.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with one sentence and no waste. However, it is too brief and lacks crucial details, making it minimally acceptable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and zero parameter coverage, the description is vastly incomplete. It fails to explain what 'details' means, any prerequisites, or return format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not mention any parameters. The description adds no meaning beyond the schema, leaving both parameters ('projectName' and 'userId') undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves details for one OpenShift project, with a specific verb and resource. It distinguishes from sibling tools like openshift_set_user_token, which are for different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. Siblings are different but context is implicit; no explicit usage context or prerequisites are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• openshift_set_user_tokenC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description mentions storage destinations (Vault, Postgres) but lacks critical behavioral details such as whether rotating a token invalidates the previous one, required permissions, or side effects. With no annotations, more burden falls on the description, which it fails to meet.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that efficiently states the core action and storage. However, it could be slightly improved by breaking into two sentences (action then storage) but remains adequate.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 7 parameters, no output schema, and no annotations, the description is severely incomplete. It lacks explanations for parameters, return value, error conditions, and side effects, making it insufficient for an agent to use effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds no meaning beyond the schema. The 7 parameters (including 'scopes', 'audience', 'expiresAt') are not explained at all, leaving the agent without guidance on their purpose or format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool sets or rotates a user-scoped OpenShift bearer token, which is specific. However, it does not explicitly contrast with sibling tools like openshift_get_user_token_metadata or openshift_deactivate_user_token, leaving some ambiguity about when to choose this over alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus siblings or any prerequisites. It does not mention scenarios, exclusions, or alternatives, leaving the agent to infer usage from context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• openshift_deactivate_user_tokenC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description indicates a destructive action (removing a secret) but does not disclose behavioral details such as whether the action is reversible, if authorization is checked, or if it triggers cascade effects. With no annotations, this level of transparency is insufficient for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, concise and to the point. However, it lacks structured breakdowns (e.g., parameter descriptions), which could improve usefulness. It earns a 4 for being efficient but not earning the top score due to missing structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two parameters (one required), no output schema, and no annotations, the description is incomplete. It fails to address parameter semantics, return values, or side effects. The agent would have insufficient context to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no information about the parameters 'userId' or 'authorizationKey'. With 0% schema description coverage, the agent has no guidance on what these parameters represent or how to use them. This is a critical gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Deactivate') and the resource ('user token'), and specifies the two locations (Vault index and direct secret). It distinguishes from siblings like 'openshift_set_user_token' and 'openshift_get_user_token_metadata' by focusing on deactivation. However, it could be more explicit that it pertains to OpenShift user tokens.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives (e.g., when to deactivate vs. query metadata). There are no prerequisites, exclusions, or contextual usage hints, leaving the agent to infer appropriateness.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• redshift_queryC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must carry the full burden. It does not disclose whether destructive SQL is allowed, error handling, return format, authentication requirements, or rate limits. The presence of an authorizationKey parameter implies auth but is not explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The single sentence is concise with no wasted words, but it lacks structure and is too brief to cover necessary details. It is not overly verbose, but efficiency does not compensate for missing information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no annotations, no output schema, and 0% param coverage, the description is severely incomplete. It fails to convey execution behavior, response, error handling, or security context needed for an agent to invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no explanation for any of the 6 parameters. The agent receives no help understanding what each parameter means beyond the schema's type constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (run a parameterized SQL statement) and the resource (against a named Redshift cluster). It is specific and immediately distinguishes from sibling OpenShift tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives, prerequisites, or when not to use it. The sibling tools are all OpenShift-related, but the description does not explicitly contrast them.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• openshift_get_user_token_metadataC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description bears full burden. It implies a read operation but does not disclose side effects, permissions, rate limits, or what 'current Vault token index state' entails. Basic behavior only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no wasted words. However, it could benefit from more structure (e.g., separating sources) without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple parameter set and lack of output schema, the description is minimal. It fails to explain return values, required permissions, or the relationship between Postgres and Vault state. An agent would need additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter userId is not explained beyond its name. With 0% schema description coverage, the description adds no meaning about expected format, source, or constraints. The name is somewhat self-explanatory but insufficient for precise use.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (get) and resource (user token metadata) with specific sources (Postgres and Vault). It distinguishes from sibling tools by its focus on metadata retrieval, but does not explicitly differentiate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. There is no mention of prerequisites, exclusions, or context about when retrieval is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.