Server Quality Checklist

Tool Scores

• get_repo_healthC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the full behavioral disclosure burden. It only says 'Get' and does not describe the return shape, data sources, required permissions, rate limits, or side effects. The read-only nature is implied but never stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one short sentence with no filler, and the core verb and resource are front-loaded. It is appropriately sized, though the brevity sacrifices useful differentiation from sibling tools.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter read tool the basics are present, but the lack of an output schema and the undefined term 'health information' leave a significant gap. An agent cannot reliably predict what the tool returns or when to choose it over get_health_score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes both parameters (owner and repo) with 100% coverage, so the parameter baseline is 3. The description adds no meaning beyond the schema's property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Get') and resource ('health information for a GitHub repository'). However, 'health information' is not defined, and the description does not differentiate this from the sibling get_health_score, so it is clear but lacks sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus get_health_score, check_ci_status, analyze_dependencies, or analyze_code_scanning. No usage context, exclusions, or alternative routing is provided; the agent must infer from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pingB
  3.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the behavioral burden. It clearly discloses the response behavior ('response with pong') and implies a side-effect-free operation by calling it 'simple'. No contradictions with annotations exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence and front-loads the essential behavior. It is concise, though the word 'simple' is somewhat redundant and the sentence structure is slightly awkward.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a trivial ping tool, the description adequately conveys the core behavior and response. However, it doesn't explain what to pass in `message`, and there is no output schema to confirm the exact return shape, leaving a small gap for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema describes the single `message` parameter, but the description is identical to the tool description and doesn't clarify what content the message should contain. Schema coverage is 100%, so baseline 3 is appropriate, but no additional meaning is added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description identifies the tool as a 'simple ping' and says it responds with 'pong', which gives a concrete result. However, it largely restates the tool name and doesn't explicitly differentiate it from sibling tools like check_ci_status or get_health_score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance about when to use this tool versus alternatives. The phrase 'simple ping' implies a liveness or connectivity check, but no explicit conditions, exclusions, or alternative tool mentions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_dependenciesB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description is the only source of behavioral information, yet it only says 'get' without addressing authentication, rate limits, pagination, or what the returned alert data looks like. The read-only nature is lightly implied but not substantiated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence with no filler. It communicates the core operation and resource efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool with only three parameters, the basic invocation context is present. However, the lack of an output schema, annotations, and any mention of response shape or severity filtering behavior leaves the description only minimally complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not add meaning beyond the input schema; repo and owner are already documented, and severity is left to its enum values. Since schema coverage is only 67% and there is no compensating parameter detail in the description, this is a gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource, 'Get Dependabot alerts for a GitHub repository,' which clearly identifies the operation. It also distinguishes the tool from the sibling analyze_code_scanning, since dependency alerts and code scanning alerts are different outputs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The use case is implied: an agent should call this when it needs Dependabot alerts for a repo. However, there is no explicit when-to-use or when-not-to-use guidance, and no mention of alternatives such as analyze_code_scanning or get_repo_health.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_health_scoreB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. 'Calculate' implies a read-only analysis operation, and the listed outputs (score, grade, breakdown, suggestions) clarify what the agent will receive. However, it does not disclose whether remote API calls are made, what inputs affect the score, or how failures are handled.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single tightly packed sentence with no filler. Every phrase adds value: the score range, the grading scale, the breakdown, and the improvement suggestions are all meaningful for an agent deciding to invoke the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description helpfully specifies the main return components and the numeric range. It is sufficient for simple two-parameter invocation, though it could be more complete by explaining how the score relates to repository metrics or by distinguishing it from get_repo_health.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for both parameters, so the schema already documents owner and repo. The description adds no additional meaning beyond stating the tool operates on a GitHub repository, so it stays at the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Calculate') and names the resource ('GitHub repository'), plus the key outputs: 0-100 score, A-F grade, category breakdown, and suggestions. However, it does not differentiate from the sibling tool get_repo_health, which appears to serve a very similar purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies it is used to compute a repository health score but gives no guidance on when to choose this tool over get_repo_health or other siblings like analyze_dependencies. No alternatives, exclusions, or context-dependent criteria are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_ci_statusA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. 'Get' clearly implies a read-only operation with no destructive side effects, but the description does not mention authentication needs, rate limits, pagination behavior, or whether this is specific to GitHub Actions. It is minimally transparent but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly scoped sentence with no filler or redundant wording. Every word contributes to identifying the purpose, and it is immediately front-loaded with the core action and subject.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 3-parameter read tool with no output schema, the description plus fully documented schema is sufficient for an agent to call it correctly. It does not describe return value fields or explicitly rule out alternative tools, but nothing essential to making the call is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all three parameters (owner, repo, limit) are already documented in the input schema. The description adds only the 'recent' and 'workflow runs' context, which does not meaningfully deepen parameter semantics beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Get'), a clear resource ('recent CI/CD workflow runs'), and the target system ('a GitHub repository'). This clearly differentiates the tool from siblings like get_repo_health, analyze_code_scanning, and get_health_score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool by naming CI/CD workflow runs as the subject, and sibling tool names suggest the alternatives. However, it does not explicitly state when NOT to use it or which sibling to prefer for related scenarios, so usage guidance remains mostly implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_code_scanningA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the transparency burden. It does disclose the key behavioral trait: setting trigger_scan=true auto-disovers, triggers, and polls the CodeQL workflow, while the default is to return existing open alerts. It does not address auth permissions, rate limits, or empty-result behavior, but the core side-effect profile is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with no fluff: purpose, return fields, and trigger behavior are each covered once. The field list is justified because the tool has no output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema and no annotations, the description covers the essential invocation context: what it returns, how to interpret the results, and how trigger_scan changes behavior. It is slightly incomplete around failure modes, such as what happens if no CodeQL workflow exists or polling times out.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already documents all five parameters. The description adds useful output-field semantics (message_text explains why code is vulnerable) but no additional parameter-level meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific action ('Get open Code Scanning alerts (CodeQL) for a GitHub repository') with a clear resource and even lists the per-alert fields returned. This clearly distinguishes it from sibling tools like check_ci_status, analyze_dependencies, and get_health_score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by naming CodeQL Code Scanning alerts but never explicitly says when to choose this tool over alternatives or when not to use it. The trigger_scan guidance is parameter-level behavior, not tool-selection guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.