Server Quality Checklist

Tool Scores

• check_repo_healthB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for disclosing behavior. It only states the tool 'analyse[s]' a repository, but does not clarify whether this is read-only, whether it performs network calls, or what metrics are involved. No side effects, permissions, or return behavior are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded: it states the purpose in one line and provides parameter documentation in another. Every word earns its place, with no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one parameter and an output schema, so return-value details may be covered elsewhere. However, the description lacks any usage guidance relative to sibling tools and does not clarify what 'contributor-friendliness' means. It's minimally sufficient but leaves significant gaps in practical context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'repo' is described as 'Full repo name, e.g. "fastapi/fastapi"' which adds practical meaning beyond the schema's bare string type and required flag. The example clarifies the expected format, compensating well for the 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's function: 'Analyse a repository's contributor-friendliness.' This is a specific verb+resource that distinguishes it from sibling tools like check_issue_status or find_issues. However, it doesn't elaborate on what 'contributor-friendliness' entails, so it's clear but not maximally precise.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention exclusions, prerequisites, or sibling tools. The intended usage context is implied by the phrase 'contributor-friendliness' but never made explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_contribution_guideA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action but gives no details about whether it is read-only, requires network access, or how it handles missing guides. There is no mention of side effects or failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, front-loading the core purpose in the first sentence and providing parameter details in the second. Every word earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with an output schema, the description adequately covers the purpose and parameter format. It does not need to explain return values due to the output schema, though it could add a note about potential errors when no guide exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful value for the 'repo' parameter by specifying the expected format ('Full repo name') and providing an example. Since the schema only has a type and title with no description, this guidance is helpful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Pull and summarise') and resource ('contribution guide + setup instructions'). It is easily distinguishable from sibling tools focused on repo health, issues, and finding issues.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description only implies when to use the tool (when contribution guide/setup instructions are needed) and does not explicitly mention alternatives or exclusions. While the sibling context makes the use case fairly obvious, explicit guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_issue_statusA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description only states the core purpose without disclosing behavioral details such as authentication requirements, what 'available' means (e.g., open, unassigned, not locked), or potential side effects. This leaves the agent to infer the tool's safety and conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, leading with the purpose, then the parameter definition. Every word is functional.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter and an output schema, the description covers the basic operation but lacks context on what 'available' entails, when to choose this over find_issues, and any prerequisites. It is adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description clarifies that the parameter should be a 'Full GitHub issue URL' (not just an issue number), adding meaning beyond the schema's basic 'Issue Url' string. Since schema coverage is 0%, this compensation is helpful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'check' and resource 'specific GitHub issue', clearly indicating it verifies availability of a single issue. It distinguishes itself from sibling tools like find_issues (searching) and check_repo_health (repo-level).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies it should be used when you have a specific issue URL and want to know if it's workable. However, it does not mention alternative tools or when not to use it, so it misses explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_issuesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure and does well: it explains that it searches GitHub, factors the scoring (repo health, freshness, clarity, merge friendliness, setup ease), and describes the default behavior of unassigned_only. It does not mention rate limits or API auth, but for a search tool this is sufficient detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a concise summary paragraph followed by a clean Args list. Every sentence adds value, and the formatting makes the tool immediately scannable. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers all 8 parameters, the ranking logic, the search source, and an edge-case behavior (unassigned_only=False to widen searches). It does not document the return format, but an output schema exists, so that burden is satisfied elsewhere. For a tool of this complexity, this is fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description fully compensates by providing detailed parameter semantics: defaults, allowed values for sort_by, example values for language and topic, and a behavioral note for unassigned_only. This is exemplary parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Find beginner-friendly open source issues ranked by likelihood of success.' It clearly distinguishes from siblings (check_repo_health, check_issue_status, get_contribution_guide) by focusing on searching and ranking issues rather than checking individual repos or issue status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the primary use case obvious (finding beginner-friendly issues) and gives contextual advice, such as setting unassigned_only=False to widen searches when narrow combinations return zero hits. However, it does not explicitly name alternatives or state when NOT to use this tool, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.