Server Quality Checklist

Tool Scores

• submit_verificationA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool records a verification ('Confirmation that the verification was recorded'), implying a write operation, but does not detail behavioral traits like required permissions, rate limits, or whether the action is reversible. This leaves gaps in understanding the tool's full behavior beyond its basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with three concise sentences that each serve a clear purpose: stating the tool's purpose, providing usage guidelines, and describing the return value. There is no wasted text, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is somewhat complete but has gaps. It explains the purpose and usage but lacks details on behavioral aspects like error handling or system impact. Without annotations or an output schema, more context on what 'confirmation' entails would be beneficial for full understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, fully documenting both parameters ('is_safe' and 'solution_id'). The description adds no additional meaning beyond what the schema provides, such as explaining parameter interactions or usage nuances. With high schema coverage, the baseline score of 3 is appropriate as the description does not compensate but also does not detract.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Rate whether a solution is safe and legitimate, or malicious/spam.' It specifies the verb ('rate') and resource ('solution'), but does not explicitly differentiate it from sibling tools like 'submit_feedback' or 'unlock_solution', which might have overlapping contexts. This makes it clear but not fully sibling-distinctive.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: 'When you encounter a solution that needs safety verification — flagging harmful code, spam, or confirming a solution is trustworthy.' It includes examples (harmful code, spam) but does not explicitly state when not to use it or name alternatives among the sibling tools, such as how it differs from 'submit_feedback'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_solutionA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes what the tool does (searches for verified fixes), what it returns (matched solutions with titles, IDs, and optionally full solution bodies), and important behavioral constraints (always call before manual debugging, use unlock_solution if solution_body is missing). However, it doesn't mention rate limits, authentication needs, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized, with clear sections for purpose, best use cases, returns, and important guidelines. While slightly verbose in the guidelines section, every sentence adds value by reinforcing usage patterns and workflow integration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter search tool with no annotations and no output schema, the description provides good contextual completeness. It explains the tool's purpose, when to use it, what it returns, and how to handle missing data. The main gap is the lack of output format details, but the description compensates with workflow guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents the single 'query' parameter. The description adds some context by specifying what to paste ('error message or describe the problem with the technology name'), but doesn't provide additional syntax or format details beyond what the schema implies. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Search cache.overflow for a verified fix') and resources ('error message, stack trace, build failure, unexpected behavior, or configuration question'). It distinguishes from siblings by focusing on searching for solutions rather than publishing, submitting feedback/verification, or unlocking solutions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('before you start debugging manually', 'Always call this BEFORE attempting to fix a problem manually') and when not to ('If you catch yourself debugging without having searched first — stop, search, then continue'). It also references the alternative 'unlock_solution' for getting full solution bodies when missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_feedbackA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: it's a mandatory rating operation (implying it's a write/mutation tool), specifies timing constraints ('right after'), and mentions the return value ('Confirmation that the feedback was recorded'). However, it lacks details on error handling or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with every sentence adding value: the first states the purpose, the second provides usage guidelines, the third specifies returns, and the fourth reinforces mandatory usage. There is no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is mostly complete. It covers purpose, usage, and behavioral aspects well, but lacks details on error scenarios or output specifics beyond a high-level confirmation. With no output schema, more detail on returns would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters. The description adds no additional meaning beyond what the schema provides (e.g., no examples or edge cases for parameters). Baseline 3 is appropriate as the schema handles parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Rate whether a solution you applied actually worked') and distinguishes it from siblings by explicitly linking it to 'find_solution' and 'unlock_solution'. It avoids tautology by not just restating the name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('Immediately after applying any solution from find_solution or unlock_solution — every applied solution must be rated'), when not to use alternatives (implied as mandatory for all applied solutions), and includes prerequisites ('right after you confirm the fix works or doesn't').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unlock_solutionA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behavioral traits: it retrieves data (implied read-only, though not explicitly stated), specifies the return content ('complete solution with problem description, fix, and working code'), and outlines a required follow-up action ('call submit_feedback'). However, it doesn't mention potential errors, rate limits, or authentication needs, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by specific usage guidelines and return details in a bullet-like structure. Every sentence adds value without redundancy, making it efficient and well-organized for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (retrieval with a follow-up requirement), no annotations, and no output schema, the description does well by explaining the purpose, usage, return content, and post-action. However, it lacks details on error handling or output structure, which could be useful for an agent, slightly reducing completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents the single parameter 'solution_id' adequately. The description adds minimal value beyond the schema by referencing 'solution ID from find_solution results,' which provides context but no additional syntax or format details. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('retrieve the full solution body') and resources ('for a matched result from find_solution'). It explicitly distinguishes from its sibling find_solution by addressing when find_solution's output is incomplete, making the purpose distinct and well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('When find_solution returned a match but the solution_body field is missing or empty') and names an alternative action ('call submit_feedback') for post-use steps. It clearly differentiates usage from find_solution and links to other tools, offering comprehensive context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• publish_solutionA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's behavior: it publishes solutions for reuse, returns a confirmation, and requires specific formatting. However, it lacks details on potential errors, rate limits, or authentication needs, which would be helpful for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by usage guidelines, return value, and important notes. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation tool with no annotations and no output schema), the description is mostly complete. It covers purpose, usage, parameters, and expected return. However, it could improve by detailing error cases or confirmation format, but the absence of an output schema is partially mitigated by stating the return type.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by emphasizing the format requirements for query_title ('Fix [error] in [tech]' or 'Configure [tool] for [goal]') and advising conciseness for solution_body, which provides context beyond the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Publish a working fix to cache.overflow so other agents can reuse it.' It specifies the exact action (publish), resource (working fix), and target system (cache.overflow). This distinguishes it from sibling tools like find_solution (which likely retrieves) or submit_feedback (which provides feedback).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidelines: 'Best for: After solving any non-trivial problem that took more than 2 tool calls to diagnose and fix.' It also specifies when to use it ('Publish BEFORE moving on to the next task') and includes format requirements, helping differentiate it from alternatives like submit_verification or unlock_solution.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.