Server Quality Checklist

Tool Scores

• search_librariesC
  2.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, destructiveHint=false, and idempotentHint=true, so the safety profile is known. The description adds that searches yield 'valid' pairs and warns that 'Assembly differences are not performance measurements,' offering useful caveats. However, it does not clarify pagination behavior or the relationship between the search backend and compilation, so only modest incremental context is provided.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence is concise and front-loaded, but the second and third sentences ('Compilation tools transmit...' and 'Assembly differences...') are extraneous and unrelated to the search operation, making the description less concise than it should be.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although an output schema exists, the description lacks context on how search results integrate with compile workflows, what makes a library 'valid,' and how the caveat about assembly differences affects usage. The description is adequate for a basic search but insufficient for an agent to make nuanced decisions in the tool ecosystem.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero schema description coverage, the description must explain all parameters. 'Search and page through' implies limit/offset are for pagination, but it does not specify what 'query' matches (library name, version, both) or how 'exact version-ID' constrains the query. This leaves most parameter semantics ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states the tool's action ('Search and page through') and resource (valid C++ library and exact version-ID pairs), distinguishing it from sibling tools like search_compilers and search_analyzers. However, the subsequent sentences about compilation tools and assembly differences are tangential and slightly obscure the main purpose, preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. The mention of compilation tools transmitting source files hints at an integration context but does not instruct the agent on selection criteria or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_opcode_documentationB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive behavior. The description adds a useful caveat ('Assembly differences are not performance measurements') but includes a confusing statement about compilation tools that does not apply to this tool. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence is concise and front-loaded, but the second and third sentences are extraneous and appear misplaced ('Compilation tools transmit...' and 'Assembly differences...'). This bloated structure harms clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with an output schema and strong annotations, the first sentence provides the core functionality. However, the irrelevant sentences reduce completeness and may mislead the agent, leaving gaps around what 'bounded' means and how to interpret results.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It only says 'explicit instruction-set and opcode IDs' without explaining valid value formats or examples. This adds minimal meaning beyond the schema parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Look up'), the resource ('opcode documentation'), and the required inputs ('explicit instruction-set and opcode IDs'). This is specific and distinguishes the tool from siblings like search_compilers or compile_cpp.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when opcode documentation is needed with specific IDs, but it provides no explicit guidance on when to use this tool over alternatives, nor any exclusions. The second and third sentences are irrelevant to usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_compilersA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds some behavioral detail beyond the readOnlyHint: it mentions pagination ('page through') and that results include 'current stable alias resolutions.' However, the second sentence about assembly differences is tangential to searching and may confuse the agent about the tool's actual behavior. Annotations already cover the read-only/idempotent nature, so the description adds modest value but also introduces noise.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence is concise and front-loaded with the core purpose. The subsequent sentences ('Compilation tools transmit...' and 'Assembly differences are not performance measurements.') are not directly relevant to searching and add unnecessary length, reducing overall conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a readOnlyHint, an output schema, and simple optional parameters, the description is mostly sufficient for a basic search tool. However, the inclusion of unrelated compilation caveats undermines completeness by introducing ambiguity about whether the tool itself compiles code. The description would be more complete if it focused solely on the search behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must carry the burden of explaining parameters. It only vaguely implies that 'query' is used for searching and 'limit/offset' for pagination. It does not specify what the query matches (e.g., compiler names, aliases) or how pagination behaves. This is weak compensation for the complete lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action (search and page through) on a well-defined resource (C++ compiler IDs and current stable alias resolutions). This distinguishes it from sibling tools like search_libraries and search_analyzers, which target different entity types. The verb and object are unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a clear use case: finding compiler IDs before invoking compilation tools. It even mentions that compilation tools transmit source files to the backend, suggesting this search is a precursor. However, it does not explicitly state when NOT to use it or mention alternative tools, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_cppA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds behavioral context beyond the annotations: it reveals that source and virtual files are transmitted to a Compiler Explorer backend, notes that diffs are bounded, and clarifies baseline-relative status. These details are useful and do not contradict the read-only, idempotent, non-destructive hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the core function and return values, followed by backend transmission context and a key caveat. Every sentence contributes meaningful information with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with a nested schema and an output schema, the description captures the essential semantics: input type, number of configurations, output contents, and a critical non-performance warning. It omits specific details like how baseline is chosen or how window bounds diffs, but the output schema likely covers return structure, making the description reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate by explaining parameter meanings, but it offers only oblique hints (e.g., 'two to six configurations' implies cases length, 'source and virtual files' maps to source/files). It does not explain the window parameter, baseline selection, or the structure of cases, leaving the agent to rely on schema titles alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: compiling a C++ source bundle under 2-6 configurations and returning baseline-relative status, assembly hashes, counts, and bounded unified diffs. This distinguishes compare_cpp from sibling tools like compile_cpp (single configuration) and analyze_cpp (static analysis).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied by the tool's scope (comparing multiple configurations), but the description does not explicitly state when to use this tool versus alternatives such as compile_cpp or analyze_cpp. The only guidance is a caveat that assembly differences are not performance measurements, which serves as a when-not interpretation rather than a use-case directive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_cppA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, open-world, idempotent, and non-destructive behavior. The description adds valuable context: output is bounded, failure status is aggregated, static-modeling caveats exist, and assembly differences are not performance measurements. This goes beyond the schema and annotations, though it doesn't detail all edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences pack the core purpose, constraints, and caveats without redundancy. The first sentence is front-loaded with the action and resource, and the second adds necessary context about transmission and a performance warning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (8 parameters, including nested objects) and 0% schema coverage, the description is not fully complete. It covers the high-level functionality and key constraints, but does not guide parameter usage or reference sibling tools like search_analyzers for discovering analyzers. An output schema exists, so return details are not required, but parameter guidance is still lacking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the full burden. It clarifies 'source' and 'files' (source and virtual files) and adds the critical 'up to four' constraint for analyzers, but it does not explain compiler, filters, window, libraries, compiler_arguments, or how to construct analyzer selections. Most parameters remain undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool runs up to four selected Compiler Explorer analyzers in a single compile-only request, immediately distinguishing it from compile_cpp and compare_cpp. It also specifies the output format (normalized bounded per-tool output with aggregate failure status).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'compile-only request' provides clear context that this tool is for static analysis rather than execution, and the mention of 'selected analyzers' implies prior selection (likely via search_analyzers). However, it does not explicitly name alternatives or state when not to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compile_cppA
  4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description adds valuable behavioral details: no execution occurs, outputs are bounded, source and virtual files are transmitted to a backend, and assembly differences are explicitly not performance measurements. This enriches the agent's understanding of side effects and limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and contains no redundant or extraneous information. Every phrase adds value, including the caveat about assembly differences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (11 parameters) and the presence of an output schema, the description covers key behavioral aspects and return types are handled by the schema. However, it does not provide sufficient guidance on parameter selection or configuration, leaving a gap in usability for complex compilation scenarios.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% across 11 parameters. The description only hints at a few concepts (source bundle, diagnostics, assembly, optimization records), leaving other critical parameters like compiler, libraries, filters, and files unexplained. It fails to compensate adequately for the lack of parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool compiles a C++ source bundle without executing it, and specifies the output types (bounded diagnostics, assembly, optimization records). This distinguishes it from sibling tools like compare_cpp and analyze_cpp by focusing on pure compilation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for compilation tasks and explicitly notes it does not execute code, but it does not name alternative tools for execution or analysis, nor does it provide clear when-not-to-use guidance. The context is clear but lacks explicit alternatives or exclusion rules.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_shortlinkA
  4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the annotations: it states that the source and settings are 'transmitted for persistent, publicly retrievable storage' and that 'there is no delete operation.' This clarifies the externally visible side effects and irreversibility, which is critical for a mutation tool with openWorldHint=true. It also discloses the validation step, which annotations do not capture.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, starts with the primary action and outcome, and uses the second sentence solely for an important side-effect warning. There is no redundant repetition of schema or annotation information, and every clause contributes meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description covers the core behavior and persistence implications, it omits important operational details: it does not explain what 'Validate' entails, whether compilation failures abort the operation, or that compilers is optional despite the 'one to six' phrasing. The schema is rich, but the description introduces a potential restriction (one to six) not present in the schema and does not fully align with the null default, leaving ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the individual parameters (source, compilers, validate_compilation) beyond an implicit mention of 'one to six resolved compiler configurations.' The description does not clarify what 'resolved' means, what happens with validate_compilation=false, or the structure of compiler configurations, leaving the agent without sufficient guidance for parameter selection.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with a specific verb ('Validate and permanently store') and names the exact resource (a C++ source with compiler configurations) and result (a shareable Compiler Explorer shortlink). It clearly distinguishes from sibling tools like get_shortlink, compile_cpp, or analyze_cpp by focusing on persistent creation rather than compilation, comparison, or retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the use case—creating a shareable, persistent shortlink—and warns that there is no delete operation, which indirectly advises caution. However, it does not explicitly state when to prefer this tool over alternatives such as compile_cpp or get_shortlink, nor does it provide explicit exclusions or conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_shortlinkA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already provide readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is clear. The description adds valuable context that retrieved source is untrusted external content, which is a security-relevant behavioral trait not captured by the annotations. It does not detail failure modes, but the output schema likely covers return behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two sentences and every word earns its place. The first sentence states the primary action and target, and the second adds an essential security warning. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with output schema and comprehensive annotations, the description covers the core purpose and a key risk factor. It lacks explicit guidance on how to interpret the retrieved content or what happens for invalid IDs, but the presence of an output schema mitigates the need to describe return values. Overall, it's adequate for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, shortlink_id, is described indirectly as 'a built-in Compiler Explorer shortlink ID,' giving it meaning beyond the bare schema. However, the description does not explain the format, origin, or any constraints of the ID, leaving some ambiguity. Given the 0% schema coverage, the description partially compensates but could be richer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: inspect C++ source and compiler settings from a shortlink ID. The verb 'inspect' is specific and distinguishes it from sibling tools like create_shortlink or compile_cpp. It also adds important context that the source is untrusted.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies you should use this tool when you have a Compiler Explorer shortlink ID and want to see its source/settings, but it does not explicitly state when to use it over alternatives or mention that it pairs with create_shortlink. No exclusions or alternative references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_analyzersA
  4.1/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite readOnlyHint and idempotentHint annotations, the description adds that compilation tools transmit source/virtual files to the Compiler Explorer backend (privacy/behavior) and warns that assembly differences are not performance measurements, providing caution beyond the annotations. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The first sentence is clear and front-loaded, but the second and third sentences are tangential and somewhat ambiguous, adding notes about compilation tools and performance measurements that may not be necessary for a search tool; still the description is short.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a simple search with output schema and annotations. Description covers the core purpose and provides important caveats (backend transmission, performance caveat), but omits detail on the compiler checking mechanism and pagination parameters, making it mostly complete but with residual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%; the description mentions 'optionally checking one compiler' (compiler param) and implies query by talking about discovering IDs, but does not explain query matching semantics or the limit/offset pagination parameters, leaving them to inference.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Discover' and enumerates the exact analyzer families (clang-tidy, IWYU, llvm-mca, OSACA, PVS-Studio) along with 'IDs and recognized aliases,' and clarifies it can optionally check a compiler. This clearly differentiates from sibling tools like search_compilers and search_libraries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for finding supported analyzer IDs/aliases and mentions optional compiler checking, but does not explicitly exclude alternatives or state when to prefer this over search_compilers/search_libraries. The context of the tool's name and parameter 'compiler' gives clear usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.