Server Quality Checklist

Tool Scores

• get_pageA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the output format ('clean markdown') and default version, but does not mention error behavior, permissions, or read-only status.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded: a one-sentence summary followed by a clear Args block with examples. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple fetch tool, the description covers purpose, parameters, and return format. It lacks error handling or edge-case details, but is sufficient for typical use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description explains both parameters in detail, including examples for the path and the default behavior for version, adding significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch one documentation page as clean markdown', with a specific verb and resource, and it distinguishes itself from sibling tools like search_docs or find_api.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It does not explicitly say when to use this tool vs alternatives, but it provides clear context by mentioning the path comes from search_docs and the version defaults to the installed cudaq version, implying a workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_examplesA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden. It adds some behavioral context by explaining the source of examples ('shipped in the CUDA-Q repository at the matching release') and that results are 'complete, runnable.' However, it does not describe return format, pagination, or failure modes, leaving gaps typical of a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured with a clear opening sentence, a brief source/usage note, and a parameter list. It is not as lean as a two-sentence ideal, but every section earns its place given the absent schema descriptions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema or annotations, but the description covers purpose, source, usage preference, and all parameters. It lacks an explicit description of the return value, but the phrase 'Find complete, runnable CUDA-Q example programs' suggests the actual code is returned, making it reasonably complete for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description's Args section is essential. It explains each parameter meaningfully: query with examples ('GHZ state', 'VQE'), language filter, version default, and limit default. This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Find complete, runnable CUDA-Q example programs,' which clearly states the verb (find) and resource (example programs). It distinguishes this tool from siblings like search_docs, find_api, and get_page by focusing specifically on examples with a stated preference for adapting them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'Prefer adapting these over writing kernels from memory.' It also clarifies the source and version matching. However, it does not explicitly state when not to use this tool or name alternatives, falling just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_apiA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains parameter defaults and filtering behavior, but does not disclose how partial matches are resolved (e.g., whether multiple results are returned) or the exact output structure beyond 'canonical definition and doc URL.' This leaves some ambiguity for the agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: one clear purpose sentence plus a compact Args list. It is front-loaded and contains no filler. Every line adds value, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with no output schema, the description is mostly complete: it covers all parameters and gives the core outcome. However, it lacks detail about the return format or how to handle ambiguous partial matches, which would be useful for the agent to know. Still, it is complete enough for a straightforward resolve operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully. It does: each parameter is clearly explained with examples (name), allowed values and default behavior (language), and default versioning (version). This goes well beyond the bare schema titles and provides rich, actionable semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific and actionable purpose: 'Resolve a CUDA-Q API symbol to its canonical definition and doc URL.' This clearly distinguishes the tool from siblings like search_docs (which presumably searches docs more broadly) by focusing on resolving a specific symbol to its authoritative definition and documentation URL.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the tool's function and parameter behaviors (e.g., 'omit for both', 'Defaults to the installed cudaq version'), giving the reader enough context to decide when to use it. However, it does not explicitly mention when not to use it or suggest alternative tools such as search_docs or get_page, so it stops short of full exclusions or alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_targetsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It explains the content coverage (simulators, hardware, cloud) and that entries include selection snippets and doc URLs, but it does not describe the return format, potential side effects, or whether data is fetched remotely. This is an acceptable disclosure for a simple listing tool but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well structured: a concise one-sentence summary, an explanatory paragraph with the supported categories and guidance, a clear usage trigger, and a compact Args section. Every sentence adds value and the most important information appears first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with one optional parameter and no output schema, the description covers purpose, when to use it, what categories are included, and the parameter semantics. It is sufficient for an agent to select and invoke the tool correctly without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description fully compensates by documenting the 'category' parameter with its allowed values: 'simulator', 'hardware', or 'cloud'. This adds meaningful semantics beyond the bare schema, which only shows a nullable string with a default of null.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and the resource ('CUDA-Q execution targets (backends)') and adds that it provides guidance on choosing. It distinguishes itself from sibling documentation/search tools by focusing on available execution backends.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage guidance: 'Call this when deciding where to run a kernel or when the user names a provider.' It does not mention when not to use it or name alternatives, but the context is clear enough for the common cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the result format (breadcrumb, excerpt, canonical doc URL) and the version fallback behavior. It implies read-only operation but does not explicitly state safety, rate limits, or other behavioral traits beyond what is already in the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a clear purpose statement, a usage directive, and a concise parameter breakdown. Every sentence serves a purpose, and the structure is front-loaded with the key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers all essential aspects: purpose, when to use, parameter semantics, and the result format. It even explains the version default behavior. For a search tool with no output schema, this is a complete and self-sufficient description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero descriptions, but the description fully compensates with an 'Args' section that explains each parameter: query (with natural language examples), version (format and default to installed cudaq version), and limit (maximum results, default 5). This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search') and identifies the resource ('NVIDIA CUDA-Q documentation') and what it returns ('ranked excerpts'). This clearly distinguishes it from sibling tools like get_page and find_api, which focus on page retrieval and API lookup respectively.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use the tool: 'Use this before answering any CUDA-Q question from memory', with a rationale about the platform moving quickly. It does not explicitly name alternatives or exclusion conditions, but the primary use case is clearly communicated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.