Server Quality Checklist

Tool Scores

• analyze_assetA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the annotations: the operation is asynchronous, it returns an admission document, and unresolved claims in a successful result are analytical states rather than failures. It does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. Action comes first, polling guidance second, and result interpretation last. Every sentence contributes to correct use.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  It covers the async workflow and result interpretation, which is useful given there is no output schema. However, it omits parameter semantics and any detail about the admission document structure, so the description is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the burden of explaining parameters, but it never mentions address, block, or chain. It adds no meaning beyond the input schema's pattern and const constraints, leaving the agent to infer what each parameter represents.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Submit') and resource ('analysis to the canonical CAI API'), and immediately distinguishes it from the polling sibling by directing the agent to use get_analysis_job. This makes the tool's role unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly frames the operation as asynchronous and tells the agent to use get_analysis_job to poll afterward. It does not provide explicit when-not-to-use guidance for list_assets or get_methodology, but the usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_analysis_jobA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, open-world, and non-destructive behavior. The description adds value beyond annotations by clarifying the meaning of missing/expired jobs and noting that the result schema is unchanged 0.6. No contradiction exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly-worded sentences deliver the core purpose and an important lifecycle caveat without wasted words. The main retrieval statement is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with rich annotations, the description covers the essential behavior: what is retrieved, when it is available, and how to interpret missing/expired states. It does not fully spell out the return shape, but no output schema exists to offload that burden.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description needs to compensate for parameter meaning, but it does not mention job_id at all. The schema only provides a name and length constraints, leaving the agent to infer what a job_id refers to and where it comes from.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the operation: retrieving canonical CAI job state and the schema 0.6 result after success. It uses a specific verb and resource, though it does not explicitly differentiate itself from the sibling tools by name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives useful context: it is used to fetch job state and results after success, and missing/expired jobs are normal lifecycle conditions rather than analytical failures. It does not explicitly name alternatives or when-not-to-use conditions, but the usage context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_methodologyA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already declare readOnly, openWorld, idempotent, and non-destructive behavior, so the description does not need to repeat those. It does add useful context about the document's contents and its canonical nature, but it does not disclose additional behavioral details such as response format, pagination, or any operational side effects. This is acceptable given the rich annotation set.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the core action and resource ('Return CAI's canonical machine discovery document') and then efficiently lists the key contents. There is no fluff, repetition, or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless, read-only retrieval tool with no output schema, the description adequately conveys what the agent will receive, including scope, methodology version, endpoint links, and Score/Confidence semantics. Combined with the annotations, it provides everything needed to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters and schema description coverage is 100%, so there is no parameter gap for the description to compensate for. The description correctly focuses on what the tool returns rather than nonexistent inputs, which is appropriate for a parameterless endpoint.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Return'), a specific resource ('CAI's canonical machine discovery document'), and enumerates its contents (scope, methodology version, endpoint links, Score/Confidence semantics). This distinguishes it sharply from the sibling tools, which analyze assets, retrieve job status, or list assets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the tool's use case obvious: use it whenever the CAI discovery/methodology document is needed. It does not explicitly name alternative tools or exclusion conditions, but none of the siblings are plausible substitutes for retrieving this canonical document, so the guidance is adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_assetsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already cover read-only, idempotent, non-destructive behavior. The description adds valuable behavioral context beyond annotations: the catalog is 'neutral', 'unranked', and 'does not imply endorsement', which is important for an agent deciding whether these assets represent a recommended or prioritized set. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: the first front-loads what the tool returns, and the second adds necessary caveats. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless, read-only listing tool with rich annotations, this description is complete. It states the resource, scope, count, neutrality, and discovery-only nature; no critical detail appears missing for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is no parameter ambiguity for the description to clarify. No parameter documentation is needed, and the baseline of 4 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Return') and resource ('canonical API's neutral 20-asset launch catalog'), and adds qualifiers ('discovery-only', 'unranked') that make the tool's scope clear. This meaningfully distinguishes list_assets from siblings like analyze_asset and get_methodology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear usage context by framing the catalog as 'discovery-only', signaling this is a lightweight listing/browsing operation. It does not explicitly name alternatives or exclusion conditions, but the context is strong enough to guide an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.