Server Quality Checklist

Tool Scores

• get_metric_summaryB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states that it retrieves aggregated TopN values, but does not say whether the call is read-only, what happens when start_time/end_time are omitted, whether threshold filtering is applied before or after aggregation, or how pagination/limits behave.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The structure is compact and effective: a precise summary line, a well-organized Args list, and a short Returns line. Every entry earns its place, and the parameter list is scannable. It loses one point because the Returns section is extremely terse, though an output schema exists to fill that gap.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (13 parameters, no annotations, 0% schema coverage), the description covers parameter semantics well but leaves critical contextual gaps: when to use it versus sibling tools, whether time ranges are required, and how the TopN grouping behaves. It is a usable but incomplete definition.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so thoroughly: every one of the 13 parameters gets context, including concrete examples for namespace and metric_name, allowed values for aggregate and combine, format guidance for time parameters, and defaults for period and top_n. This is exactly the kind of compensation needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The one-line summary states a specific verb and resource: fetch aggregated TopN metric values grouped by a label_key. It is clear about the operation, but it does not explicitly differentiate itself from siblings like get_metric_data or search_metric; it relies on the phrase 'aggregated TopN' to imply the distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to prefer this tool over alternatives such as get_metric_data or search_metric. It lists parameters and return type, but never states the conditions, prerequisites, or scenarios for which this tool is the right choice.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_apiA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the transparency burden. It does disclose the return behavior, noting that Query APIs return only core parameters and queryableFields, which is useful context beyond the tool name. However, it does not mention potential errors, authentication needs, or other behavioral traits, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured, with a clear purpose statement followed by Args and Returns sections. Each sentence earns its place, and the example parameter value makes the usage instantly understandable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with an output schema, the description is largely complete. It covers the purpose, parameter meaning, and return behavior. It could be slightly stronger by clarifying why a user would choose describe_api over search_api, but that is more of a usage-guideline gap than a completeness issue.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains that api_name is an API name and gives a concrete example, which is sufficient for this single-parameter tool. More detail about accepted formats or validation rules would push it higher, but the provided semantics are clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches detailed parameter documentation for a specified ZStack API, using a specific verb and resource. However, it does not explicitly differentiate itself from siblings like search_api or execute_api beyond the tool name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when you already know the API name and need its parameter details, illustrated by the example "QueryVmInstance". There is no explicit guidance on when to prefer this tool over search_api or execute_api, so usage context is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metric_dataA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full behavioral burden. It discloses output-size scaling with a formula, explains multi-series behavior when labels are omitted, and gives practical warnings for avoiding overly large responses. It does not cover auth, timeout, or error behavior, but for a read-only metric query it is fairly transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The structure is clear and purposeful: a one-line purpose, an Args section covering all parameters, and a valuable note about output size. The length is justified for a 7-parameter tool with no schema descriptions, though the Returns section adds little beyond what an output schema would already provide.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For 7 parameters, no annotations, and an output schema present, the description covers the essential call semantics, data-volume behavior, and return type. Remaining gaps are minor: behavior when start_time/end_time are omitted, and the exact effect of summary_only on the returned structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates completely by explaining every parameter with examples, units, defaults, and format notes. It also clarifies labels and summary_only semantics beyond the schema, and the volume formula gives practical meaning to start_time, end_time, and period.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with '获取 ZStack 监控数据' (get ZStack monitoring data), naming a clear verb and resource. It does not explicitly contrast with siblings like get_metric_summary or search_metric, so differentiation is inferred from names rather than stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus alternatives such as get_metric_summary or search_metric. The description focuses on how to use parameters and warns about output size, but it never states when this tool is the right choice or when to prefer a sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_apiA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It discloses the critical safety trait that only read-only APIs are allowed by default and that write operations require an explicit environment variable. It also states the return format (JSON) and documents pagination/field-selection behavior, which goes beyond the sparse schema. It could add error behavior or mutation warnings, but the key behavioral constraints are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a short purpose, a critical safety note, clearly labeled Args, Returns, and Example sections. Every part adds value: the read-only guard is front-loaded, the conditions format is essential, and the example demonstrates realistic usage. There is no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a generic executor with an open-ended parameters object and no annotations, the description provides the essential information: how to name an API, how to format query conditions, pagination/field controls, and the default write restriction. It is slightly incomplete regarding non-query API parameter conventions and possible error behaviors, but the example and Query API details make it sufficiently complete for common use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so by explaining api_name with an example and by detailing the parameters dict, including the conditions array format, default limit, offset, and fields selection. This adds substantial meaning beyond the bare schema. It does not document parameter formats for write APIs, but the generic nature of the tool makes exhaustive documentation impractical.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('执行 ZStack API' / execute ZStack API) and the target resource (ZStack API), with a concrete example (QueryVmInstance). It does not explicitly contrast itself with siblings like search_api or describe_api, so it misses the top tier for sibling differentiation, but the intended purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear operational guidance: read-only APIs are allowed by default, and write APIs require setting ZSTACK_ALLOW_ALL_API=true. It also explains Query API conditions and pagination defaults, which helps an agent use the tool correctly. However, it does not explicitly state when this tool should be used instead of sibling tools, nor when to avoid it, leaving usage context mostly implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_apiA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full disclosure burden and largely meets it: it reveals the camelCase-splitting matching mode, optional category filtering, and the default cap of 15 results. It does not cover edge cases like empty results or case sensitivity, but for a read-only search tool the core behavioral traits are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of one front-loaded purpose line followed by compact Args and Returns sections. Every clause earns its place, and there is no repetition of schema structure, boilerplate, or redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter search tool with an output schema, this is nearly complete: it covers purpose, matching behavior, all parameters, defaults, and return content. The only notable omissions are explicit sibling routing and edge-case behavior, which are minor at this complexity level.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% — the input schema contains only titles and types. The Args section fully compensates by defining keywords as a list with camelCase matching, category as an optional filter, and limit as a max-return-count defaulting to 15, adding operational meaning the schema itself lacks.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description '根据关键词搜索 ZStack API' names a specific verb (search) and resource (ZStack API), and the Returns section clarifies that it yields API metadata (name, description, category, call type) rather than executing calls. This clearly differentiates it from siblings like execute_api and search_metric, whose targets are different.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a concrete matching-behavior example ('vm' matches 'QueryVmInstance' via camelCase splitting), which helps an agent phrase queries effectively. However, it does not explicitly state when to prefer this tool over describe_api/execute_api or when to use search_metric instead; routing is left implied by sibling names rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_metricA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full behavioral burden. It discloses non-obvious behaviors: camel-case keyword splitting, fuzzy namespace matching, match_mode AND/OR logic, and prefer_namespaces sorting. These details give an agent a realistic model of how search results are filtered and ranked.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The text is front-loaded with the core purpose, then organized under Args and Returns headings. Each line provides necessary operational detail (examples, defaults) without excessive fluff, making the definition scannable and efficient for an LLM to consume.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although an output schema exists, the description still summarizes return contents (name, description, namespace, available labels) and fully documents all five parameters, defaults, and matching/sorting behaviors. With no annotations and no schema-level descriptions, nothing essential is missing for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates completely. Every parameter—keywords, namespace, limit, match_mode, prefer_namespaces—has a format explanation, examples, and defaults. For instance, keywords is shown with ['CPU', 'Usage'] and the camel-case splitting rule, and match_mode explicitly defines 'and'/'or' and the default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific statement: '搜索可用的 ZStack 监控指标' (search available ZStack monitoring metrics). It clearly identifies a search operation over a distinct resource (monitoring metrics), separating it from sibling tools like search_api or get_metric_data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is implied through the name and purpose—searching available metrics before fetching data—but no explicit guidance is provided about when to choose this tool over siblings such as get_metric_data or get_metric_summary. There are no when-not or alternative conditions, leaving an agent to infer the positioning.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.