Server Quality Checklist

Tool Scores

• set_device_valueA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry burden. It mentions simulation and fault injection but lacks details on side effects, persistence, or permissions needed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that conveys the core functionality without unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given it's a simple setter with no output schema, the description could be complete, but it omits parameter details and behavioral implications, leaving gaps for a write operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter descriptions and the tool description does not explain device_id, metric, or value. Parameter semantics are almost entirely missing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it writes device parameters, simulating PLC writes, and mentions fault injection. This distinguishes it from sibling read/list tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides usage contexts (simulating PLC writes, fault injection demos) but does not explicitly contrast with alternatives; still clear enough for typical use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_alertsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full disclosure burden and does meaningfully convey the all-device iteration default and the warning/critical tiering of results. It doesn't disclose whether the operation is safe/idempotent, whether the all-device scan has rate-limit or permission implications, or what happens with an invalid device_id. For a simple read-like check, it's adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded Chinese sentence delivers the verb, resource, conditional behavior, and return structure with zero fluff. Every clause earns its place, and the most decision-relevant behavior (the no-argument path) is placed at the core of the sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and an available output schema (which covers return structure), the description adequately covers purpose, the default no-argument path, and result tiering. The main omission is contrasting the provided-device_id case explicitly, but for this low complexity (1 param, no required fields, no nested objects), the description is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% — the schema only defines an unadorned nullable string with a null default. The description partially compensates by clarifying the meaningful null case (scan all devices), which is genuinely useful semantics beyond the schema. However, it never explicitly states what providing a device_id does (filtering to that device, presumably) or the expected format, so parameter meaning remains only partially established.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb-resource pairing ('检查告警' / check alerts) and adds concrete behavioral scope: when device_id is omitted, it iterates all devices and returns tiered alert lists ('warning / critical 分级告警列表'). It stops short of 5 because no sibling tool in the list (list_devices, get_device_metrics, etc.) obviously overlaps with alerts, so explicit differentiation isn't tested.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys usage context by clarifying the default behavior when device_id is not passed ('不传 device_id 时遍历全部设备') and identifying the two severity tiers in the response, which helps an agent understand the tool's scope. However, it provides no explicit when-to-use vs. alternatives guidance, exclusions, or performance caveats about scanning all devices. The usage guidance is implicitly conveyed rather than explicitly directed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metric_historyA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must convey all behavioral traits. It only states the action (reading sample values) and does not disclose side effects, permissions, or error behavior, nor explicitly confirm it is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no redundant information, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, inputs, and intended use case. Output format is likely handled by the output schema (which exists). It lacks edge-case details but is adequate for a simple read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning by indicating that '某指标' refers to the metric parameter, '设备' to device_id, and '最近 N 个采样值' to points. However, it does not explain the default value of points or any constraints, leaving some gaps given the schema lacks descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool reads recent N sample values of a device metric, specifying the verb (read), resource (metric history), and scope (device). It differentiates from siblings by mentioning time series data and visualization use cases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions a use case (trend charts / 3D visualization) but does not explicitly advise when to use this tool over alternatives like get_device_metrics. The guidance is implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_device_metricsA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral transparency. It discloses a critical side effect: each call advances a sampling period, which is not obvious from the schema. This is valuable context beyond the basic read operation, though it doesn't detail auth or rate limits, which are less critical for a read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, compact sentence that conveys purpose, output, and a key behavioral trait. It is front-loaded with the core function and contains no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, so the description doesn't need to detail return values. The tool is simple with one parameter, but the description doesn't address edge cases like invalid device IDs or what happens if no metrics are available. It also doesn't connect to siblings for context, so completeness is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for undocumented parameters. It mentions device_id implicitly as 'device' but doesn't add any extra meaning like format, constraints, or default values. The description adds minimal value beyond the schema, and with only one parameter, the baseline is 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads real-time device metrics and returns metric name/value/unit. It also notes that each call advances one sampling period. This distinguishes it from siblings like get_metric_history and get_scene_snapshot, though it doesn't explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for real-time data retrieval, and the note about advancing a sampling period hints at when to use it (call repeatedly to get successive samples). However, it does not explicitly state when to prefer get_metric_history over this tool for historical data, or provide clear exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_devicesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool lists all devices with type and location, which is a read-only behavior by implication. However, it does not explicitly state safety (read-only, no side effects), potential performance implications (e.g., large result sets), or any authentication requirements. This is adequate for a simple list operation but lacks explicit behavioral guarantees, so a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence in Chinese that is front-loaded with the action and resource. It contains zero filler words and directly states the purpose and key output fields. This is exemplary conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (no parameters, no nested objects) and the existence of an output schema, the description is fully complete. It states exactly what the tool does and what information is included. There is no ambiguity or missing context for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema has 100% coverage (vacuous truth). The rubric sets a baseline of 4 for zero parameters. The description adds no parameter-specific info, but none is needed. It also hints at output content (type and location), which is not parameter semantics but does enrich the tool's usage context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (list), resource (all devices), and scope (digital twin scene), and specifies that type and location are included. It clearly distinguishes from sibling tools like get_device_metrics or get_metric_history, which focus on metrics and history, not device enumeration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context ('in digital twin scene') and it's obvious when to use this tool (to retrieve a device roster). It does not explicitly state when not to use it or name alternatives, but the sibling tool list makes the differentiation clear. This earns a 4 as clear context with no exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_scene_snapshotA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. It discloses a key behavioral trait—a single call returns all devices' data—and specifies the content categories (coordinates, real-time metrics, graded alerts). It does not detail potential payload size or data freshness, but for a zero-parameter read-only snapshot it provides meaningful transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the core purpose and includes the key differentiator ('one call returns all devices'). Every word contributes value; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema and no destructive or mutating behavior, this description is complete. It states the return contents, scope, and recommended usage context, making it sufficient for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and an empty input schema, so the baseline is 4. The description adds no parameter-specific details because none are needed; all behavior is encapsulated in the fixed snapshot semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-plus-resource pattern: 'returns all devices' coordinates, real-time metrics, and graded alerts in one call.' It clearly distinguishes this aggregate snapshot tool from sibling tools like get_device_metrics and check_alerts by emphasizing the scene-wide scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states the intended use case: 'exclusively for visualization / monitoring polling.' This gives clear context for when to choose this tool, though it does not explicitly name sibling alternatives or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.