Server Quality Checklist

Tool Scores

• query_downtimeC
  2.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It correctly implies a read-only query (not destructive) via 'Return downtime events', which is appropriate. However, it does not disclose potential performance impacts of unfiltered queries, date format expectations, or whether results are sorted or paginated (the limit parameter hints at pagination but is not explained).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it omits necessary detail for a tool with 5 parameters and no output schema. Every sentence should add value; here, the sentence is adequate but incomplete for the complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, 0 required, no output schema, and complex sibling tools, the description is under-specified. It does not mention return format (e.g., list of events with fields), default behavior (e.g., recent events if no time range), or error handling. The description covers only the general purpose, leaving gaps for an agent to resolve.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 20% (only cause_code has a description). The description mentions filtering by asset id, time range, or cause code but does not explain the format of start/end (e.g., ISO 8601, Unix timestamp) or the meaning of asset_id. It adds value by grouping filters but lacks critical details that would help an agent use parameters correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns downtime events with optional filters by asset id, time range, or cause code. This distinguishes it from siblings like list_assets (assets) or query_production (production events), though it could more explicitly contrast with these related tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like find_anomalies or summarize_throughput. The description does not mention exclusions, prerequisites, or context for interpretation (e.g., that downtime may overlap with production data).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_productionB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It doesn't specify whether this is a read-only operation, if it requires specific permissions, if results are paginated beyond the limit parameter, or the structure of the returned data. The hint about ISO 8601 timestamps is helpful but insufficient for a data-returning tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short—two sentences with no fluff. It's front-loaded with the main purpose and then adds the optional filter details. It's concise but perhaps too sparse given the lack of annotations and output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description must explain what the tool returns, but it doesn't. The tool has 4 optional parameters, which may require guidance on default behavior (e.g., if no time range given, does it return the last hour? all data?). The lack of behavioral transparency and return value documentation leaves significant gaps for an agent to use this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description mentions 'asset id and/or time range' which maps to the asset_id, start, and end parameters, but adds no extra semantics beyond what the schema already states. The time range format hint ('ISO 8601 timestamps') is valuable but repeats the schema's property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns hourly production counts and notes optional filtering by asset id and/or time range. The verb 'query' plus the resource 'production counts' makes the purpose specific, but it doesn't explicitly differentiate from sibling tools like query_downtime or summarize_throughput.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides context on when to use it—for hourly production counts with optional filters. However, there are no explicit guidelines on when not to use it versus alternatives like summarize_throughput or find_anomalies, leaving the agent to infer based on the tool name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_assetsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility for behavioral transparency. The description indicates it's a read operation listing items with metadata, which implies non-destructive behavior. However, it doesn't disclose whether the list is paginated, ordered, or if authentication is required. A score of 3 is appropriate as it conveys basic read behavior but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the tool's purpose and the data it returns. Every word contributes meaning, with no wasted content. It is optimally concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has zero parameters, no output schema, and no annotations, the description provides adequate but minimal information. It explains what is listed and what metadata is included, but lacks details on pagination, sorting, or authentication. For such a simple tool, the description is sufficient to understand its core function, but not thoroughly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameters and 100% description coverage, so the baseline is 3. The description does not add parameter-level details since there are none, but it explains what metadata is returned (type, line, target throughput), providing some value beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all machines/production lines with their metadata. It specifies the verb 'list' and resource 'machines / production lines', and distinguishes itself from siblings that focus on production queries, downtime, throughput, and anomalies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites or situations where one of the sibling tools would be more appropriate. The agent is left to infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_throughputA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It describes the tool as a read-style roll-up returning aggregated metrics, which implies non-destructive behavior. However, it does not mention error handling, time zone assumptions, data freshness, or whether the operation is safe/read-only, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core purpose. Every sentence adds value: the first defines the action, the second lists return fields. No redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no output schema, the description partially compensates by listing return fields. However, it does not describe the output structure (e.g., list of objects, keys), error responses, or handling of empty results. Given the low complexity, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It names the time window as 'optionally bounded' and implies start and end are the boundaries. But it does not specify the expected format (e.g., ISO 8601), inclusive/exclusive behavior, or defaults when omitted. The addition is modest but not fully sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: it rolls up production counts versus target throughput per asset. It specifies the return values (total produced, target, attainment ratio, downtime minutes) and the time window. This differentiates it from sibling tools like query_production (raw data) and query_downtime (specific metric).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as query_production or query_downtime. The description only states what the tool does, without context for choosing among siblings or noting prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_anomaliesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the burden. It adds context about the algorithm (z-score, standard deviations from mean) and the meaning of the threshold, but does not explicitly state whether the operation is read-only, how it handles missing asset_id (all assets), or any side effects. It also does not clarify the full return structure beyond timestamps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long and immediately states the tool's purpose and output. There is no redundancy or irrelevant information; every word contributes to understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple with 2 optional parameters and no output schema. The description explains the algorithm and that timestamps are returned, but it does not specify the full return format (e.g., whether it includes the anomaly score or value) or the scope when asset_id is omitted. This is a gap given the absence of an output schema, but the description is otherwise adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds meaningful context by explaining the threshold's role ('more than threshold standard deviations away from its mean') and introduces the concept of 'hourly count,' which is not in the schema. This adds value beyond the schema's parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Run simple z-score anomaly detection') on a specific resource ('production counts'), and explains the output ('Returns timestamps'). This distinguishes it from sibling tools like list_assets or query_production, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need to detect anomalies in production counts), but does not explicitly state when not to use it or mention alternative tools like query_production. There are no exclusions or comparisons, so guidance is only implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.