Server Quality Checklist

Tool Scores

• open_incidentB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing side effects. It does reveal two behavior modes (JSONL log in demo, ticketing webhook in live), which is useful. However, it does not disclose permissions required, reversibility, failure modes, or any other potential side effects beyond where the record is sent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action and resource. It includes essential context about demo vs live modes without unnecessary words. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 required parameters and no annotations or schema descriptions, the description is too minimal. It omits parameter semantics and usage guidance. The demo/live distinction is helpful but does not make the tool usable without additional information about inputs and expected behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 4 required parameters with zero description coverage, and the tool description does not explain any of them. The names (summary, severity, diagnosis, model_name) are somewhat self-explanatory, but there is no guidance on expected format, allowed values (no enums), or how they relate. The description adds no value beyond the parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Open an incident record', with a specific verb and resource. It also adds context about demo (JSONL log) and live (ticketing webhook) environments, which distinguishes it from sibling tools like rollback and drift report operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool vs alternatives. The description only mentions demo/live modes, but does not state conditions for opening an incident or situations where another tool would be more appropriate. No exclusions or alternative suggestions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_rollbackA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It mentions token rejection but omits critical details such as what the rollback actually changes, whether it is destructive, what permissions are required, and what happens on success. For a mutation tool, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no filler. 'Execute an approved rollback' front-loads the core purpose, and 'Rejects missing/invalid tokens' adds a key behavioral constraint. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a high-stakes rollback tool, the description is critically underspecified. It lacks guidance on when to use it relative to alternatives, side effects, prerequisites beyond the token, and error behavior. The existence of an output schema does not compensate for the absence of usage context and behavioral disclosure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no parameter-specific explanations. While the parameter names (model_name, target_version, approval_token) are somewhat self-explanatory, the description does not clarify their roles or constraints beyond what the schema already shows.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Execute an approved rollback.' The verb 'execute' is specific and distinguishes this from the sibling 'propose_rollback.' It also mentions token validation, adding a behavioral detail that clarifies the tool's scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'approved rollback' implies the tool should only be used after approval, and 'Rejects missing/invalid tokens' indicates that a valid approval token is a prerequisite. This provides clear contextual guidance without explicitly naming alternatives, but the sibling name 'propose_rollback' strongly implies the distinction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• propose_rollbackA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states 'Read-only — does not execute,' which is a critical safety trait. However, it does not mention what the generated plan contains, whether any state changes occur, or any permission requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded: 'Generate a rollback plan for human review' states the core purpose immediately, followed by the key safety qualifier. Every word earns its place, and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity of 3 required parameters and no parameter descriptions, the description is too minimal to enable correct invocation. It explains the main purpose and safety, but the lack of parameter semantics and any contextual guidance about when to generate a rollback plan leaves significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 3 required parameters with zero description coverage, and the tool description does not explain what model_name, current_version, or reason should contain or how they are used. The description completely fails to compensate for the lack of schema-level parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Generate a rollback plan for human review.' It clearly distinguishes the tool from siblings like execute_rollback by adding 'Read-only — does not execute,' making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it is for generating a plan for human review, not for executing it. The 'does not execute' phrase implicitly tells the agent not to use this tool when an actual rollback is needed, but it does not explicitly name alternatives or give more detailed when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diff_deploymentsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the behavior of comparing specific attributes across versions, which is useful. However, it does not explicitly state whether the operation is read-only or if any side effects occur. The term 'diff' implies non-mutating, but this is not confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and a specific list of diffed attributes. The second sentence adds contextual value by labeling it as a root-cause tool. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While an output schema exists (so return values are covered), the description lacks details on parameter formats and any prerequisites or constraints (e.g., whether versions must be consecutive). It adequately conveys the tool's primary purpose but leaves gaps in operational details that could trip up an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions 'two model versions' which maps to version_a and version_b, but does not explain the expected format for versions or model_name. The description lists diff categories (training data span, params, etc.) but those are not the parameters themselves. Thus, an agent gets no additional clarity on how to fill the three required parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Diff two model versions' with a specific list of what is diffed (training data span, params, eval metrics, feature schema). It also distinguishes itself from siblings by labeling itself as 'the primary root-cause tool,' making its role unique among the listed tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool—for root-cause analysis of what changed between deployments. It says it correlates 'what changed between the incumbent and the newly deployed version,' giving clear context. However, it does not mention specific exclusions or when to prefer sibling tools like get_drift_report or summarize_metric_anomalies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_deployment_historyA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavioral traits. 'List' implies a read-only operation, and 'newest first' discloses ordering. However, it does not disclose whether transitions include all types, how the limit parameter affects behavior, or any prerequisites. This is moderate transparency but leaves gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundant words. It conveys the action, resource, and ordering efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, return values are already specified. The description covers the core purpose and resource, and mentions ordering. It could be more complete by explaining limit behavior, but for a simple list operation it is largely sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It implicitly explains model_name as the model to retrieve history for, but it does not explain the limit parameter. The description adds some meaning but not complete compensation for both parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists recent deployments/version transitions for a model, with an explicit ordering of newest first. This distinguishes it from siblings like propose_rollback or diff_deployments, which target different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: use this tool to retrieve deployment history for a model. It does not explicitly mention when not to use it or name alternatives, but the sibling names and the verb 'list' provide adequate contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_metric_anomaliesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses the algorithm (rolling z-score), the output style (metric statistics with anomaly windows flagged), and a key limitation (no raw time series). This goes beyond a simple statement and gives behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences, front-loaded with the main purpose followed by a clarifying detail about output and algorithm. Every word earns its place, with no redundant fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no annotations, presence of an output schema), the description is sufficiently complete. It explains what the tool does, how it flags anomalies, and what it returns, leaving only minor gaps around error cases or edge behavior that are likely covered by the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not explicitly explain the parameters model_name or lookback_hours, and schema coverage is 0%. However, the parameter names are self-explanatory and the schema provides the default for lookback_hours. The description adds little parametric meaning beyond what the schema already communicates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'summarize' and the resource 'serving-metric anomalies' with specific metric types (latency, error rate, prediction stats). It distinguishes itself from sibling tools by focusing on anomaly summarization rather than rollback, incidents, drift, or deployment history.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by noting it is 'suitable for an LLM to reason over without raw time series,' which implies when to use it. However, it does not explicitly name alternatives or state when not to use the tool, stopping short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_drift_reportA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description is the sole source of behavioral disclosure. It explains that it computes a report and returns drift metrics, but does not disclose potential side effects, permissions, rate limits, or performance implications. The read-only nature is implied but not explicit, and there is no mention of prerequisites beyond a deployed model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with a clear summary, and uses a simple Args list for parameters. Every sentence adds value, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema, so return values need not be repeated. The description covers purpose, method, and all parameters adequately. Minor gaps like usage guidance relative to siblings and explicit behavioral traits prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description fully compensates by providing meaningful explanations for all three parameters. It defines model_name as the registry name, and gives both default values and units for the window sizes, adding clarity beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it computes a data/prediction drift report, comparing serving windows with PSI + KS tests per feature. It also lists specific return outputs (per-feature scores, overall drift flag, top drifted features), distinguishing it from sibling tools like rollback or incident management.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies the context (deployed model drift detection) and compares current vs reference windows, making clear when it is relevant. However, it does not explicitly mention alternatives or exclusions, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.