Server Quality Checklist

Tool Scores

• query_mobility_metricsC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior, but it only mentions 'allowlisted' without explaining the whitelist or constraints. No mention of data scoping, rate limits, or side effects of parameters like null city or date range.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise but under-specified for a 5-parameter tool. It lacks structure and front-loads no key details beyond the basic action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters and no schema coverage or annotations, the description is insufficient. It does not explain allowlisting, date range behavior, grouping, or city filtering, making it incomplete for effective tool selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description adds no information about parameters. Despite the schema having enums for metric and group_by, the description fails to explain them, leaving the agent without guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it queries a metric from the mobility warehouse. However, it does not differentiate from sibling tools like query_charging_risk, missing context on what makes this tool unique.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description lacks context about prerequisites, typical use cases, or scenarios where querying mobility metrics is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• retrieve_mobility_policyC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It implies read-only retrieval but does not state this explicitly, nor does it mention auth requirements, rate limits, or any side effects. The description is too sparse to inform the agent of behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, making it concise. However, it is too terse and omits critical information about parameters and usage, making it under-specified rather than efficiently brief.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has 2 parameters with no descriptions and sibling tools that differ in purpose, the description is incomplete. It does not explain the output format (though an output schema exists) nor how to effectively use the parameters. The minimal information forces the agent to guess.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% for both parameters ('query' and 'top_k') and the description adds no explanation of their meaning or usage. For a tool with two parameters, the absence of any parameter semantics severely hampers correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool retrieves policy passages for grounded mobility-operations guidance. The verb 'Retrieve' and resource 'policy passages' are specific, and the scope distinguishes it from sibling tools (query_charging_risk, query_mobility_metrics) which focus on risk and metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description mentions the domain (mobility-operations guidance) but does not provide exclusions, prerequisites, or comparative context with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_charging_riskC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. The description only states what is returned but does not disclose any behavioral traits such as side effects, data freshness, error conditions, or pagination. With no annotations, the description should provide more behavioral context but fails to do so.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence. It provides the core purpose without verbosity. However, it could include more value without being overly long.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, return values are covered externally. But the description lacks information on how to use the optional city parameter and what constitutes 'charging failure' or 'state-of-charge metrics'. The context is incomplete for effective usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The single optional parameter 'city' is not explained in the description. The agent has no indication of what the parameter does or how it affects results. The description completely fails to add meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Return depot-day charging failure and state-of-charge metrics' clearly states the tool returns specific metrics (charging failure and SOC) at depot-day level. It distinguishes itself from siblings: query_mobility_metrics (likely general mobility) and retrieve_mobility_policy (policy retrieval).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus its siblings or any prerequisites. The description does not mention any context or exclusion criteria for using the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.