Server Quality Checklist

Tool Scores

• get_usageC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does not state whether this is a read-only operation (though 'get' implies it), whether it requires authentication, what the response format looks like, or what happens when zero usage exists. For a tool with zero annotation coverage, more context is needed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence with no wasted words. It clearly states the action and resource. However, it is somewhat under-specified given the ambiguity about what 'usage statistics' means, so it sits at 4 rather than 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple (1 optional parameter, no output schema), which lowers the complexity bar. However, with no annotations and an ambiguous term like 'usage statistics', the description doesn't fully clarify what data is returned or how it relates to the sibling tools. The lack of an output schema means the description should hint at return semantics, which it doesn't.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema fully documents the single 'days' parameter with its default value. The description adds no information about the parameter beyond what the schema provides—for instance, it doesn't clarify the valid range or what units. Baseline 3 is appropriate when the schema carries the parameter documentation burden.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Get recent usage statistics from your Farebox account' which is a clear verb+resource combo. However, it doesn't distinguish itself from sibling tools like get_balance, which could also be confused as a usage/account tool. The phrase 'usage statistics' is somewhat ambiguous—does it mean API usage, token usage, or something else?

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. With siblings like get_balance and chat, an agent might struggle to decide whether get_usage or get_balance is appropriate for a query about account consumption. No exclusions or alternative recommendations are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• call_skillB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full disclosure burden. It does not mention any behavioral aspects: whether the tool is read-only, whether it incurs costs/usage, effects on model state, or how output is structured. For a tool that processes content through a model with potential cost implications, the lack of any behavioral disclosure is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence that conveys purpose and a comparative benefit with minimal waste. It's appropriately front-loaded, stating the action first and examples after. Could arguably mention a caveat or additional context, but for its brevity it is well structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 3 params, full schema coverage, and no output schema, the description gives the essential purpose and a usage hint. It is adequate for agent selection. However, it does not explain what each skill produces, whether output formatting varies, or any cost considerations—information that would matter for an agent deciding which skill to invoke, especially since there is no output schema to fill these gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are fully documented structurally. The description enumerates example skill names that overlap with the enum but adds no meaning beyond the schema—it doesn't explain what 'input' expects in terms of format/length, or what the 'model' override does functionally. With full schema coverage, the baseline is 3, and the description does not exceed it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource ('Call a Farebox built-in skill') and lists example skills (summarize, translate, code-review, etc.), making the core purpose clear. It does not explicitly distinguish from sibling tools, though 'Farebox built-in skill' implies a narrower pre-defined scope versus the generic 'chat' sibling. The examples help but are somewhat redundant with the schema enum.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers a comparative benefit ('Faster than crafting a prompt from scratch'), suggesting when users would prefer this over a chat/prompt approach. However, it does not explicitly state when NOT to use it or describe alternatives like chat for custom/unstructured needs. There is no guidance on when the generic chat sibling would be more appropriate than a fixed skill.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chatB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It states what it does (chat completion, returns reply) but doesn't disclose cost implications, rate limits, or whether message history needs to include prior responses for multi-turn conversations. For a cost-incurring tool, this is a moderate gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that state purpose and return value with zero waste. Efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a moderately simple 4-param tool with 100% schema coverage, the description is largely sufficient. It could mention that this is a cost-incurring operation for balance tracking (given get_balance/get_usage siblings exist), but the core usage is well covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with all four parameters documented (model, messages, max_tokens, temperature). The description adds nothing beyond the schema, so baseline 3 is appropriate given full coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb+resource ('Send a chat completion to any Farebox model') and states the return value. It's clear, though it doesn't distinguish itself explicitly from siblings like list_models or call_skill beyond the obvious chat-purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (chatting with a model) and the 'any Farebox model' phrasing suggests it works across models. But it doesn't explicitly address when not to use it, or contrast with sibling tools like call_skill (e.g., when a skill is needed).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the return type ('Returns an array of model objects') which adds behavioral context. However, it doesn't disclose details like whether pricing is always included, ordering guarantees, pagination behavior, or failure states. For a read/list operation the safety profile is fairly evident from the name, but the description is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficient at two sentences without filler. It front-loads the core purpose (list models with pricing) and adds the return type detail in the second sentence. It's appropriately sized for a zero-parameter list operation, though it could arguably be even more informative about the return structure in the same space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with zero parameters and no output schema, the description covers the essential information: what it lists (all Farebox models) and what's included (pricing). The return type is disclosed. Given the low complexity, the description is reasonably complete, though it could mention whether models are returned in a specific order or whether any filtering is possible.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, so schema description coverage is trivially 100%. With zero parameters, the description's job is minimal here. The description doesn't need to explain parameter details since there are none, earning a solid baseline score for this dimension.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states what the tool does: lists all available Farebox models with pricing. The verb 'List' plus the specific resource 'available Farebox models' with the scope of including pricing is specific enough. It distinguishes from siblings like get_balance and get_usage which are clearly different resources, though it doesn't explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the tool to call when you need to enumerate available models and their pricing. However, it doesn't explicitly state when to use this vs alternatives, nor does it mention any context in which it shouldn't be used. There's no explicit exclusion or alternative guidance given other than what the resource name naturally conveys.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_balanceA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full disclosure burden. It states it returns balance and spending totals, which implies a read-only operation. However, it does not describe the return format, whether spending totals are date-scoped, or any rate-limit/auth requirements. It's adequate but thin.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence with zero waste. It efficiently communicates the resource (Farebox), the action (Get balance and spending totals), and the unit (USD). No filler or redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with a clear purpose and no output schema, the description is mostly sufficient. It explains what is returned (balance and spending totals in USD). It could add a note about whether spending totals reset or cover a specific period, but given the simplicity of the tool, this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, so the baseline is 4 per the rubric. The description adds value by clarifying that the tool requires no input and returns balance plus spending totals in USD, going beyond the empty schema. Score 5 is justified because with no params, nothing is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and specific resource ('Farebox account balance and spending totals in USD'). It distinguishes itself from siblings like get_usage by focusing on Farebox account balance specifically rather than general usage metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (when you need account balance/spending totals) but provides no explicit guidance on when NOT to use it or which sibling tools to use instead. get_usage is a natural alternative, but no differentiation is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.