Server Quality Checklist

Tool Scores

• billingserv_getA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must bear the burden of transparency. It states 'Fetch live billing data' implying read-only, but does not explicitly confirm no destructive side effects, authentication needs, or rate limits. Listing many endpoints gives some context, but behavioral details are insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the purpose and listing data types concisely. The second sentence adds usage guidance and a constraint. It is efficient, though the list of data types is somewhat long.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description does not explain return values or structure for the many endpoints. It covers what data can be fetched but lacks details on output format. For a tool with numerous endpoints, more completeness would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description does not add parameter-specific information beyond the schema; it only provides high-level context about data types. The schema already describes 'path', 'query', and 'endpoint' adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it fetches live billing data and lists many data types (customers, invoices, etc.). The verb 'fetch' and the context distinguish it from sibling tools 'billingserv_create' (write) and 'billingserv_list_endpoints' (endpoint listing), making its read-only purpose evident.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises 'Use this for any question about billing, invoices, customers, orders, or revenue,' providing clear context. It also notes the restriction 'Only allowlisted endpoints can be called.' However, it does not explicitly state when not to use it or directly compare with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• billingserv_list_endpointsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden. It accurately describes the tool as a read-only listing operation covering many areas. No destructive or hidden behaviors are implied. It is straightforward and complete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first gives the primary action, second details categories. No wasted words, front-loaded with key information. Excellent structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no output schema, the description provides sufficient context about what endpoints are covered (read vs write). It could mention return format (e.g., list of endpoint objects) but is otherwise complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, so baseline is 4. The description does not add parameter info, but none is needed. Schema coverage is 100% (empty schema), so no deficiency.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all BillingServ API endpoints with descriptions and required parameters, and distinguishes itself from sibling tools 'billingserv_create' and 'billingserv_get' by being a discovery tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for exploring available endpoints before calling create or get, but does not explicitly state when to use or not use it. Sibling tool names provide context, but more direct guidance would improve clarity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• billingserv_createA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It warns that the tool changes real billing data and can email customers, requiring user confirmation. It also notes that only allowlisted endpoints are callable and that deletions/payments are unavailable. This discloses key behavioral traits, though it could mention whether it returns created records or errors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and front-loaded: first sentence states purpose and lists record types, second adds email capability, third warns about behavior and limitations. Every sentence adds useful information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description doesn't explain return values, but it references billingserv_list_endpoints for detail on required/optional fields. It covers input parameters well and provides necessary caveats. For a creation tool, missing return details is a minor gap, but overall completeness is good for agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds value by explaining dotted field names and how to pass them (nested objects vs dotted keys), and references billingserv_list_endpoints for per-endpoint field details. This goes beyond the schema's own descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The title and description clearly state the tool creates or updates various BillingServ record types and can send emails. It distinguishes itself from sibling tools (billingserv_get, billingserv_list_endpoints) which are read/list operations. The description is specific about the resource and action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool (create/update records, send reminders) and when not to use it (deleting records, capturing payments not available). It advises confirming details with the user before calling, providing clear usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.