Server Quality Checklist

Tool Scores

• chatB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the payment requirement ('Pay per request with a Lightning spend token') which is crucial context, but doesn't describe rate limits, authentication needs beyond the token, error behavior, response format, or whether this is a read/write operation. For a tool with financial implications and no annotations, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with two sentences that efficiently convey core functionality and key constraints. It's front-loaded with the primary purpose, though the second sentence could be slightly more structured. Every sentence earns its place by providing essential information about the service and model support.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given this is a paid API call tool with financial implications and no output schema, the description is incomplete. It doesn't explain what the tool returns (response format, error handling), doesn't mention cost implications beyond requiring a token, and provides no guidance on model selection or performance characteristics. For a tool with this complexity and no structured output documentation, the description should do more.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all four parameters thoroughly. The description adds minimal value beyond what the schema provides - it mentions the payment token requirement and model providers, but doesn't provide additional parameter semantics, constraints, or usage examples beyond the schema's parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Send a message to an AI model via LightningProx'), identifies the resource (AI models from multiple providers), and distinguishes this tool from its siblings (payment/invoice/balance/model-listing tools) by focusing on message sending rather than payment management or metadata retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context ('Pay per request with a Lightning spend token') but doesn't explicitly state when to use this tool versus alternatives. It mentions the tool supports 19 models from specific providers, which provides some contextual guidance, but lacks explicit 'when-not-to-use' instructions or named alternatives for similar functions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_balanceB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions the return format ('balance in sats'), which is helpful, but lacks critical details: whether this is a read-only operation (implied but not stated), authentication requirements, rate limits, error conditions, or what happens with invalid tokens. For a financial tool, this is insufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It front-loads the core purpose ('Check the remaining balance'), specifies the resource, and states the return value—all essential information in minimal text. Every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (single parameter, no output schema, no annotations), the description is minimally complete. It covers what the tool does and the return unit, but lacks behavioral context (e.g., safety, errors) that would be needed for robust agent use. Without annotations or output schema, the description should provide more operational details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents the single parameter 'spend_token' with its type, description, and format hint ('starts with lnpx_'). The description adds no additional parameter semantics beyond what's in the schema, meeting the baseline score of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Check the remaining balance') on a specific resource ('LightningProx spend token') with the expected return unit ('balance in sats'). It distinguishes from sibling tools like 'check_payment' (which likely verifies payment status) and 'generate_invoice' (which creates payment requests).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. While it's clear this tool checks balance on a spend token, there's no mention of prerequisites (e.g., token must be valid), when not to use it, or how it differs from 'check_payment' which might have overlapping functionality. The agent must infer usage from the name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the return data structure but lacks details on error handling, rate limits, authentication requirements, or whether the list is static or dynamic. For a read-only list tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and efficiently adds specific details about the return content and model count in a single, well-structured sentence. It avoids unnecessary elaboration, though it could be slightly more concise by integrating the model count into the main clause.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate but not fully complete. It covers the purpose and output semantics but lacks behavioral context like error cases or update frequency. For a basic list tool, this is minimally viable but leaves room for improvement in transparency.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the inputs. The description appropriately does not discuss parameters, focusing instead on the output content. This meets the baseline for tools with no parameters, as it avoids redundancy.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all AI models'), identifies the resource ('available through LightningProx'), and provides distinguishing details about the return content ('model IDs, names, providers, and pricing') and scope ('19 models across Anthropic, OpenAI, Together.ai, Mistral, and Google'). This differentiates it from sibling tools like chat or check_payment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites, context for usage, or comparisons with sibling tools. The agent must infer usage based solely on the tool name and description without explicit direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_invoiceA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context by stating what the tool returns ('BOLT11 payment request and charge ID') and how to use the output ('Pay the invoice with any Lightning wallet'), but it lacks details on permissions, rate limits, or error handling, which are important for a financial transaction tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose and efficiently structured in two sentences that provide essential information without waste. However, it could be slightly more concise by integrating the payment instruction more seamlessly, but overall it is well-sized and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a financial tool with no annotations and no output schema, the description is moderately complete. It covers the purpose and output but lacks details on error cases, authentication needs, or response format. For a tool generating invoices, more contextual information would enhance completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with the parameter 'amount_sats' clearly documented. The description does not add any additional meaning or details beyond what the schema provides, such as constraints or examples not already covered. The baseline score of 3 is appropriate as the schema handles the parameter documentation adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Generate a Bitcoin Lightning invoice') and resource ('to top up a LightningProx spend token'), distinguishing it from sibling tools like check_payment or get_balance. It precisely defines the tool's function without being vague or tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning 'top up a LightningProx spend token' and 'Pay the invoice with any Lightning wallet,' but it does not explicitly state when to use this tool versus alternatives like check_payment or get_balance. No exclusions or prerequisites are provided, leaving usage guidance incomplete.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_paymentA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It describes the polling behavior and the fact that it retrieves a spend token upon confirmation, which are useful behavioral traits. However, it lacks details on error handling, rate limits, or what happens if the invoice is unpaid, leaving some gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero waste: the first states the core functionality, and the second provides critical usage guidance. It is front-loaded with the main purpose and appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (polling operation), no annotations, and no output schema, the description is mostly complete. It covers purpose, usage, and basic behavior, but lacks details on return values or error cases, which would be helpful for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents the charge_id parameter. The description adds minimal value by referencing 'Charge ID returned by generate_invoice', which reinforces but doesn't significantly expand beyond the schema. Baseline 3 is appropriate as the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('check if...has been paid', 'retrieve the spend token') and identifies the resource ('Lightning invoice'). It distinguishes from siblings by referencing generate_invoice and implying a workflow relationship, making it unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance on when to use this tool ('Poll this after generate_invoice until the payment is confirmed'), including a prerequisite (after generate_invoice) and a stopping condition (until payment confirmed). This clearly differentiates it from other tools like get_balance or list_models.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.