Server Quality Checklist

Tool Scores

• penni_get_pnlB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must cover behavior. It says 'Get' implying read-only, but no confirmation of idempotency, side effects, or constraints like date range limits or permissions. Minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely short single sentence with no fluff. Every word earns its place, but could benefit from slight expansion. Still effective for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and simple but not trivial parameters, the description omits what the report contains, period behavior, and error handling. Incomplete for confident use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds no extra meaning to the start_date and end_date parameters beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get', the resource 'Profit & Loss report', and the source 'QuickBooks'. It uniquely identifies this tool among siblings like get_balance or get_invoices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives, no prerequisites or exclusions provided. The description lacks context for appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• penni_create_expenseB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It only mentions the action and a requirement, lacking details on idempotency, error behavior, side effects, or validation. A create operation needs more behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, front-loading the key information. It could be slightly more structured, but it is efficient and to the point.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters and no annotations, the description is too brief. It lacks information on return values, error scenarios, restrictions on categories, or duplicate handling, making it incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema; it does not elaborate on any parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (create), resource (expense entry in QuickBooks), and a key prerequisite (Penni subscription). It distinguishes itself from sibling tools like penni_create_invoice by specifying the target resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for creating expenses but does not explicitly state when to use this tool versus alternatives like penni_create_invoice. No guidance on when not to use or prerequisites beyond the subscription.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• penni_create_invoiceB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral traits. It only notes that sending is optional and requires a subscription, but does not disclose side effects (e.g., what happens in QuickBooks), permissions, rate limits, or return values. This is insufficient for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single 12-word sentence that efficiently conveys the core purpose and a key prerequisite. It is front-loaded with the action, but could benefit from slightly more structure without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters (one optional boolean) and no output schema, the description leaves gaps: it does not explain what the tool returns after creation, how the 'send' option works in detail, or any error conditions. The agent lacks sufficient context to use it reliably.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 4 parameters have descriptions in the input schema (100% coverage), so the baseline is 3. The description adds no extra meaning or context beyond what the schema already provides, such as format hints or relationships between parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create and optionally send an invoice') and the resource ('through QuickBooks'). It distinguishes itself from sibling tools like penni_get_invoices (viewing) and penni_create_expense (expenses) by specifying the exact operation and target system.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions a prerequisite ('requires Penni subscription') but provides no explicit guidance on when to use this tool versus alternatives. It implies usage when an invoice needs to be created, but lacks exclusions or references to sibling tools for different tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• penni_get_balanceB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only states the action (get balances) but does not mention side effects, authentication requirements, rate limits, or whether it is read-only. As a simple retrieval tool, the lack of additional transparency is a gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no unnecessary words. It is front-loaded with the action and resource, making it easily scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters, no output schema, and no annotations, the description is complete enough to understand the tool's purpose. It might benefit from noting what 'current' means or that it returns multiple account balances, but it is adequate for a simple retrieval tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so the description does not need to add parameter-level details. The schema description coverage is 100% trivially, and the description adequately implies the lack of parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it gets current bank account balances from QuickBooks, with a specific verb and resource. However, it does not explicitly differentiate from sibling tools like penni_get_invoices or penni_get_pnl, so the distinction is implicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, nor does it mention any prerequisites or exclusions. The usage context is purely implied by the tool's name and description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• penni_askA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose behavioral traits such as read-only nature, side effects, or rate limits, leaving the agent uninformed about tool behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no unnecessary words, efficiently conveying the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple Q&A tool with one well-documented parameter, the description is sufficient; however, it could be improved by hinting at the response format or providing an example question.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the description adds no extra meaning beyond the schema's parameter description, meeting the baseline but not exceeding it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Ask') and resource ('Penni a bookkeeping question about your QuickBooks data'), distinguishing it from sibling tools that perform specific actions like create or get.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for bookkeeping questions but does not explicitly state when to use this tool versus siblings, nor does it provide exclusions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• penni_get_invoicesA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description accurately identifies the operation as a read ('Get') with a specific filter ('open and overdue'). No annotations are provided, but the description itself is transparent about the behavior. It does not disclose any potential side effects, but none are expected for a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that conveys the essential purpose without any extraneous words. It is front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters and no output schema, the description is adequate. It explains what the tool does and what it returns. However, it could mention the return format or data structure to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so the description does not need to add parameter meaning. The baseline for no parameters is 4, and the description sufficiently implies the tool returns all open and overdue invoices without filters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves 'open and overdue invoices' from QuickBooks, specifying a specific verb and resource. It distinguishes from siblings like 'penni_create_invoice' and 'penni_get_balance' by focusing on a specific subset of invoices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. For example, it does not explain when to use this over 'penni_get_balance' or other tools, nor does it mention any prerequisites or context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.