Server Quality Checklist

Tool Scores

• list_ticketsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does add a key behavioral detail: the result is limited to tickets created in this session, which implies a linguistic scope or filter. However, it gives no indication of read-only status, possible pagination, or whether the session is the current conversation or something else, leaving significant behavioral unknowns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely short, which is appropriate for a zero-parameter tool. It is front-loaded with the key word 'Tickets' and the scope 'in this session', but it is a fragment rather than a proper sentence. Still, it is concise and to the point with no waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although an output schema exists, it is not visible here, so this review relies on the description alone. The description gives the essential scope but leaves 'this session' undefined and does not mention any feature like ordering or limit. For a trivial list tool, this is near adequate, but the phrase could cause confusion about the meaning of 'session'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, so the schema is empty and the baseline is 4. The description correctly does not repeat parameter information, and no additional parameter semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Tickets created in this session' is a noun phrase rather than a clear action statement; it implies the tool returns tickets scoped to the current session but does not explicitly say 'list' or 'retrieve'. It does suggest the resource (tickets) and the scope, which helps distinguish it from get_customer, get_transactions, and create_ticket, but the wording leaves room for interpretation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to choose this tool over alternatives like search_knowledge (which might find tickets) or create_ticket (which creates them). The description does not explain that this is a quick way to see session-created tickets or that it avoids calling a broader search, so an agent has no explicit selection criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_customerB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the output fields, which is useful, but it does not mention read-only behavior, authentication requirements, potential errors, or data freshness. For a tool with no annotations, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tight phrase with no filler words. It front-loads the resource name and immediately lists the key output fields, making it easy to scan and understand.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool, the description plus schema are enough to understand the broad purpose and required input. However, missing behavioral context and parameter explanation leave notable gaps, especially with no output schema or annotations to supplement the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only the parameter name and type (customer_id: string) with zero description coverage. The description does not explain the parameter, its format, or how it identifies the customer. It adds no value beyond the structured schema for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource as a customer profile and enumerates the returned fields (verification status, VIP tier, balance, active bonus). It does not use an explicit verb like 'retrieves', but the tool name and field list make the purpose unambiguous and distinct from siblings like get_transactions or list_tickets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage is implied: this is the tool to call when a customer profile is needed. However, the description does not explicitly state when to prefer this over alternatives, nor does it mention any exclusions or related tools. There is no direct guidance about scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_knowledgeA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavior disclosure. It does reveal the output shape (ranked sections with scores), yet it omits any statement about side-effects, authentication needs, or read-only status. For a search tool, the lack of such details is not alarming, but the coverage is only minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only two sentences, front-loading purpose and a clear usage constraint. The wording is direct and free of redundant detail. It loses a small amount because the final set of consumers is directive rather than purely behavioral, but it is still appropriately sized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema already present, the description need not detail return values. However, it leaves the meanings of `top_k` and `category` unclear, and with no annotations the agent relies entirely on the scarce description. The tool is simple enough that this is acceptable, but it is not fully self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description is the only place to define parameters. While the action 'search the support knowledge base' is clear, it never explains what `query` should contain, how `top_k` controls results, or how `category` filters them. Therefore two of three parameters are effectively undefined.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('search') and a resource ('support knowledge base'), and adds output details ('ranked sections with a score'). It clearly stands apart from the sibling tools like get_customer, get_transactions, list_tickets, and create_ticket, since this is the only knowledge search tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'state policies only from these' is an implied rule for when to trust the results, and the tool name obviously points to knowledge-base lookups. But it doesn't explicitly explain when to choose this tool over alternatives or when it should not be used, leaving the selection decision partly to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_transactionsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does reveal key behaviors: it returns deposits and withdrawals only, ordered newest first, and is evidently a read operation by tone. It does not mention the limit default of 5, pagination/truncation, or whether only settled transactions appear, which are meaningful behavioral gaps for a no-annotation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single 12-word sentence that is front-loaded with the main verb, resource, and ordering. Every word earns its place and no irrelevant details are included.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple two-parameter tool and the presence of an output schema, the description covers the essential retrieval intent, scope, and ordering. It does not explain the limit parameter or any possible restrictions, but the tool is simple enough that an agent can reasonably call it with customer_id alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for both parameters. It implies that customer_id identifies whose transactions appear, but it never names the parameter or explains that limit restricts how many transactions are returned. The 'newest first' phrase weakly signals count-limiting, but the description does not define either parameter's semantics with certainty.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific resource (customer transactions) and the exact scope (deposits and withdrawals) with ordering (newest first). This clearly distinguishes get_transactions from siblings like get_customer, list_tickets, and create_ticket without needing to open schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The usage context is implied: use this when you need a customer's most recent deposits and withdrawals. However, there is no explicit guidance about when not to use it or which alternative tool fits other transaction-related needs, leaving the agent to infer boundaries from sibling tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_ticketA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavior disclosure. It states the core side effect (escalate to human agent) and return value, but gives no detail on side effects such as ticket creation side effects, persistence, or whether a transcript is attached.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short, purposeful sentences with the core behavior front-loaded. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers the trigger conditions and return value, and names the core side effect (escalating). However, it omits parameter-level semantics and any operational context, such as how category/transcript should be supplied, given the low schema coverage and no annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description does not explain any parameter. Parameters like category and transcript are left to inference, so the description adds no semantics beyond the parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action and destination: 'Escalate to a human agent' and 'Returns the ticket.' This unambiguously identifies the tool as a ticket-creation/escalation operationless and distinguishes it from the read/search sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use only on explicit request or when the tools cannot answer,' which gives a clear trigger condition and an implicit exclusion of routine use when other tools suffice. This is strong usage guidance for an AI agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.