Server Quality Checklist

Tool Scores

• repairdesk_inventoryC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The tool is a search, which implies a read-only operation, but the description does not state that explicitly, nor does it mention pagination behavior, response structure, or any side effects. It only states the purpose without disclosing observable behaviors or constraints. Both the read-only nature and pagination are left entirely to inference.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single short sentence with no fluff or redundancy, which is technically concise. However, it is under-specified for an agent; the brevity does not earn its place because it omits essential information. It is neither front-loaded with useful details nor structured to guide usage. It is compact but not informative, so a middle score is appropriate.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 parameters, no output schema, and no annotations, the description is severely incomplete. It does not explain what the search returns, what fields are searched, how to paginate, or what a typical response looks like. An agent cannot confidently call this tool correctly because it lacks the context needed to construct a valid query or interpret results. The description is virtually absent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% coverage (no descriptions for page, keyword, or pagesize), so the description must compensate and explain parameter meaning. It does not mention any of the parameters at all. The agent is left to guess what 'keyword' searches against (e.g., part name, SKU) and how pagination is applied. The description provides zero value beyond the schema structure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Search') and a clear resource ('inventory/parts stock'). It distinguishes the tool from siblings like repairdesk_tickets, repairdesk_customers, and repairdesk_invoices by naming the resource domain. However, it could be more specific about what 'inventory/parts' means in the repair-desk context (e.g., parts availability), but the core purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any context such as 'use this to find parts available for a repair' or 'use this to check stock levels.' There are no exclusions or references to sibling tools. The only implied usage is searching inventory, which is not explicit enough to route an agent effectively.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_employeesC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'List', which implies a read operation, but it does not disclose pagination behavior, return format, or any side effects. For a simple listing tool, the lack of even basic behavioral context (e.g., that results are paged) leaves the agent guessing about what to expect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the verb and resource. It is not verbose or redundant, and it follows a clear structure. However, it is so terse that it borders on under-specification, but the dimension is about conciseness and structure, not completeness, so it earns a 4.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description should at least hint at pagination and typical response characteristics. The description covers only the basic action and does not mention that results may be paged (as implied by the parameters) or what fields to expect. An agent cannot fully anticipate the tool's behavior from this description alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not mention the 'page' or 'pagesize' parameters at all. Since the schema properties have no descriptions, the agent has no explanation of these parameters beyond their names and defaults. The description fails to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and the resource ('employees/technicians'). It is not a tautology and provides a specific verb+resource pair. However, it does not explicitly differentiate from sibling tools, though no other sibling handles employees, so it is reasonably distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any conditions, exclusions, or contexts where another tool like repairdesk_tickets or repairdesk_customers would be more appropriate. The only intended use is implicit from the resource name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_invoicesC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions three distinct operations but does not disclose side effects, permission requirements, rate limits, or return behavior. The description adds little beyond what the tool name implies, leaving the agent uncertain about operational impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no waste, front-loading the core purpose. However, given the tool's complexity (8 parameters, 3 actions), it is under-specified. The conciseness itself is good, but the structure lacks the necessary detail to be fully valuable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 8 parameters, an action enum, and no output schema, yet the description provides almost no contextual information. It does not explain how the three actions map to the enum values, what each parameter does, or what the response format looks like. This is critically incomplete for an agent to use effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% because the description does not mention any parameters. It fails to explain the meaning of action values, how keyword filtering works, date ranges, pagination, or the purpose of invoice_id. With 8 parameters and zero description coverage, the agent is left to guess parameter usage from the schema alone, which is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb+resource: search, get details, and check amounts specifically for invoices. It distinguishes itself from sibling tools that handle tickets, customers, etc., even though it doesn't name them. The three actions are explicit and convey the tool's scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It does not mention any exclusions, prerequisites, or conditions for using one action over another. An agent must infer usage context from the action enum, but the description provides no such context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_customersB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It states 'look up' which suggests a read-only operation, and the action enum (list/get) reinforces this. However, it does not explicitly confirm read-only semantics, nor does it mention pagination behavior or any permissions required. The description is adequate but leaves room for inference about side effects and constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence with no redundancy. It efficiently conveys the core purpose. However, it could include a bit more detail (e.g., mention of action modes) without losing conciseness, so it's not a perfect 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters including an action enum and pagination, yet the description does not explain the 'action' parameter, the difference between list and get, or how pagination works. It also doesn't address how 'customer_id' vs 'keyword' are used together. For a tool of this complexity, the description is incomplete and requires the agent to infer critical usage details from the schema alone.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is only 20% (only 'keyword' is described). The tool description adds partial meaning by mentioning name, phone, email, or ID, mapping roughly to 'keyword' and 'customer_id', but it does not explain the 'action' parameter, nor does it clarify how 'keyword' and 'customer_id' relate or when to use each. It also omits pagination parameters (page, pagesize). The description adds some value but insufficiently compensates for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (look up) and resource (customers), and lists the search fields. It is distinct from sibling tools (tickets, invoices, etc.) which target different resources. However, it does not mention the 'action' parameter that distinguishes between listing and getting a single customer, leaving some ambiguity about the exact operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for customer lookup but provides no explicit guidance on when to use it versus alternatives. Since siblings are for other resources, the use case is fairly obvious, yet there is no mention of scenarios like retrieving a specific customer vs. browsing all customers, which the action parameter addresses.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_shop_infoB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden of disclosing behavioral traits. It indicates a read operation ('look up') and hints at a dependency for problems ('by device'), but it does not explicitly state that the tool is read-only, has no side effects, requires authentication, or handles pagination. The description lacks essential behavioral context for an agent to call it safely without assumptions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, consisting of a single sentence that front-loads the verb and resource. It efficiently enumerates the lookup types without unnecessary filler. While the list is long, it is still concise and well-structured, earning a high but not perfect score for brevity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, one required, low schema coverage, no output schema), the description is far too sparse. It does not explain return values, pagination behavior, required dependencies (e.g., device_id for problems), or how to effectively use filters. An agent would need to infer much from the sparse schema descriptions, making the tool under-documented.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 43% (only brand, device_id, and device_name have descriptions), so the description needs to compensate by explaining parameter usage. However, it only lists the lookup enum values and mentions 'problems (by device)' without tying that to the device_id parameter. It does not clarify the role of page, pagesize, keyword, or how filters work, leaving a significant semantic gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb ('Look up') and a specific resource ('reference data'), then enumerates the exact entity types supported. This makes the tool's purpose unambiguous and distinguishes it from siblings like repairdesk_tickets or repairdesk_inventory, which handle operational entities rather than lookups.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for retrieving reference data, and its distinct name and content signal a different usage context from the sibling tools. However, there is no explicit statement about when to use this tool versus alternatives, nor exclusions for when not to use it. The guidance is only implied by the tool's nature, not stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_ticketsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It uses read-oriented verbs ('search', 'look up', 'list', 'get') implying a non-destructive operation, but it never explicitly states that no changes are made. It also discloses filtering capabilities but omits details like pagination behavior or any side effects, which is a moderate gap for a read tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no fluff. It front-loads the primary action and then lists the filter options in a logical order. Every word contributes to understanding the tool's purpose and usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with no output schema and moderate complexity (9 parameters, but only one required), the description covers the main actions (list with filters, get by ID). It does not explicitly mention pagination or the response format, but given the simple read nature and the schema covering parameters, it is reasonably complete. A slightly more detailed mention of return contents would push it higher.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 78% (slightly below the high threshold), so the description partially compensates by enumerating filter categories (status, date, keyword, tech) and mentioning 'get by ticket ID'. This maps to several parameters, but it adds little beyond the schema's own descriptions, which already explain each parameter's purpose. Overall, it meets the baseline for a moderately covered schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as a search/lookup for repair tickets with specific verbs (search, look up, list, get) and a resource (repair tickets). It effectively conveys the read-only nature, though it does not explicitly mention the sibling write tool or contrast itself, so it stops short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for reading/searching but provides no explicit guidance on when to use this tool versus alternatives like repairdesk_tickets_write. There are no statements about when not to use it or prerequisites, leaving the agent to infer usage context from the action enum and filter descriptions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• repairdesk_tickets_writeA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the major side effect: status changes may trigger automatic customer SMS/email and lists the specific statuses that notify. It also mandates confirmation. This covers the primary behavioral risk, though it does not mention authentication or rate limits, which is acceptable for this context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and effective: a single sentence states the purpose, then a warning in caps draws attention to the critical side effect and the confirmation requirement. Every sentence earns its place with no fluff, and the critical warning is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex with 15 parameters and multiple actions, but the schema provides detailed per-parameter descriptions and the action enum clarifies usage. The description covers the essential context: what the tool does and the notification risk. It does not need to explain return values since there is no output schema. The description is sufficiently complete for an agent to call it correctly with the schema's help.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 87%, so the schema already explains most parameters well (e.g., note, ticket_id, customer_id). The description's warning about status notifications adds context to the status parameter, but the schema already includes 'MAY TRIGGER NOTIFICATION'. The description does not add significant semantic value beyond the schema; it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's actions: create/update tickets, add notes, assign techs. It names the specific resource (tickets) and verb, distinguishing it from the read-focused sibling repairdesk_tickets. The purpose is unambiguous and not a tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes explicit guidance to always confirm with the user before status changes, which is critical for safe usage. It implies this is the write tool vs. the read tool repairdesk_tickets, though it does not explicitly name alternatives. The confirmation requirement is a clear usage rule.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.