Server Quality Checklist

Tool Scores

• customer_moneyA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the content of the response (balance, invoices, etc.) and that it comes 'in one response', but does not explicitly state read-only behavior, error handling, or any side effects. It adds some context beyond a bare description, but not rich behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences and every word earns its place. The first sentence states the tool's output and scope; the second gives usage guidance and prerequisite. It is concise, front-loaded, and free of fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (per context signals), the description does not need to explain return values, and it does cover the main use case and key parameter. However, the lack of any explanation for 'limit' leaves a small but real completeness gap, so not a perfect 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It only implicitly explains customer_id ('Needs a customer id'), but the 'limit' parameter is left completely unexplained. Without any schema descriptions or additional context, an agent cannot know what 'limit' controls (e.g., number of invoices or transactions). This is a notable gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a consolidated set of financial data (balance, estimates, invoices, payments) for a single customer, using specific nouns and a clear scope ('for one customer'). This distinguishes it from siblings like customer_overview or find_customer, which likely cover broader or different data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('what do they owe' or billing-history questions) and mentions the prerequisite of a customer ID, pointing to find_customer as the source. However, it does not explicitly mention when not to use it or how it compares to customer_overview, so it lacks full when-not coverage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_by_external_idA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It discloses a key prerequisite (ST_APP_GUID) and clarifies the matching mechanism ('externalData key/value pair'), which is useful. However, it does not describe return behavior, edge cases (e.g., not found, multiple matches), or potential side effects, leaving notable gaps for a no-annotation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact: two sentences, no fluff, and it front-loads the core purpose. The prerequisite and use-case are included without redundancy, making every sentence earn its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with only two parameters and an output schema, the description provides adequate context: the lookup mechanism, the source of the external id, and a required app GUID. It does not explain the response when no customer is found, but the output schema likely covers return structure. Slightly more context about limits or filtering could improve completeness, so it stops short of a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for the minimal schema. It does add meaning by explaining that 'key' and 'value' refer to an externalData pair and that the value is another system's id. Yet it lacks specifics such as accepted key formats, value types, or examples, so it does not fully bridge the coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: 'Find a customer by an externalData key/value pair.' It also explains that the value is 'another system's id', which distinguishes it from the sibling 'find_customer' tool by indicating it searches through external data rather than standard customer fields.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: 'Use when you hold an id from a CRM synced into ServiceTitan.' This tells the agent when the tool is appropriate, though it does not explicitly mention when not to use it or name alternative tools. This is clear guidance but lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_customerA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It explains the one-match versus several-match result behavior and explicitly notes the email lookup limitation. However, it does not mention what happens when there are zero matches or whether any permissions are needed, leaving minor gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise, using only three short lines to cover purpose, search keys, result behavior, and a limitation. Every sentence earns its place, and the most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the relatively simple tool, an output schema, and clear sibling context, the description is largely complete. It covers the main behavioral nuance (single vs. multiple matches) and a critical limitation, though it omits the no-match case and the exact effect of the limit parameter.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no descriptions, but the description gives useful meaning to the 'query' parameter by listing valid search fields (name, phone, external id). The 'limit' parameter is not explained beyond its schema default, so the description only partially compensates for the 0% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds a customer by name, phone, or external id, making the purpose and resource explicit. It also distinguishes itself from siblings like find_by_external_id by noting that no ServiceTitan id is needed, and clarifies it is customer-focused rather than job-focused.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys when to use this tool: when you have a name, phone, or external id rather than a ServiceTitan id. It also provides an explicit exclusion ('Email lookup is not supported'), which helps steer users away from unsupported inputs, though it does not name specific alternative tools for those cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_jobsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure. It goes beyond a simple statement by revealing that invalid status values return the valid list rather than an empty result—a non-obvious failure mode—and that dates use ISO format. The inlined next appointment also hints at the return structure. This is substantial added context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences each add unique value: purpose and a key feature, date format, and an exclusion with an alternative. The information is front-loaded and there is no fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, the description is not required to explain return values. It covers purpose, date format, an important edge case, and an explicit alternative. For a 5-parameter tool with no annotations, it could further clarify filters like customer_id or pagination, but the existing guidance is sufficiently robust for an agent to avoid major pitfalls.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only partially does. It clarifies date format for since/until and mentions status behavior, but leaves customer_id and limit semantically implicit. The parameter names are self-explanatory, yet the description could do more to define filtering semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource ('List jobs') and a distinguishing feature ('with the next appointment inlined'), making the tool's purpose immediately clear. It also explicitly differentiates from the sibling tool find_customer, reducing ambiguity about scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear when-not case: 'Not for finding a customer — use find_customer.' This directs the agent away from a likely misuse. It also notes a behavioral edge case (invalid status returns a valid list) that affects usage, though it does not enumerate all sibling distinctions (e.g., job_detail).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• job_detailA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of disclosing behavior. It adds useful context by noting that appointments and technicians are included, but it does not explicitly state that the operation is read-only nor mention any potential side effects or prerequisites. For a simple retrieval, this is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: the first sentence states the core purpose, and the second sentence gives usage guidance without redundancy. Every word adds value, and it is front-loaded with the primary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (single parameter, no enums) and the existence of an output schema that describes return values, the description covers the essential context. It explains what the tool does and when to use an alternative. It lacks information on error handling or edge cases, but for a straightforward retrieval tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate for parameter documentation. The only parameter, job_id, is self-explanatory by name and type, and the description implies its role by stating it returns 'one job'. However, it does not provide additional detail such as acceptable formats or source of the ID. The meaning is clear enough, but more explicit description would be better.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a single job with its appointments and assigned technicians. It distinguishes this from the sibling tool find_jobs, which handles multiple jobs, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The second sentence explicitly provides guidance on when to use find_jobs instead, listing examples like a customer's list, date range, or status. This effectively differentiates the tool from its sibling and prevents misuse.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• customer_overviewA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the required input type and lists the returned data categories. It does not explicitly state whether the tool is read-only or what happens if the ID does not exist, but the described behavior ('full card') implies a read operation. Slightly more detail on error handling would push it higher.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, perfectly front-loaded: the first sentence states the tool's purpose and contents; the second gives prerequisite and alternative. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with an output schema, the description covers all necessary context: what data is returned, what input is needed, and when to use a different tool. The presence of an output schema means return values need not be spelled out. This description is complete and self-sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter description), so the description must compensate. It adds key meaning by specifying the customer_id must be a 'ServiceTitan customer id', which distinguishes it from other possible identifiers. It stops short of providing examples or format hints, but the single parameter is adequately clarified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific function: returns a 'full card' for one customer, enumerating contents (contacts, locations, 5 recent jobs, external ids). This is distinct from sibling tools like find_customer (search by name/phone) and job_detail (single job focus).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance: 'Needs a ServiceTitan customer id' and gives a direct alternative: 'use find_customer if you only have a name or phone.' This tells the agent exactly when to choose this tool over a sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.