Server Quality Checklist

Tool Scores

• get_roles_for_customerC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves roles but doesn't cover critical aspects like authentication requirements, rate limits, error handling, or the format of returned data (e.g., list of role objects). This leaves significant gaps in understanding how the tool behaves in practice.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence that efficiently conveys the core purpose without unnecessary words. It is front-loaded and wastes no space, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a tool that likely returns structured data. It doesn't explain what 'roles' entail (e.g., permissions, names), how results are formatted, or any limitations (e.g., pagination). For a retrieval tool with zero structured context, more detail is needed to guide effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the parameter 'customerId' documented as optional and defaulting to auth context. The description adds no additional meaning beyond this, as it only repeats 'customer ID' without elaborating on format, validation, or implications of omitting it. Baseline 3 is appropriate since the schema adequately covers the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('all roles for a specific customer ID'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'get_role_id_by_name' or 'get_teams_for_customer', which also retrieve customer-related data but focus on different resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_role_id_by_name' (for fetching a single role by name) or 'get_teams_for_customer' (for teams data), leaving the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_teams_for_customerC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While 'Get all teams' implies a read operation, the description doesn't address important behavioral aspects like authentication requirements, rate limits, error conditions, or what 'all teams' means in terms of pagination or filtering. It's minimal and lacks operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any unnecessary words. It's appropriately sized and front-loaded, making it easy to understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'teams' are in this context, what data is returned, or any behavioral traits. For a tool that likely returns multiple items, more context about the response format or limitations would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the single parameter 'customerId' fully documented in the schema as optional with a default from auth. The description doesn't add any additional meaning beyond what's already in the schema, so it meets the baseline of 3 for adequate but not additive parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get all teams') and resource ('for a specific customer ID'), providing a specific verb+resource combination. However, it doesn't differentiate this tool from its sibling 'get_roles_for_customer', which has a similar structure but returns roles instead of teams.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided about when to use this tool versus alternatives. The description doesn't mention the sibling tools or explain scenarios where this tool is appropriate versus other customer-related operations. It simply states what the tool does without context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• invite_user_for_customerC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions that the user receives an email invitation and gets onboarded, which hints at a mutation with side effects, but does not cover critical aspects like required permissions, error handling, or whether the operation is idempotent. This leaves significant gaps for a tool that creates users.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with two sentences that directly explain the tool's purpose and outcome. There is no unnecessary information, and each sentence adds value by clarifying the action and its effects, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a user invitation tool with no annotations and no output schema, the description is incomplete. It lacks details on behavioral traits (e.g., permissions, errors), return values, and usage guidelines. For a mutation tool that likely involves system changes and email notifications, more context is needed to adequately guide an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, providing clear documentation for all parameters (email, roleId, teamIds, status). The description adds no additional parameter semantics beyond what the schema states, such as explaining the meaning of 'status' values or constraints on teamIds. With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Invite a user for a customer') and the outcome ('creating a user for the customer', 'user receives an email invitation and then gets onboarded'). It specifies the verb (invite/create) and resource (user for a customer), but does not explicitly differentiate from sibling tools like get_role_id_by_name, which are read-only operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions the action but does not specify prerequisites, conditions, or exclusions, such as when a user already exists or what happens if the email is invalid. Without such context, the agent lacks clear usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_role_id_by_nameA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the search behavior ('searches through all roles') and conditional return ('returns the role ID if found'), which is useful. However, it doesn't mention error handling (e.g., if role not found), authentication needs, or rate limits, leaving gaps for a lookup tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with zero waste: the first states the purpose, the second clarifies the search scope and conditional outcome. It's front-loaded and appropriately sized for a simple lookup tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with no annotations and no output schema, the description is adequate but has gaps. It covers the purpose and basic behavior, but lacks details on error cases, return format (beyond 'role ID'), or performance considerations, which could be helpful given the search scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds no additional meaning beyond what the schema provides (e.g., no examples beyond 'Admin', no clarification on 'customerId' default behavior). Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get role ID by role name'), resource ('role'), and scope ('searches through all roles for the customer'). It distinguishes from siblings like 'get_roles_for_customer' (which lists roles) and 'get_teams_for_customer' (different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need to find a role ID by name, but doesn't explicitly state when to use this tool versus alternatives like 'get_roles_for_customer' (which returns all roles). No guidance on prerequisites or exclusions is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.