Server Quality Checklist

Tool Scores

• sync_connectionC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool 'syncs' a connection, implying a mutation or update action, but doesn't clarify whether this is a read-only operation, what syncing involves (e.g., data refresh, configuration update), potential side effects, or error conditions. This is a significant gap for a tool with implied mutation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded, with the purpose stated in the first sentence and parameter details following. There's no wasted text, but it could be more structured by explicitly separating purpose from parameters. Overall, it's efficient but slightly under-specified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation tool with no annotations) and the presence of an output schema, the description is minimally adequate. It covers the basic purpose and parameter, but lacks behavioral details and usage context. The output schema may help, but the description doesn't reference it, leaving gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal semantics beyond the input schema, specifying that 'id' is the 'id of the connection'. With 0% schema description coverage, this provides some context, but it doesn't explain the format or source of the ID (e.g., from list_connections), leaving the agent with incomplete guidance. The baseline is 3 due to the single parameter being straightforward.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'syncs a fivetran connection by ID', which provides a clear verb ('sync') and resource ('fivetran connection'). However, it doesn't differentiate from sibling tools like 'list_connections' or 'invite_fivetran_user', leaving the specific role of 'sync' somewhat vague without further context about what syncing entails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description lacks any mention of prerequisites, such as needing an existing connection ID, or context for when syncing is appropriate compared to listing connections or inviting users. This leaves the agent without direction on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_connectionsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context about authentication requirements ('auth_token must be set'), API interaction ('making a GET request to the Fivetran API'), and error handling ('does not handle exceptions'). However, it doesn't mention rate limits, pagination, or what happens when no connections exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (description, returns, note) and avoids redundancy. However, the 'Returns' section could be more concise, and some information about authentication could be integrated more smoothly. Overall, most sentences earn their place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 0 parameters, an output schema exists, and no annotations, the description provides good contextual completeness. It explains what the tool does, authentication requirements, return format, and exception handling limitations. The main gap is lack of guidance on when to use this versus other tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the baseline is 4. The description appropriately doesn't discuss parameters since none exist, focusing instead on authentication and behavioral aspects.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'listing all connections' IDs in the Fivetran account' and 'retrieves all connection IDs from the Fivetran account'. It specifies the verb (list/retrieve) and resource (connection IDs), though it doesn't explicitly differentiate from sibling tools like 'sync_connection' which likely performs a different action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions authentication requirements and exception handling, but doesn't indicate scenarios where this tool is preferred over other tools or when it should be avoided. No explicit when/when-not/alternatives are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• invite_fivetran_userA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that the tool 'sends an invitation' (implying a mutation/write operation) and requires authentication via AUTH_TOKEN. However, it doesn't mention potential side effects (e.g., email notifications, user creation), rate limits, error conditions, or what happens if the user already exists. It adds some context but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, parameters, returns, example, note). It's appropriately sized but could be more front-loaded; the core purpose is stated upfront, but some details like the parameter count could be omitted as they're redundant. Most sentences earn their place by adding value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (returns Dict[Any, Any]), the description doesn't need to detail return values. It covers authentication requirements and parameter semantics well. However, for a mutation tool with no annotations, it could better explain behavioral aspects like what 'inviting' entails operationally. The example adds practical value, making it reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% description coverage. It explains each parameter's purpose and constraints: email must be valid format, given_name and family_name cannot be empty, phone should include country code. This fully compensates for the schema's lack of descriptions, providing clear semantics for all 4 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'sends an invitation to a specified email address to join a Fivetran account.' It specifies the verb ('sends an invitation') and resource ('users to Fivetran'), but doesn't explicitly differentiate from sibling tools like list_connections or sync_connection, which have different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions the tool requires an AUTH_TOKEN environment variable, but doesn't specify prerequisites like account permissions or when to choose this over other user management methods. No explicit when/when-not or alternative tools are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.