Server Quality Checklist

Tool Scores

• codemagic_cancel_buildA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It only states the action (cancel) without detailing effects, permission requirements, error cases, or whether it is irreversible.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise with no extraneous text. One sentence plus parameter list fits the simple tool perfectly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core action but lacks context on return values, side effects, or edge cases. Since an output schema exists, the missing return explanation is acceptable, but behavioral nuances are omitted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must explain parameters. It provides a brief but clear explanation that build_id is 'the id of the build to cancel', adding meaning beyond the schema's title.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the verb 'Cancel' and resource 'running or queued build', clearly distinguishing it from sibling tools that get, list, or start builds.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use versus alternatives, such as when a build should not be cancelled or prerequisites. The tool has siblings but no comparative instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codemagic_list_appsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It discloses that the tool reads and returns list of apps with specific fields. It does not mention authentication, rate limits, pagination, or ordering, which are relevant for a list operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core action, and no unnecessary words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters and an output schema, the description adequately explains what is returned and hints at usage. Could mention lack of pagination or filtering, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema coverage is 100% (trivial). The baseline is 3. The description adds value by explaining the return structure and usage, but not directly about parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists all applications connected to the account and specifies the returned fields. However, it does not explicitly differentiate from siblings like get_app or list_builds, though the distinction is implied.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description suggests using returned ids with start_build but does not provide guidance on when to use this tool versus alternatives like get_app or list_builds. The usage context is clear but not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codemagic_list_buildsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description bears full burden for behavioral disclosure. It states the core behavior (list with filters) but omits details like whether unfiltered calls return all builds, pagination, sorting, or authentication requirements. The behavior is minimally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise: a one-sentence summary followed by a bullet list of arguments. It is front-loaded with the main action. Every sentence is necessary and adds value. No extraneous text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the basic listing functionality with optional filters. An output schema exists, so return format is assumed documented. However, it lacks context on default behavior (e.g., returns all builds when no filters), potential pagination, or sorting. For a simple tool, it is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains each parameter's purpose concisely and provides an example for 'status' (e.g., 'building'). This adds meaning beyond the schema's title fields. However, it does not enumerate all possible values or format constraints, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'List builds' with optional filters. The verb 'list' and resource 'builds' specify the action and object. It distinguishes from sibling tools like 'codemagic_get_build' (single build) and 'codemagic_list_apps' (different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not specify that this should be used for querying multiple builds or that 'codemagic_get_build' retrieves a single build. No exclusions or recommendations are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codemagic_get_appA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It describes a read operation ('Get details') without mentioning side effects, but does not explicitly state read-only behavior, authentication needs, or other constraints. The behavior is standard for a 'get' tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with three short lines covering purpose, return categories, and parameter guidance. It is front-loaded with the main purpose and avoids unnecessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one parameter and an output schema, so the description adequately covers what is returned (categories) and sourcing the app_id. It could be more explicit about output structure, but the output schema covers that. Overall sufficient for the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description provides meaningful context for the only parameter app_id: 'The Codemagic application id (from codemagic_list_apps)'. This tells the agent where to obtain the value, adding value beyond the schema's string type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get details for a single Codemagic application' with a specific verb and resource, and lists the returned information (repository, branches, workflows, builds). It distinguishes from siblings like codemagic_list_apps (listing all) and codemagic_get_build (specific build).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after codemagic_list_apps via the arg note 'from codemagic_list_apps', but does not explicitly state when to use or not use this tool, nor contrast with siblings. No alternative tool names are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codemagic_start_buildA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden. It states the tool triggers a build (mutation) and returns a buildId, which is helpful. However, it omits details like idempotency, rate limits, or that builds may be queued, leaving gaps for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: a one-line summary followed by bullet-point parameter explanations. Every sentence provides necessary information without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists (mentioned in context), the description covers the return value (buildId) and refers to a polling tool. It is fairly complete, though it could mention potential error conditions or build queue behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds significant meaning via the Args section: it explains each parameter (app_id, workflow_id, branch, tag) and their mutual exclusivity. This goes well beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Trigger a new build for an app/workflow on a branch or tag.' This is a specific verb+resource combination that distinguishes it from sibling tools like codemagic_cancel_build (cancel) and codemagic_get_build (get).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context: 'Provide exactly one of branch or tag.' and 'Returns the new buildId, which you can poll with codemagic_get_build.' It implies when to use (starting a build) but does not explicitly state when not to use or list alternatives, though siblings cover that.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codemagic_get_buildA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the possible status values and mentions that 'finished' includes success/failed result. This is sufficient for a read-only operation, though it could elaborate on response format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the purpose, then lists status values and parameter details. Every sentence adds value, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter, no nested objects). The description covers purpose, status values, and parameter source. It does not detail the response, but an output schema exists (context signal). For a single get, it is adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only has build_id as a string. The description adds essential context: 'The build id returned by codemagic_start_build or listed by codemagic_list_builds.' With 0% schema coverage, this fully compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get the status and details of a single build,' specifying a verb and resource. It lists possible status values, distinguishing it from sibling tools like codemagic_list_builds (lists builds) and codemagic_start_build (starts builds).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains that the tool requires a build_id, which comes from codemagic_start_build or codemagic_list_builds, providing context on when to use it. It does not explicitly state when not to use, but the sibling tools imply alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.