Server Quality Checklist

Tool Scores

• shopify_admin_graphql_preview_mutationA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the key non-execution behavior and lists the output (variables, scopes, IDs, change summary, confirmation token). This goes beyond a simple 'validates' by explaining what the user can expect, though it omits details like error handling or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly worded sentence. It front-loads the most critical information (validates, does not execute) and lists the key return values efficiently. There is no wasted verbiage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and absence of output schema or annotations, the description covers the essential aspects: what it does, what it returns, and the confirmation token. It does not explain how the token should be used for a subsequent execution, but that is outside the immediate scope of a preview tool. Overall, it provides enough context for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero description coverage for its three parameters. The description does not explain the query, variables, or operationName parameters beyond the tool's overall purpose. It does not add syntax, format, or usage details for these inputs, leaving the agent to infer from the parameter names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates but does not execute a Shopify mutation, using a specific verb ('validates') and resource ('Shopify mutation'). It also differentiates from the sibling execute_mutation by explicitly noting it does not execute, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: when you need to validate or preview a mutation without executing it. The contrast with execute_mutation is implicit rather than explicit, but the context is clear enough that an agent would know to use this for dry-run validation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• shopify_admin_graphql_execute_mutationA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare destructiveHint=true and readOnlyHint=false, so the mutation behavior is known. The description adds useful context about the one-time token and environment gating, but does not elaborate on side effects or error behavior beyond what annotations imply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences that front-load the core purpose and immediately follow with the critical prerequisite. Every part is necessary and no irrelevant details are included.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single parameter, sibling tools, and annotations, the description covers the essential workflow and gating requirement. It does not explain response payloads, but this is acceptable given the execution-oriented nature and lack of output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only provides format constraints for confirmationToken, but the description explains its meaning as a one-time token from a previously previewed mutation. This adds semantic clarity that compensates for the 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb-resource pair: 'Executes a previously previewed mutation' and identifies the one-time confirmation token as the key input. This clearly distinguishes it from the sibling preview mutation and query tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states that this tool is for executing a mutation that was already previewed, implying that preview_mutation should be used first. It also mentions the SHOPIFY_ENABLE_MUTATIONS=true prerequisite, giving clear context for when the tool is available.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• shopify_admin_graphql_queryA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the operation is read-only, cannot override the store/API version, and that pagination is bounded and requires explicit connection metadata. These are meaningful behavioral constraints. However, it omits response format, error handling, and authentication expectations, which would have made it more transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core action and scope. Every sentence provides useful information: the first states what and how, the second adds limitations and pagination requirements. No fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 parameters, nested objects, no output schema) and lack of annotations, the description covers the key constraints: read-only, single query, store/API fixed, pagination boundary. It falls slightly short by not mentioning expected return shape or error behavior, but for a raw GraphQL execution tool the most critical contexts are addressed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50%, so the description must compensate. It explicitly associates 'query' with the GraphQL operation and adds the crucial constraint that pagination requires explicit connection metadata. However, it does not clarify the 'variables' or 'operationName' parameters, leaving them to the schema. Thus it adds some value but does not fully compensate for the coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Executes'), a clear resource ('Shopify Admin GraphQL query'), and a key scope ('read-only', 'against the configured store'). It clearly distinguishes itself from sibling mutation tools by emphasizing read-only, and from product_counts by being a general query tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it is read-only, executes a single query, and the store/API version cannot be overridden. While it doesn't explicitly name alternative tools, the read-only label implicitly tells the agent not to use this for mutations, and the mention of 'one query' sets expectations. No exclusions are stated, but the context is useful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• shopify_admin_product_countsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the underlying method (productsCount) and the output categories, which helps. However, with no annotations, it does not mention whether this is read-only, any rate limits, or how counts are calculated. It leaves some behavioral aspects unstated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that efficiently conveys the method and the list of counts returned. It contains no filler and is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple, no-parameter tool, the description is complete: it specifies what it does, how it does it, and what the output includes. The absence of an output schema is offset by the explicit enumeration of counts, and no further context seems necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema reflects that. Baseline for zero parameters is 4; the description does not need to add parameter semantics, and it doesn't, which is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: returning total, active, draft, archived, and unlisted product counts using the productsCount method. It explicitly distinguishes itself from paginated product queries and from generic GraphQL operations, making it unique among siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Uses productsCount—not paginated products' implies that for product count needs, this tool should be used instead of paginating through products via GraphQL. This gives clear context, though it stops short of explicitly naming when-not-to-use or listing alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.