Server Quality Checklist

Tool Scores

• search_suiteA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It reveals that 'Returns all matches so the caller can disambiguate if >1', which is important. However, it does not describe the return structure, error behavior, or required permissions, leaving gaps for a tool with no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences front-load the purpose, provide usage context, and note a key behavioral detail. Every sentence contributes value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose and usage, and mentions that it returns matches and yields a suite_id. However, with no output schema and no annotations, it stops short of fully describing the return format and edge cases, making it slightly incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both project_id and name_or_prefix well described. The description adds no parameter-specific meaning beyond the schema, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the verb 'Resolve' and resource 'suite name or prefix' with an output of 'suite reference'. It distinguishes itself from siblings by positioning as a prerequisite lookup for create_test_case and list_test_cases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs when to use the tool: 'Call this before create_test_case or list_test_cases to get the suite_id.' This is clear, though it does not mention when not to use it or direct alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_test_caseA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. 'Get' implies a read-only operation, and it mentions returning full detail with all steps. However, it does not explicitly state safety (no side effects) or error behavior, leaving some ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first states the primary function, second provides a concrete usage guideline. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple single-parameter tool with no output schema, the description adequately explains what is returned (full detail, all steps), how to identify the test case, and when to use it. No additional documentation is necessary.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, providing full parameter documentation. The description adds a usage example ("APA-3") and implies the parameter's role in retrieval, but does not significantly extend beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gets full detail of a test case by display_id, including all steps. It distinguishes itself from siblings like update_test_case by implying a read-only retrieval operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using this tool before update_test_case to review current state, giving a clear use case. It does not mention exclusions for other siblings like list_test_cases, but the context is specific.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_test_caseA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing the dry-run validation vs. commit behavior, the requirement for human approval, and that display_id is assigned by TCM on commit. It stops short of describing validation failures or side effects, but the key behaviors are transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences and front-loaded with the primary verb and resource. The REQUIRED dry-run workflow is stated in a deliberate, direct manner without any extraneous content. Every sentence contributes essential operational guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 10-parameter tool with 3 required parameters and no output schema, the description covers the critical workflow and key constraints (steps required, dry_run first, display_id on commit). It doesn't explain the return value or failure modes, but the rich schema and concise workflow guidance make the tool usable; a small gap remains regarding what the dry-run summary contains and follow-up actions on validation failure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema fully documents all 10 parameters. The description adds no new parameter details beyond reinforcing that steps are required (already in schema) and noting that display_id is assigned on commit (a behavioral outcome, not a parameter semantic). Baseline 3 is appropriate given complete schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Create a test case with steps,' which uses a specific verb and resource, clearly distinguishing this creation tool from sibling tools like update_test_case, get_test_case, and search_suite. The added step requirement and commit workflow reinforce its unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit instructions on when to use dry_run true (first), when to get human approval, and when to commit with dry_run false/omitted. However, it does not explicitly compare against sibling tools or state when not to use this tool at all, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_test_casesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does disclose the lightweight projection fields and the default/max limit, which are useful behavioral insights. It does not explicitly state the return shape or read-only nature, but the verb 'List' and the projection list make these reasonably clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences and front-loads the core purpose. Every sentence adds functional value: projection, filters, and limit behavior. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list tool with 4 optional parameters and no output schema, the description covers the essentials: what is returned (lightweight fields), how to filter, and result limits. It does not mention sorting or pagination, but given the max limit of 200, this is likely acceptable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters, so the baseline is 3. The description's mention of 'filterable by project, suite, or search term' provides a useful summary but does not add new semantics beyond what the schema already states (e.g., ilike matching, UUIDs).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource ('List test cases') and clearly distinguishes this tool from siblings by mentioning 'lightweight projection' and the specific fields returned. This differentiates it from get_test_case (single, detailed) and search_suite (searches suites).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool—when you need a filtered, lightweight list of test cases—and mentions the filterable dimensions (project, suite, search). However, it does not explicitly name alternative tools or state exclusions, such as 'for full details use get_test_case', so it is strong but not a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_test_caseA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It discloses a critical destructive behavior: 'ALL existing steps are wiped and replaced (full-replace — not partial)' and mandates a dry-run first for safety. This goes beyond a generic update description and alerts the agent to the mutating consequence.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, tightly packed. The purpose is front-loaded, followed by the two most critical safety warnings. Every sentence earns its place with zero filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 10 parameters, no annotations, and no output schema, the description covers the essential workflow and the most dangerous edge case (steps replacement). It doesn't mention that tags and platform_tags also replace-all, but the schema already documents those semantics, so the description is not incomplete. It could briefly note other replace-all fields but is otherwise adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, so baseline is 3. The description adds value by emphasizing 'by display_id', repeating the steps full-replace warning, and explaining that dry-run shows 'the full before/after diff including steps' — a detail not in the schema. This is a modest but meaningful addition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Partially update an existing test case by display_id', using a specific verb ('update') and resource ('test case') and key parameter. It distinguishes from sibling tools like create_test_case (create) and get_test_case (read), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly requires 'call with dry_run: true first' and 'Only call with dry_run: false after explicit human approval', providing a clear mandatory workflow. It also clarifies the full-replace behavior for steps, giving the agent precise conditions for safe use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.