Allure TestOps MCP

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v1.2.6
Disambiguation4/5
Most tools have clearly distinct purposes, with detailed descriptions that differentiate similar operations like 'get_defect_test_cases' vs 'get_defect_test_results'. Potential confusion exists between 'get_test_case_steps' and 'get_test_case_scenario', but descriptions clarify they return different data. Overall, agents can distinguish tools reliably.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern in snake_case (e.g., 'create_test_case', 'list_launches', 'get_defect_test_results'). Verbs like 'add', 'apply', 'archive', 'create', 'get', 'list', 'update' are used systematically, with no mixing of conventions or styles.
Tool Count2/5
With 100 tools, the server far exceeds the typical well-scoped range. While Allure TestOps is a complex platform requiring many operations, this high count overwhelms agents and increases the risk of misselection. Many tools are highly specialized, but a leaner set covering core workflows would be more coherent.
Completeness4/5
The tool set covers CRUD for main entities (test cases, launches, defects, shared steps, dashboards, test plans) and includes specialized operations like defect matchers, trend analysis, and environment variable queries. Minor gaps exist (e.g., no reorder steps, no bulk update for test case fields), but most agent workflows are supported.
Average 3.3/5 across 92 of 100 tools scored. Lowest: 2.1/5.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 8 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI is failing
This repository is licensed under MIT License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior (e.g., idempotency, side effects, output nature). It only states 'Get launch statistics' with no further details, leaving critical behavioral aspects unknown.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness2/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is only three words, which is overly terse. While concise, it sacrifices necessary information and does not earn its place by providing enough context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of output schema and the presence of many similar sibling tools, the description is incomplete. It does not explain what return values to expect, leaving users without sufficient context to use the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description provides good guidance ('Must be a number (integer), not a string'). The tool description adds nothing about parameters, but the schema already handles it, resulting in a baseline score of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Get launch statistics' clearly states a verb and resource, but it is vague: it does not specify what statistics are included, making it hard to distinguish from similar tools like 'get_launch' or 'get_launch_progress'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. Among many sibling tools related to launches, there is no mention of context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must fully disclose behavioral traits. The minimal description 'List test cases for a project' gives no information about pagination, sorting, default behavior when no parameters are provided, or any side effects. This is insufficient for an agent to understand the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness2/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (one sentence) but at the cost of essential information. For a tool with 7 parameters and many siblings, a single sentence fails to provide adequate guidance. Conciseness should not compromise completeness; this is under-specification, not conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (7 parameters, no output schema, no annotations), the description is grossly incomplete. It does not mention pagination behavior, default sorting, or how to effectively use the parameters. The agent has insufficient information to correctly invoke this tool without relying on external knowledge or trial and error.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 71% of parameters with descriptions, but the tool description adds no additional meaning beyond the schema. For example, it does not clarify how 'search' interacts with other parameters or suggest typical combinations. The description should have added context like 'Use search for text filtering across test case names' to compensate for low-enrichment.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'List test cases for a project' clearly states the action (list) and resource (test cases), and implies scope (for a project). It is specific and uses a verb+resource structure. However, it does not differentiate from sibling tools like 'search_test_cases' or 'list_test_results', which could cause confusion about when to use this tool.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives such as 'search_test_cases'. It lacks any explanation of prerequisites, typical use cases, or exclusions, leaving the agent to infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. The description only states a mutation, but does not disclose effects on existing tags, authorization needs, or other behavioral traits essential for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely concise at one sentence, but lacks necessary detail. It is front-loaded but incomplete, earning a middle score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and low schema coverage, the description should provide more context on behavior and return values. It fails to do so.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 50% (testCaseId described, payload not). The description adds no meaning beyond the schema, failing to explain the payload structure or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Set tags for a test case' clearly conveys the action and resource. It distinguishes from siblings like 'add' and 'remove' by implying replacement, though not explicitly stating overwrite behavior.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like add_test_case_tags_bulk or remove_test_case_tags_bulk. The description does not specify whether it replaces or merges tags.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must disclose behavioral traits. It only says 'Update' but does not explain what fields can be updated, whether the operation is idempotent, side effects, authentication needs, or response behavior. This is insufficient for an agent to understand the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness2/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise at 4 words, but it sacrifices useful information. It is not appropriately sized given the tool's complexity; every sentence should earn its place, but here there is only one vague sentence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that this is a mutation tool with two parameters (one undocumented), no output schema, and no annotations, the description is severely incomplete. It fails to provide essential context about the payload structure, allowed operations, or what the tool returns.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 50% coverage (only 'id' has a description). The description adds no parameter explanations beyond the schema. The 'payload' parameter is an arbitrary object with no constraints or examples, leaving the agent without guidance on valid values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb (update) and resource (launch), making the basic purpose unambiguous. However, it does not distinguish from sibling tools like close_launch or delete_launch, though the name itself implies modification.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives, prerequisites, or when not to use it. The description lacks any contextual usage advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description lacks any behavioral details beyond the redundant 'list launches'. It does not mention pagination, sorting, or that the tool might return all launches for a project. With no annotations, the description should disclose side effects, limits, or data scope.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence, but it is underspecified for a tool with 7 parameters. It could be expanded with key details without becoming verbose. The structure is front-loaded but insufficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity (7 parameters, no output schema, no annotations), the description is severely incomplete. It fails to explain how to use the tool effectively, such as which parameters are needed, how to combine projectId and projectName, or what the response contains.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 71%, but the description adds no meaning to the parameters. Parameters like 'search' and 'sort' lack descriptions in the schema, and the description does not clarify them. The baseline would be higher if coverage were >80%, but here it is not compensated.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('list launches') and the resource ('for a project'), but it does not differentiate from the sibling tool 'search_launches', which may offer filtering. A more specific verb like 'list all' or 'list without filtering' would improve clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'search_launches' or 'get_launch'. The description is silent on prerequisites, required parameters, or context for usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior. It only states the action without mentioning idempotency, side effects, or prerequisites. Agents cannot understand the impact.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence, but it is too brief, lacking necessary structure. It earns a middle score for not being verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the nested payload parameter and absence of output schema, the description fails to provide sufficient context for correct tool invocation. It is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds no parameter meaning. The schema describes 'id' but 'payload' is vague (additionalProperties: true). The tool description should clarify the expected payload structure, which it does not.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'add' and resource 'test cases to a launch', distinguishing it from siblings like 'add_test_plan_to_launch'. However, it lacks detail on what 'add' entails.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. Sibling tools like 'add_test_plan_to_launch' are similar but not differentiated in usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavior. It only says 'Add', implying mutation, but lacks details on idempotency, overwrites, permissions, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise (one sentence) and front-loaded with the verb. However, it is too brief to be fully informative, missing critical details about the payload.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has a nested object parameter and no output schema, the description is insufficient. It does not explain the expected structure of 'payload' or the result of the operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds no meaning beyond the schema. The 'id' parameter is already well-described in the schema. The 'payload' parameter has no description in the schema or the tool description, leaving its structure and purpose unclear.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Add') and the objects ('test plan', 'launch'), distinguishing it from create_plan or run_plan. However, it does not clarify the nature of the association (e.g., link vs. assign).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs. alternatives (e.g., add_test_cases_to_launch). No prerequisites or context about existing launches/test plans are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It only mentions 'assign' and a payload constraint, but does not explain side effects, idempotency, authorization needs, or whether it replaces existing assignments. This is insufficient for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, making it concise. It conveys the essential action and a key requirement without extraneous text. However, it could be more structured to include usage context or examples without losing brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that the tool has a nested payload object and no output schema, more context is needed. The description does not explain what happens upon successful assignment, error conditions, or the return value, leaving the agent with incomplete information to use the tool safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50% (id described; payload undocumented). The description adds that 'payload must include username', providing a constraint beyond the schema. However, it does not elaborate on the structure of payload or other allowed fields, so meaning is only partially enhanced.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Assign a test result', which gives a verb and resource but is vague. It does not clarify what 'assign' means in this context (e.g., link to a user or launch) or distinguish it from sibling tools like 'create_test_result' or 'update_test_result'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. The only hint is that payload must include username, but there is no explanation of prerequisites, contexts, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior. It only implies a mutable operation but does not explain what happens on creation (e.g., validation, idempotency, required fields in payload, error scenarios, or permissions). This is insufficient for an AI agent to anticipate side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence without extraneous words. It is concise, though it lacks important details. Given the simple schema, it is not overly verbose, but could benefit from a brief example or expected fields while remaining concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a create tool with no output schema and minimal description, the agent lacks context on what a test result is, how the payload should be structured, and what to expect in response. Important details like required payload keys or success conditions are omitted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters1/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% description coverage, meaning the description adds no meaning to the 'payload' parameter. The parameter is an object with additionalProperties: true, but the description does not hint at expected properties or structure, leaving the agent to guess the required format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Create a new test result', specifying the action (create) and resource (test result). It is not a tautology, though it does not differentiate from sibling create tools like create_test_case or create_defect, but the resource name provides sufficient distinction.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidelines are provided on when to use this tool versus alternatives like assign_test_result, update_test_result, or create_launch. There is no mention of prerequisites, conditional usage, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, and the description does not disclose behavioral traits such as read-only nature, authentication requirements, or side effects. The description is too minimal to inform the agent about important behaviors.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, short sentence that is efficient. However, it could be slightly more informative without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description is incomplete. It does not explain what the widget data represents or how it should be used, leaving the agent with insufficient context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter 'id', but the description adds no additional meaning beyond what the schema already provides. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the verb 'Get' and resource 'launch progress widget data', but it is vague and does not differentiate from siblings like 'get_launch' or 'get_widget_data'. It lacks specificity about what constitutes launch progress widget data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. The description does not mention context or provide any usage cues.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description does not disclose any behavioral traits such as authentication needs, rate limits, or side effects. The presence of pagination parameters (page, size) and sorting is not mentioned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, which is concise but lacks structure. It delivers the core purpose but omits critical details that could be presented concisely.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 9 parameters, no output schema, and no annotations, the description is insufficient. It does not explain the combined use of parameters, response format, or the significance of optional fields like project identification.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 56%, but the description adds no parameter-specific meaning. Important parameters like 'query', 'sort', 'global', and 'testCaseSearch' are left unexplained in both schema and description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('List values') and the resource ('custom field'), and implies a project context. It distinguishes from sibling 'list_project_custom_fields' which lists fields themselves, not values.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus other list tools. Does not specify prerequisites or scenarios where listing custom field values is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden but provides no behavioral details—e.g., pagination, ordering, or filtering behavior. The description only states the action without any transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Extremely concise single sentence, front-loaded. However, the brevity sacrifices clarity and detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters, no output schema, and no annotations, the description is severely incomplete. It fails to explain return type, pagination, or the difference from search.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67% (4 of 6 params have descriptions). The description adds no additional parameter meaning beyond what the schema provides, but it does not contradict it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb-resource combination 'List test results' and scopes to 'for a launch.' However, it does not differentiate from sibling tools like 'search_test_results' or 'get_test_result'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'search_test_results'. No information about prerequisites or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description only states 'Set linked issues' without explaining effects like whether existing issues are replaced or appended, authentication requirements, or side effects. This leaves significant gaps in understanding tool behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with a single sentence, but it is overly terse and lacks necessary detail. While there is no wasted text, the brevity undermines its value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of annotations, no output schema, and an under-specified payload parameter, the description fails to provide sufficient contextual completeness. It does not explain the impact of the operation, return values, or structure of the payload.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50%; only testCaseId has a description. The payload parameter is described as an 'array of objects' with no further detail, and the description adds no additional meaning to the parameters. The incomplete schema and lack of elaboration reduce usefulness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Set linked issues for a test case' clearly states the verb and resource, making the tool's purpose understandable. However, it does not differentiate from sibling tools like 'add_test_case_external_links_bulk' or 'link_issue_to_defect', which could cause confusion for the AI agent.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. There is no mention of prerequisites, context, or conditions under which this tool should be preferred.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility for disclosing behavioral traits. It only states the operation ('update') and identification method ('by ID'), but omits critical details such as permission requirements, destructive effects, return value, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single short sentence, which is concise but lacks necessary detail. It front-loads the main purpose but omits information that would improve usability without significant extra length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (2 parameters, one with nested additionalProperties) and no output schema, the description is incomplete. It does not specify permissible fields for the payload, expected behavior on success or failure, or any constraints, leaving significant gaps for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema covers 50% of parameters (only 'id' is described). The description adds minimal value by mentioning 'metadata (name, etc.)', but does not clarify the structure or allowed properties of the 'payload' object, which is a free-form additionalProperties object.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Update' and the resource 'shared step metadata (name, etc.) by ID', providing a clear purpose. However, it does not differentiate from sibling tools like update_dashboard or archive_shared_step, which limits higher scoring.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives, no prerequisites, and no conditions under which it should or should not be used. This leaves the agent without decision support.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description carries the full burden. It only says 'Update an existing test plan' with no details on what fields can be updated, whether it's partial or full replacement, error behavior, or required permissions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence is concise but too brief to be helpful. It could be expanded with parameter details and usage context while still being concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness1/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of the input schema (nested payload object) and many sibling tools, the description is insufficient. No output schema exists, and the description does not guide the agent on how to construct the payload or when to use this tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description does not explain the parameters: 'id' and 'payload'. The payload object is undocumented in both schema and description, leaving the agent to guess its structure. Schema coverage is 50%, but description adds no value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (update) and resource (test plan), and distinguishes it from siblings like create_test_plan and delete_test_plan.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidelines are provided on when to use this tool versus alternatives, such as update_test_case or run_test_plan. No prerequisites or exclusions are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It mentions a default for projectId, but does not disclose mutation effects, required fields, error conditions, or output behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no unnecessary words. Front-loaded the primary action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, no annotations, and a freeform object parameter, the description is insufficient. It does not explain the response, required subfields, or any constraints beyond the default.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It only mentions 'payload.projectId defaults to ALLURE_PROJECT_ID env when omitted,' but payload is a freeform object with no schema hint, leaving the agent without knowledge of other valid fields.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states 'Create a new test plan.' This is a specific verb+resource pair. It distinguishes from sibling tools like update, delete, run, get, and list test plans.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives such as update_test_plan or add_test_plan_to_launch. No prerequisites or context are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as pagination, sorting, or what the history includes. The description is too minimal to inform the agent about side effects or constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness2/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise but omits essential information such as pagination and sorting behavior. It is front-loaded but under-specified, lacking detail expected for a tool with multiple parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of an output schema, the description should hint at the response structure (e.g., list of historical entries). It fails to do so and also does not explain the pagination parameters, leaving the tool's behavior incomplete for the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 75%, so the schema already documents most parameters. The description does not add any parameter-specific meaning beyond the schema, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'history for a test result'. It is specific and distinguishes from siblings that operate on different resources, though it does not contrast with get_test_result.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like get_test_result or get_test_case_history. There is no mention of context or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as side effects, idempotency, permissions, or data limitations. The word 'Get' implies a read-only operation, but no further context is given (e.g., whether the call is expensive or requires project scoping). This is a significant gap with zero annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise, using a single sentence. While there is no wasted text, the brevity leaves out important details (e.g., return format, parameter usage). It is minimally acceptable but could benefit from structured clarification.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 3 parameters, no output schema, and no annotations, the description is too sparse to fully understand the tool's behavior. It does not explain return values, the relationship between projectId and projectName, or how this tool differs from related siblings like 'get_test_case'. The tool is thus incomplete in contextual coverage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with each parameter having clear descriptions (e.g., 'Test case ID. Must be a number (integer), not a string.'). The tool description adds no additional semantic value beyond what the schema already provides, achieving the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'custom field values for a test case', making the tool's purpose apparent. However, it does not explicitly differentiate from siblings like 'get_test_case' (which might include custom fields) or 'list_custom_field_values' (which lists all custom fields), so it lacks precise differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives such as 'set_test_case_custom_fields' or 'get_test_case'. No prerequisites, context, or exclusions are mentioned, requiring the agent to infer usage from the tool name and schema alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It only states 'Get test case overview data,' which implies a read operation but lacks details on what the overview contains, authorization needs, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very short (7 words) and front-loaded, which is efficient. However, it sacrifices necessary detail, making it under-specified. It earns its place but could be expanded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description should provide enough context for the agent to understand the tool's output and behavior. It fails to describe what the overview includes, leaving a significant gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter 'testCaseId' is well-described in the schema (type, format). The description adds no additional meaning beyond the schema, so the baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves 'test case overview data', identifying the verb and resource. However, it does not differentiate from siblings like 'get_test_case' which might return full details, so specificity is limited.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'get_test_case' or 'get_test_case_steps'. The agent is left to infer the purpose without context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must fully describe behavior. It only states the basic operation, omitting details like default ordering, pagination limits, or whether partial project identification is allowed. This is insufficient for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single clear sentence, which is concise, but it sacrifices necessary detail. It should include information about pagination and available filters without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 6 optional parameters, no annotations, and no output schema, the description is far from complete. It does not explain pagination parameters, sort syntax, or how to effectively filter results. An agent would lack critical usage context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 67%, leaving two parameters (search, sort) undocumented. The description adds no parameter-level information beyond what the schema already provides. For high coverage, baseline is 3, and the description does not improve it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'List test plans for a project.' clearly states the action (list) and resource (test plans), and scopes it to a project. It is specific enough to differentiate from single-get or mutation tools, but could explicitly note pagination or filtering capabilities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., get_test_plan for a single plan), or how to choose between projectId and projectName. There is no mention of required parameters (none required) or pagination behavior.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. The description only says 'Update' without disclosing behavioral traits like idempotency, permission requirements, or side effects. For a mutation tool, this is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with no redundancy. Efficiently communicates the core purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and two required parameters (one complex), the description should provide at least a note on return values or payload structure. It fails to complete the context for proper usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: 'id' is described, but 'payload' lacks any description beyond 'additionalProperties: true'. The description does not explain what the payload should contain, leaving ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Update an existing test result,' specifying the verb and resource. It distinguishes from sibling tools like create_test_result or get_test_result, but lacks detail on scope of updates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives, such as resolve_test_result or assign_test_result. No prerequisites or conditions mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description does not disclose key behaviors like whether tags are appended or overwritten, handling of duplicates, or permission requirements. Only states the obvious 'add' action, lacking depth for an AI agent to predict side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, concise but too brief. Could include more detail in a few additional words (e.g., 'appends tags without removing existing ones'). Conciseness is good but at the cost of clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 6 parameters, nested objects, no output schema, and no annotations, the description is incomplete. It does not explain how to use optional parameters (tag vs tags, projectId vs projectName) or expected outcomes, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50% but descriptions only enforce type constraints (e.g., 'Must be a number'). The tool description adds no extra meaning beyond the schema; it does not explain the structure of the 'tag' object or the relationship between projectId/projectName. This does not help an agent choose correct parameter values.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it adds tags to test cases in bulk, distinguishing it from siblings like add_test_case_tags (likely singular) and remove_test_case_tags_bulk. The verb 'Add' and resource 'bulk tags to test cases' is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool over alternatives like set_test_case_tags (which may replace) or add_test_case_tags (singular). Does not explain bulk vs individual usage, leaving the agent to infer from parameter names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavior but only mentions the action 'Get trend'. It does not explain the response format, required parameters, or constraints like needing projectId or projectName, nor how time range filters work.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that front-loads the purpose. No extraneous words, but could be slightly more informative without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 8 parameters, no output schema, and many sibling tools, the description is insufficient. It fails to explain return values, necessary filters, or differentiation from other trend tools like 'get_mute_trend'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema, such as parameter relationships (e.g., projectId vs projectName mutually exclusive) or format examples for RQL filters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'test result statistic trend over time for a project'. It distinguishes from siblings like 'get_mute_trend', though it could be more specific about the type of statistic (e.g., pass/fail rates).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'get_mute_trend' or 'get_launch_statistic'. The description lacks context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description minimally states the action without disclosing behavioral traits such as pagination, response format, or side effects. The read-only nature is implied but not explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with no unnecessary words. However, it is very brief and could be more informative without adding much length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of sibling history tools and no output schema, the description lacks completeness. It does not explain what fields are returned or how the history is structured, limiting contextual understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is high (75%), with clear descriptions for id, page, and size. The description adds no extra meaning beyond the schema, so the baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'test case run history', making the purpose understandable. However, it does not differentiate from sibling tools like 'get_test_result_history', which might cause confusion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. The description lacks any context about prerequisites or scenarios for use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It only restates the action without disclosing read-only nature, permissions, side effects, or response format. This is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (four words) with no wasted text. It is front-loaded and efficient, though slightly under-specified.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Without output schema or annotations, the description lacks completeness. It does not explain what the tool returns (e.g., a string, an object), which is needed for agent decision making.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one well-described parameter (id). The description adds no extra meaning beyond the schema, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Get scenario for a test case,' which matches the tool name and distinguishes it from sibling tools like get_test_case and get_test_case_steps. It is specific enough for an AI agent to understand the resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. The description does not provide context about when 'scenario' is needed compared to other test case retrieval tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description should disclose behavioral traits (e.g., read-only, auth requirements). It only says 'Get data', implying a read operation but not explicitly. Lacks details on rate limits or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, very concise and easy to parse. No wasted words, but could be slightly more structured with additional context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having 5 parameters and no output schema, the description is too minimal. It omits context about time range filtering and the relationship between projectId and projectName, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema already describes all parameters with high coverage (100%). Description adds no additional meaning beyond what is in the schema, so baseline score applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves data for a dashboard widget. It distinguishes from siblings like get_dashboard, but 'data' is vague; it could specify what kind of data.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. Does not explain when not to use it or provide context about filtering or constraints.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Lacks any behavioral context (e.g., pagination, authentication, side effects). Annotations absent, so description fails to compensate.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no wasted words, but misses important details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Four parameters with pagination, no output schema; description lacks pagination details and return structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameter descriptions; description adds no extra meaning beyond that.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description uses specific verb 'List' and resource 'dashboards', scoped to 'for a project'. Clearly distinguishes from sibling list tools for other resources.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use vs alternatives, no prerequisites or exclusions provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states the tool changes a launch from closed to open, but with no annotations, it fails to disclose side effects, permissions, or impact on associated data. Behavioral transparency is minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single clear sentence with no fluff, efficiently conveying the tool's purpose. However, it could be slightly more informative without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, no output schema), the description is minimally sufficient. However, it does not explain what 'reopen' means operationally, leaving some gaps for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already provides 100% coverage with a clear description for the 'id' parameter, including type and format guidance. The description adds no extra meaning, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Reopen a closed launch,' which directly indicates the action and resource among sibling tools like 'close_launch' and 'create_launch,' but lacks any additional detail about what reopening entails.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., when to reopen vs. close or bulk reopen defects). The description gives no context for appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided. Description does not disclose behavioral traits such as side effects (e.g., creating test results), idempotency, or required permissions. The agent gets no insight into what running a test plan entails.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (one sentence) and front-loaded. While not verbose, it is efficient and communicates the core purpose. However, it may be too brief given the gaps in other dimensions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, so the description should explain return values but does not. The 'payload' parameter's purpose and structure are unexplained. The description leaves critical contextual gaps for a tool that executes a test plan.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 50%, with 'id' having a description but 'payload' lacking one. The tool description does not add any parameter information beyond the schema. The schema's description for 'id' is helpful, but the overall parameter semantics are not enhanced.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action (Run) and the resource (test plan) with a specific identifier (by ID). It distinguishes from sibling tools like create_test_plan or get_test_plan.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. Does not mention prerequisites, effects, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior1/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and description lacks any behavioral details such as read-only nature, authentication requirements, error handling, or rate limits. The agent has no clue about the tool's side effects or constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise (one sentence) with no wasted words. However, it could be slightly more structured with key details upfront.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple get-by-ID tool with no output schema, the description is adequate but misses common context like error behavior or return format. Fairly minimal but functional.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear description for the id parameter. The description adds no extra meaning beyond what the schema already provides, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (Get), resource (test case), and identifier (by ID). It distinguishes from sibling tools like list_test_cases and search_test_cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives like list_test_cases or search_test_cases. No mention of exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the burden of behavioral disclosure. It only mentions 'bulk API' without detailing idempotency, error handling, or side effects. The agent lacks crucial behavioral insights.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no filler. Every word adds value relative to the tool's primary action.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has nested objects, no output schema, and no annotations, the description is insufficient. It omits return values, error scenarios, and behavior for bulk operations, leaving gaps for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50% and the description does not clarify the structure of the 'link' or 'links' parameters, which are objects with minimal schema details. The agent cannot determine the required fields for a link.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Add', the resource 'external links to test cases', and the scope 'one or multiple' using a bulk API. It distinguishes this tool from siblings that deal with tags or issues.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives, no prerequisites (e.g., project/test case existence), and no exclusions. The agent is left to infer context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only states 'Copy an existing dashboard', implying a write operation, but lacks details on side effects, permissions, or limitations (e.g., what is copied, whether linked resources are duplicated).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action. It could be more informative without being verbose, but it earns its place with no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (5 parameters, nested object, no output schema, no annotations), the description is insufficient. It does not explain return values, behavioral details of copying, or the purpose of the 'payload' parameter.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 5 parameters are fully described in the schema (100% coverage). The description adds no additional meaning beyond the schema, such as clarifying the 'payload' object's use. Baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Copy' and resource 'dashboard', and specifies that it can be copied to the same or another project. This distinguishes it from sibling tools like create_dashboard (new from scratch) and update_dashboard (modify existing).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives, such as create_dashboard or update_dashboard. The description does not include when-not-to-use or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description carries full burden for disclosing behavioral traits. It only says 'Create a new dashboard' (implying mutation) but omits permissions, side effects, rate limits, or success response. This is insufficient for a create tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, efficient, and front-loaded. It could be slightly expanded to cover context, but as a minimal statement it is concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 5 parameters including nested objects, no output schema, and no annotations, the description is too brief. It doesn't explain project identification (projectId vs projectName), the payload structure, or how the response looks. Lacks completeness for a creation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no parameter-level detail beyond the schema, which already explains each field. No extra value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create') and resource ('new dashboard in a project'), distinguishing it from sibling tools like 'update_dashboard', 'delete_dashboard', and 'list_dashboards'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives, such as when to use 'projectId' vs 'projectName', or prerequisites like existing project. The description only states what it does, not when it's appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full behavioral burden. It only states a deletion operation but offers no details about side effects, safety, permissions, or error behavior. This is insufficient for a destructive action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise (one sentence) and front-loaded with the action and resource. While it could include more context, it avoids unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has a single required parameter and no output schema, the description should provide more context (e.g., success/failure indicators, effects on related data). It lacks completeness for an agent to reliably invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter. The description adds no meaning beyond the schema's description of the 'id' parameter. Baseline score of 3 is appropriate as the schema already provides the necessary semantic detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Delete'), the resource ('a launch'), and the method ('by ID'). It effectively distinguishes this tool from other launch-related tools like close_launch, reopen_launch, or update_launch.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. There's no mention of prerequisites (e.g., launch must exist, cannot be active) or cases where deletion should be avoided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description only says 'delete' without revealing behavioral traits such as irreversibility, cascading effects, permission requirements, or side effects. With no annotations, the description carries the full burden but fails to provide meaningful behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence with no wasted words. However, it sacrifices critical details for brevity, which prevents a perfect score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity of the tool (one required parameter, no output schema, no annotations), the description should at least mention that deletion is permanent or irreversible. It provides no such context, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The tool description does not add any extra meaning to the 'id' parameter beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Delete' and the resource 'a test plan by ID'. It distinguishes from sibling tools like delete_dashboard or delete_launch by specifying the exact resource type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives (e.g., archiving, updating status). It lacks any context about prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and description does not disclose any behavioral traits beyond a basic summary; no mention of side effects, data freshness, or pagination.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence is concise and front-loaded, but could benefit from additional structure or detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 8 parameters and no output schema, the description is too brief to provide complete context for using the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema documentation coverage is 100%, so description adds no extra meaning beyond what the schema already provides; baseline score applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Get', resource 'automation trend chart data', and scope 'for a project (test automation coverage over time)', distinguishing it from sibling tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives; lacks context on prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must disclose behavioral traits. Only states action, no mention of read-only nature, permissions, side effects, or return value characteristics.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, no wasted words. Concise yet clear. Could be slightly expanded for context without harming brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, description should hint at return format (e.g., time series of counts). No mention of default behavior for missing parameters or date range handling. Insufficient for a trend tool with zero annotations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, baseline is 3. Description adds no extra meaning beyond the schema, does not explain parameter relationships or formatting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
States verb (get), resource (trend of muted test cases), and context (over time for a project). Clearly distinguishes from siblings like get_statistic_trend which covers general statistics.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. No prerequisites, exclusions, or mention of alternatives like get_statistic_trend.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. Only states basic purpose without disclosing behavioral traits such as ordering, pagination, or performance implications. Lacks details on what 'last' means (e.g., by time or other criteria).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single efficient sentence, front-loaded with the action. No wasted words, though additional context could be added without compromising brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no output schema and only two optional parameters, the description is incomplete. Does not explain what happens when both parameters or neither are provided, and does not describe the return format or structure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for both parameters (projectId and projectName). Description adds no additional meaning beyond schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (Get), resource (last test result), and scope (for each test case in a project). It effectively distinguishes from siblings like get_test_result and list_test_results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives. Does not mention when not to use or provide context for choosing over similar tools like get_test_result or get_test_result_history.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so the description carries full burden. It only states a read-like operation ('Get') but lacks details on behavioral traits such as what happens if testCaseId is invalid, if the operation is read-only, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no redundancy. It is concise, though could slightly benefit from mentioning the return format.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, yet the description does not specify what is returned (e.g., list of issue IDs or objects). The scope of 'linked issues' is ambiguous. For a simple tool, it is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear description that testCaseId must be an integer. The description does not add additional meaning beyond the schema, so baseline score applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Get linked issues for a test case' uses a specific verb ('get') and resource ('linked issues') with clear context ('for a test case'). It distinguishes well from siblings like 'set_test_case_issues' (set vs get) and 'get_defect_test_cases' (different direction).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., set_test_case_issues, get_defect_test_cases). No exclusions, prerequisites, or context provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, and the description only states a read operation without disclosing error handling, permission requirements, or potential side effects. The burden is entirely on the description, which is minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no wasted words. It is appropriately front-loaded but may benefit from slightly more detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, and the description does not specify the return format (e.g., list of tag strings) or handle edge cases like missing test case IDs. For a retrieval tool, more detail on output is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% for the single parameter, and the description does not add any additional meaning beyond the schema's parameter description. Baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves tags assigned to a test case, using a specific verb ('Get') and resource ('tags') with context ('assigned to a test case'). It is distinct from sibling tools that modify tags (e.g., set_test_case_tags, add_test_case_tags_bulk).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives, nor does it mention prerequisites or when not to use it. Sibling tools are not referenced.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must fully disclose behavioral traits. It only states that payload must include status, but does not mention if the operation is reversible, what permissions are needed, or any side effects. This is insufficient for understanding the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no unnecessary words. It is concise and directly states the core requirement.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool is a mutation with no output schema and only basic parameter descriptions, the description should provide more context about expected behavior, allowed status values, and idempotency. It currently lacks completeness for an agent to use it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema covers 50% of parameters (only 'id' has description). The description adds that 'payload must include status', which compensates for the missing parameter description for 'payload'. However, it does not specify other possible properties or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('resolve') and resource ('test result'), and differentiates from siblings like 'create_test_result' or 'update_test_result' by using the specific verb 'resolve', which implies a status update. However, it does not explicitly distinguish from 'assign_test_result' or other result-related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'assign_test_result' or 'update_test_result'. It does not specify prerequisites, context, or conditions that trigger its use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden for behavioral traits. It only says 'Search test cases by AQL query' and omits crucial details: no mention of being read-only, no disclosure of permissions, rate limits, pagination behavior, or side effects. The minimal description leaves significant gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness2/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (5 words) but at the expense of necessary detail. For a complex tool with an AQL query language, it should provide more structure or examples. This is under-specification, not efficient conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the rich schema for rql, the tool description is incomplete. It does not explain return values, pagination behavior, or how to effectively use the query language. Given the complexity (6 params, no output schema), the description is insufficient for an agent to confidently invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is high (83%), and the description adds no additional meaning beyond what the schema provides. The rql parameter has a very detailed schema description, so the tool description's lack of parameter info is acceptable but does not enhance understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Search test cases by AQL query', which specifies the verb (search), resource (test cases), and method (AQL query). It effectively differentiates from siblings like list_test_cases (simple list) and other search tools (search_launches, search_test_results).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies that this tool is for complex filtering via AQL, but it does not explicitly state when to use it versus alternatives like list_test_cases or other search tools. No exclusions or when-not-to-use guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It states 'Add custom field values' implying a write operation, but does not clarify whether it overwrites or appends to existing values, idempotency, required permissions, or any side effects. This is insufficient for safe invocation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the action and then specifying the payload format. Every word contributes value, and it avoids redundancy with the schema. Highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description lacks information about return values, error handling, whether the tool replaces or merges custom fields, and the role of optional projectId/projectName. For a mutation tool with 4 parameters, this is incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 75% of parameters with descriptions (testCaseId, projectId, projectName). The tool description adds meaning by explaining the payload structure (grouped vs flat values), which is not detailed in the schema. This helps the agent construct correct payloads.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Add' and the resource 'custom field values for a test case', mentioning bulk API. It distinguishes from sibling tools like `set_test_case_tags` and `set_test_case_issues` by specifying custom fields, but does not explicitly contrast with them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like `set_test_case_tags` or `set_test_case_issues`. It does not mention prerequisites, limitations, or exclusions, leaving the agent without context for choosing this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description must disclose behavioral traits. It only states the operation without indicating side effects (e.g., what happens if no link exists), authorization needs, or error conditions. For a mutation tool, this is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is concise and front-loaded with the action verb. It is not verbose, but could be slightly more informative without harming conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and lack of output schema, the description should clarify what happens after removal (e.g., success indication, return value). It does not, leaving the agent with incomplete context for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents the 'id' parameter as 'Defect ID. Must be a number (integer), not a string.' The description adds no additional meaning beyond the schema, meeting baseline expectations.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Remove the external issue link from a defect record,' specifying the verb (remove) and the resource (external issue link from a defect record). This is distinct from sibling tool 'link_issue_to_defect,' which adds a link.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No usage guidelines are provided. The description does not indicate when to use this tool versus alternatives (e.g., 'link_issue_to_defect' or 'update_defect'), nor does it mention prerequisites or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must disclose behavioral traits. It only says 'Update', implying mutation, but fails to specify partial vs full update, permission requirements, reversibility, or side effects. The payload schema lacks details on field optionality, adding to the opacity.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, very concise. It front-loads the action and resource, but at the cost of omitting important details. It earns its place but could be slightly expanded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (nested object parameter, no output schema), the description is insufficient. It does not explain return values, update semantics, error conditions, or how partial updates work, leaving critical gaps for agent invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no new information beyond the schema examples ('name, shared status'), and does not clarify payload structure or field constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (Update) and the resource (dashboard fields) with examples (name, shared status), making it distinct from sibling tools like create_dashboard, delete_dashboard, or get_dashboard.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool over alternatives, such as prerequisites (e.g., dashboard must exist) or when not to use it. The agent must infer context from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must convey behavioral traits. It only states the action without explaining side effects, permission requirements, or whether the action is reversible.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise (6 words), which is efficient for a simple tool. However, it could include more context without excessive length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the low complexity (single parameter) and high schema coverage, the description is minimally adequate but lacks context about the meaning of 'close' and any consequences.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the parameter description in the schema is clear. The tool description adds no additional meaning beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states the action 'Close' and the resource 'launch', clearly distinguishing from sibling tools like 'reopen_launch' and 'delete_launch'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives, nor any prerequisites or conditions for closing a launch.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It states the tool creates a launch, but does not mention side effects, idempotency, permissions, or what happens on duplicate creation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise with a single sentence that delivers one key piece of information. It could benefit from additional structure but avoids verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a creation tool with a free-form payload and no output schema, the description lacks details about expected payload fields, return values, and constraints, making it incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters2/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has one required parameter 'payload' with no schema descriptions (0% coverage). The description adds meaning only for the projectId field inside payload, leaving other potential fields undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Create a new launch' clearly states the verb (create) and the resource (launch), distinguishing it from siblings like update_launch, close_launch, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a helpful default behavior for payload.projectId using an environment variable, but does not offer guidance on when to use this tool versus alternatives or any prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description should disclose behavioral traits. It only states the action and method but omits key details like irreversibility, permission requirements, or cascading effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise at 6 words, front-loading the action and resource. Every word is necessary and no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple deletion tool, the description is minimal but lacks critical context about consequences (e.g., permanence) and prerequisites. The lack of output schema and annotations means the description should provide more behavioral context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers the single parameter 'id' with a clear description and type. The tool description adds no extra parameter info beyond what the schema provides, warranting the baseline score.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Delete' and resource 'dashboard', with method 'by ID'. It distinguishes from siblings like create, get, update, copy, and list dashboards.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use or when-not-to-use guidance is given. There is no mention of prerequisites, alternatives, or whether deletion is irreversible.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description only states 'Delete' without disclosing side effects, irreversibility, or any prerequisites. Since no annotations are provided, the description carries the full burden and fails to add behavioral context beyond the verb.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no unnecessary words. It is front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple deletion tool, the description is minimal. It omits important context such as whether the operation is reversible (sibling restore_test_case exists), any cascading effects, or required permissions. More detail would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes the 'id' parameter with type and constraint ('must be a number (integer), not a string'). The description adds no further meaning, so it meets the baseline for full schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (delete), the resource (test case), and the required identifier (by ID). It is specific and distinguishes from sibling deletion tools like delete_dashboard or delete_test_case_attachment.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as restore_test_case or other deletion tools. The description lacks context for appropriate usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It mentions only the purpose and a usage hint, omitting key details such as response format, pagination defaults, error handling for invalid defect IDs, or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise at two sentences, covering purpose and a usage hint without any fluff. Every sentence contributes meaning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of an output schema and the tool's nature (retrieving list of test cases), the description should clarify the return format (e.g., test case IDs or full objects) and pagination behavior. It falls short of providing sufficient context for an agent to anticipate the tool's output.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already describes all three parameters with 100% coverage. The description adds a minor constraint for the id parameter ('Must be an integer'), but provides no additional meaning beyond the schema for page and size.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves test cases affected by a defect, distinguishing it from sibling tools like get_defect_test_results (test results) and get_defect (defect details). However, it could be more specific about what 'test cases affected' means (e.g., linked test cases or those with failures).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a single use case ('understand which test scenarios are impacted') but does not guide when to prefer this tool over alternatives like get_defect_test_results or get_defect_launches, nor when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It fails to disclose behavioral traits such as whether it's read-only, requires authentication, or has any side effects. The description only states the basic action without additional context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, which is concise, but it lacks depth. Additional sentences could provide usage guidelines or behavioral context, making it slightly under-specified for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 3 parameters, no output schema, and many siblings, the description is incomplete. It does not explain output format, parameter relationships, or when to choose this over alternatives like 'get_automation_chart'.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% coverage for parameter descriptions, so the description adds no extra value beyond the schema. Baseline is 3; the description does not enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'Get', resource 'test case counts', and context 'grouped by automation status' for a project. It distinguishes from sibling tools like 'get_group_by_status' which groups by status, not automation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage: when you need counts grouped by automation status. However, no explicit when-to-use or alternatives are mentioned, leaving the agent to infer from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, and the description does not disclose any behavioral details such as whether the operation is read-only, any authentication requirements, rate limits, or side effects. The description carries the full burden but adds nothing beyond the name.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-front-loaded sentence with no wasted words. It conveys the essential purpose efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is too brief for the complexity (7 parameters, no output schema). It does not explain the histogram's output format, how to choose between projectId and projectName, or any constraints. An agent would lack critical context to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 7 parameters have detailed descriptions in the input schema (100% coverage), so the schema already provides meaning. The tool description does not add any additional parameter semantics beyond what is already in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Get histogram of launch durations for a project' uses a specific verb and resource, clearly indicating the tool's primary function. It distinguishes itself from sibling tools like 'get_launch_statistic' or 'get_group_by_status' by explicitly focusing on histograms.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as 'get_launch_statistic' or 'get_launch'. The description does not mention any prerequisites, optimal contexts, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It only states the action without confirming side effects (e.g., read-only, no modifications) or error handling (e.g., what happens if ID does not exist).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no fluff, front-loading the essential purpose. It is appropriately sized for a simple retrieval operation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema, the description should hint at the return structure or format. It does not, leaving the agent uncertain about what data the test result contains. The description is incomplete for decision-making.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (the id parameter is described as 'Test result ID. Must be an integer, not a string.'). The description does not add extra meaning beyond the schema, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Get a test result by ID' clearly states the operation (get) and the resource (test result), with a specific identifier (ID). It distinguishes itself from sibling tools like 'get_test_result_history' and 'list_test_results' by focusing on a single test result retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives such as 'list_test_results' or other getters. It lacks any indication of prerequisites, exclusions, or context for appropriate use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavioral traits, but it only repeats required fields and notes it is a bulk API. It does not describe side effects, permissions, error handling, or idempotency, which are critical for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no superfluous content; purpose and required fields are communicated efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Lacks output schema and annotations, and the description omits return value, error handling, or idempotency information. For a data linking operation, an agent would benefit from knowing what confirms success or failure.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds 'bulk API' context but does not elaborate on parameter format, validation, or examples beyond what the schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Link one or multiple test results to a defect record') and specifies it is a bulk API, which distinguishes it from sibling tools like 'link_issue_to_defect' or 'assign_test_result'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool versus alternatives; lacks contextual cues or exclusions. The tool name and description imply use for linking test results to defects, but does not mention when not to use or compare to siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must cover behavioral traits but only states 'removes' and 'bulk API', failing to disclose destructive nature, error handling, or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no unnecessary words, making it easy to read.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of a bulk operation and no output schema, the description should explain return values or error handling; it provides none, leaving the agent uninformed about what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67% (4 of 6 params described) and the description adds minimal context beyond the schema, e.g., implying single or multiple via parameter names but not explaining the relationship between testCaseId and testCaseIds.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (remove), the resource (tags from test cases), and the bulk capability, distinguishing it from sibling tools like add_test_case_tags_bulk.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., single removal), no prerequisites or exclusion criteria are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavioral traits. It only lists updatable fields but omits details like whether it overwrites fields, requires specific permissions, or is idempotent. The mutation effect is implied but not elaborated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no superfluous words. It front-loads the action and resource, making it efficient for an AI agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has a nested payload parameter and no output schema, the description lacks important context: what the tool returns (if anything), whether the update is immediate, and any side effects. This leaves gaps for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50% (only 'id' has a description). The description partially compensates by implying that 'payload' contains fields like name, description, or status, but does not specify their types or constraints. The meaning added is adequate but not comprehensive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('update'), the resource ('defect'), and the specific updatable fields ('name, description, or status'). This distinguishes it from sibling tools like 'create_defect' and 'get_defect'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives (e.g., 'update_test_case' or 'bulk_close_defects'). It does not specify prerequisites, constraints, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must carry full burden. It only states the basic operation and omits behavioral traits like output format, permissions, or whether it is read-only. A simple get operation, but lacks detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with 11 words, front-loaded with key verb and resource. No unnecessary content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Description is minimal for a tool with no output schema. Does not explain return format (e.g., map of status to count) or whether tcRql is optional. Adequate but could be more complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. Description adds no additional meaning beyond schema; it implies a project context but does not elaborate on parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Get', resource 'test case counts grouped by status', and scope 'for a project'. It distinguishes from sibling 'get_group_by_automation' by specifying status grouping.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. Does not mention when not to use or any prerequisites. Sibling tool 'get_group_by_automation' exists but is not differentiated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries full burden. The tool is clearly read-only ('Get'), but no additional behaviors (e.g., permissions, rate limits, data freshness, pagination) are disclosed. Minimal but adequate for a simple query.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence with no filler, achieving high conciseness. Front-loaded with the key concept. Could benefit from brief structuring (e.g., listing what it returns), but current form is efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite 8 parameters and no output schema, the description fails to explain what 'success rate analytics' entails (e.g., output format, time series, aggregation). Incomplete for a moderately complex analytics tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no extra meaning beyond the schema; it only reiterates the temporal and project focus already present in parameter descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves 'test case success rate analytics over time for a project', specifying the verb (Get), resource (success rate analytics), and temporal scope. This distinguishes it from sibling tools like get_statistic_trend or get_automation_chart.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives such as get_statistic_trend, get_mute_trend, or get_automation_chart. The description does not mention conditions, prerequisites, or exclusions, leaving the agent to guess the appropriate context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as pagination, sorting, filtering options, or whether the operation is read-only. The brief description leaves essential behavior unaddressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, concise and to the point. While it is appropriately short, it could be slightly more informative without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of output schema and annotations, the description is insufficiently complete. It does not mention pagination, sorting, filtering options, or the structure of custom fields, leaving significant gaps for an agent to correctly invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 67% schema description coverage, most parameters have descriptions in the schema. The tool description does not add further meaning beyond what is already in the input schema, so it meets the baseline but does not enhance understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'List custom fields configured for a project.' It uses a specific verb 'list' and resource 'custom fields' with the context 'for a project', distinguishing it from sibling tools like 'get_test_case_custom_fields' which target test cases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives, nor are there any prerequisites or exclusions. The description is functional but lacks context for decision-making.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description lacks any behavioral details such as pagination, output format, or side effects. For a list operation, the agent is left blind to how results are returned or any limitations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, short sentence that is front-loaded with the verb and resource. No unnecessary words or information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is minimal. Given no output schema, the agent lacks information about the return structure (e.g., what fields are included). However, for a simple list tool with one parameter, it may be considered barely adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter testCaseId is fully described in the schema (type, description). The description adds no additional meaning; it merely echoes the schema. Since schema coverage is 100%, a score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'list' and the resource 'attachments for a test case', making the tool's purpose unambiguous. It distinguishes from sibling list tools by specifying the test case scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus related tools like 'get_test_case_attachment_content' or 'upload_test_case_attachment'. Agents have no context for selecting this over alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It lacks disclosure of behavioral traits such as pagination details, sorting behavior, response format, or rate limits. The main description is minimal and does not compensate for the missing annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The main description is a single concise sentence that front-loads the tool's purpose. It is appropriately sized for a simple search tool, though additional context could be included without sacrificing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of annotations and output schema, the description is too brief. It does not explain pagination, default sorting, response structure, or other contextual details needed for correct invocation. The parameter descriptions help but do not fully compensate for the lack of high-level context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is high (83%), so baseline is 3. The description adds significant meaning to the rql parameter by explaining syntax, operators, and field examples. However, it does not describe the sort parameter, and the overall parameter semantics benefit from the schema descriptions but are not fully enhanced by the main description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Search launches by AQL query.' clearly states the tool's action (search) and resource (launches) with a specific filter language (AQL). It distinguishes from siblings like list_launches (no filtering) and search_test_cases (different resource).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for filtered searching via AQL but does not explicitly state when to use this tool over alternatives like list_launches or get_launch. No when-not or exclusion criteria are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description partially discloses behavior: it notes that if expectedResult is provided, the expected result node is created automatically if it does not exist. However, it does not mention whether the update is idempotent, any side effects on related data, or permission requirements. With no annotations, this is acceptable but not thorough.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, consisting of two sentences. The first sentence states the core purpose, and the second adds key details. Every sentence contributes meaning without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description lacks information about the return value or confirmation of success, which is important for a tool that modifies data. No output schema is provided, and the description does not indicate what the tool returns (e.g., the updated step or a success status). Additionally, it does not cover error conditions or required permissions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already fully documents the three parameters with descriptions. The description adds value by clarifying that both body and expectedResult can be updated in one call and by specifying the auto-creation behavior for expectedResult, which goes beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Update a test case step by ID', specifying the verb and resource. It mentions the specific fields that can be updated (body text and expectedResult), which helps distinguish it from sibling update tools like update_test_case or update_shared_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives, such as update_test_case (which updates test case-level fields) or creating a new step. The description does not include any context about prerequisites or typical usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It states the tool is destructive ('Delete'), but provides no additional behavioral details such as side effects, error conditions, or what happens to the test case. This is minimal for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, direct sentence with no unnecessary words. It is appropriately concise and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description covers the essential action. However, it lacks context about preconditions (e.g., obtaining the attachment ID) or error handling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The tool description does not add extra meaning beyond the schema's parameter description; it merely references the attachment ID.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Delete') and resource ('attachment from a test case'), and specifies the method ('by attachment ID'). It clearly distinguishes from sibling tools like upload, list, and get content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., listing attachments first to get the ID). No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description is minimal; no annotations exist. Does not disclose behavioral traits such as read-only nature, error handling for non-existent IDs, or permission requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded, no wasted words. Highly concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple retrieval tool, description is nearly complete. However, missing return value description (no output schema) and behavioral details slightly reduce completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with clear parameter description. Tool description adds no additional meaning beyond 'by ID', which is redundant. Baseline score appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Get', resource 'dashboard', and method 'by ID', distinguishing it from sibling tools like list_dashboards and create_dashboard.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance provided on when to use this tool versus alternatives such as list_dashboards or other get_* tools. Lacks context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description implies a read-only operation but does not disclose any additional behavioral traits (e.g., permissions, rate limits, or return structure). With no annotations, the description could be more explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence, but it lacks any structure that might help scanning. It is efficient but could be slightly more descriptive without verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple getter with one parameter and no output schema, the description is adequate but does not specify the return value (e.g., full launch object). It is minimally complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The parameter 'id' is well-described in the schema (type, format), achieving 100% schema description coverage. The tool description does not add further meaning, but the schema already provides adequate semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Get a launch by ID' clearly states the verb (Get), resource (launch), and method (by ID), effectively distinguishing it from sibling tools like list_launches or search_launches.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives, such as list_launches for multiple launches or search_launches for filtered queries.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states 'restore', which implies a state change, but provides no information about side effects, required permissions, error conditions (e.g., already active), or return value. With no annotations, more detail is needed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that conveys the core action without any unnecessary words. It is front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
As a simple mutation tool with no output schema, the description omits important context such as what the tool returns, what happens if the step doesn't exist or is already active, and any prerequisites. More completeness is needed for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter 'id' is already fully described in the schema (must be a number). The description adds no further semantics, so it meets the baseline for 100% schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'restore' and resource 'archived shared step', clearly indicating the action. It directly contrasts with the sibling tool 'archive_shared_step', making its purpose distinct.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies it is for reversing an archive operation, but does not explicitly state when or when not to use it, nor does it mention prerequisites like the shared step must be currently archived.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are present, so the description carries full burden. It only states the action of updating but does not disclose behaviors such as idempotency, error handling, required permissions, or side effects. Critical gaps for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The first sentence states the core purpose, and the second provides a concrete example for a complex parameter. Efficient and to the point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a complex input (payload object) and no output schema. The description does not explain what the response looks like, what constraints exist on payload fields, or how to handle the customFields structure. Incomplete for a mutation tool with nested objects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50% (id described, payload not). The description adds a format example for payload.customFields but does not compensate fully for the missing description of the payload parameter. AdditionalProperties allows any fields, but no guidance is given.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Update an existing test case,' which is a specific verb and resource. It distinguishes this from sibling tools like create_test_case, get_test_case, and delete_test_case.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for modifying test cases but does not explicitly state when to use this versus alternatives like set_test_case_custom_fields or delete_test_case. No when-not-to-use guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description briefly mentions marking as resolved but omits side effects, permissions, or limits on batch size.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences convey purpose and required parameter without superfluous words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers basic purpose and parameter need but lacks return value details, error conditions, or prerequisite defect states.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers the single parameter with description; tool description repeats the same info, adding no new meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb 'close' and resource 'multiple defects', differentiating from siblings like bulk_reopen_defects and update_defect.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage after fix confirmation but provides no explicit when-to-use vs alternatives such as update_defect or bulk_reopen_defects.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It mentions 'fastest way' and 'distinct failure patterns' but fails to disclose pagination, error behavior, or auth requirements. Minimal behavioral insight.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences: first states action, second adds context. No redundant words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with no output schema, the description is adequate but lacks details on return values, pagination, and error cases. Could be improved for completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the description adds little beyond what the schema already provides. The description does not elaborate on parameter formats or constraints beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool gets all defects for a specific launch, and provides a use case ('fastest way to get a defect summary'). It distinguishes from siblings like get_defect (single defect) and list_defects (all defects).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for getting a defect summary, but does not explicitly state when not to use or provide alternatives. There is no guidance on avoiding this tool when detailed defect info is needed.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It only says 'Restore' without explaining side effects, requirements (e.g., test case must be deleted), or constraints (e.g., restoration window).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence with no unnecessary words, efficient and to the point.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description is minimal but does not explain the restore operation's effect or return value. Lacks behavioral context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema coverage is 100% and the description adds no additional meaning beyond the schema's 'Test case ID. Must be a number (integer), not a string.' Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Restore') and resource ('a deleted test case'), distinguishing it from sibling tools like delete_test_case or create_test_case.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when a deleted test case needs to be restored, but provides no explicit guidance on when not to use it, prerequisites, or alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It states 'reopen' implying status change but lacks details on permissions, side effects, limits, or what happens if defects don't exist. Minimal behavioral disclosure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no filler, directly stating purpose and required payload. Highly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema, no annotations, and a bulk operation: the description lacks details on error handling, idempotency, or limits, making it insufficient for safe and complete usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has full description coverage for the payload parameter. The tool description repeats the schema info ('ids (array of defect IDs)') without adding extra meaning, meeting the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Reopen multiple closed defects' and specifies the use case 'when a regression is detected after a fix was marked complete,' providing a specific verb+resource and distinguishing from siblings like bulk_close_defects.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'use when a regression is detected after a fix was marked complete,' giving a clear condition for use. However, it does not specify when not to use or mention alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behaviors. It only restates parameter requirements (name, projectId) without mentioning what happens on success (e.g., returns defect ID), error conditions, or authentication needs. This is insufficient for a creation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one sentence plus a list of required/optional fields. It is extremely concise with no redundant information, earning its place efficiently.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple but lacks mention of return value (e.g., defect ID) or error handling. No output schema exists, and the description provides no behavioral context beyond parameters. This is incomplete for guiding agent invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 100% coverage but lacks type information for inner properties (additionalProperties: true). The description adds types: 'name (string), projectId (number)' and optional description, which clarifies meaning beyond the schema. It adds value but could be more detailed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Create a new defect record to group a novel failure pattern.' It uses a specific verb ('Create') and resource ('defect record'), and distinguishes from sibling tools like update_defect by emphasizing 'novel failure pattern'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for new failure patterns but does not explicitly state when not to use it or mention alternatives like update_defect for existing defects. It provides minimal guidance on context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must cover behavioral traits. It mentions minimum required fields but fails to disclose side effects, permissions, return value, or error behavior. The description adds little beyond what the schema already indicates.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no wasted words. First sentence states core purpose, second provides usage requirement and motivation. Information is front-loaded and efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and minimal schema description, the description lacks important context: no mention of what the tool returns (e.g., created step ID), no prerequisites, no permission requirements. Leaves agent underinformed for a creation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. Description repeats required fields with types (name string, projectId number), adding minimal extra meaning beyond schema. No additional semantics about optional fields or constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'Create' and resource 'shared step', with context 'in a project'. It distinguishes from siblings like update_shared_step and archive_shared_step by focusing on creation and providing a use case: 'extract repeated test steps into a reusable library'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides clear context for when to use the tool: 'Use this to extract repeated test steps into a reusable library.' No explicit exclusions or alternatives, but the purpose is distinct from other tools, making usage straightforward.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description must carry the full behavioral burden. It indicates a read operation but does not disclose error handling (e.g., behavior for invalid IDs), permissions needed, or side effects. However, the read-only nature is implied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence plus return field list, concise and frontloaded. Every word is necessary; no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple retrieval tool, the description sufficiently covers purpose and return values. Given no output schema, the metadata list is helpful. Could mention the broader context of shared steps, but not critical.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds no extra meaning beyond what the schema already provides for the 'id' parameter (shared step ID, must be number). Baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Get a shared step by ID' and specifies the returned metadata (name, project, archived status). This distinguishes it from siblings like list_shared_steps (browse all) and get_shared_step_steps (retrieve steps within a shared step).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as list_shared_steps for browsing or get_shared_step_steps for detailed steps. The description does not define scope or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It only says 'List' without disclosing pagination, filtering, default behavior, or that it is a read-only operation. Lacks behavioral details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, concise and front-loaded; every sentence adds value without extra words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite 6 parameters and no output schema, description does not mention return structure, pagination, or any constraints. Incomplete for a list tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. Description adds no additional meaning beyond what the schema already provides for parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists shared steps, explains they are reusable step libraries, and mentions their usage via sharedStepId, distinguishing it clearly from sibling tools like get_shared_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear context for when to use this tool ('before reading test case steps'), though it does not explicitly state when not to use it or mention alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It states what the tool does but lacks details on side effects (e.g., whether test results are modified, if it creates links, idempotency, permissions required). Critical behavioral traits are missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences with no redundant information. It is front-loaded with the main action and provides quick understanding. Every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and one parameter, the description is partially complete. It explains what the tool does and when to use it, but lacks behavioral details and prerequisites (e.g., that defect matcher rules must be configured beforehand). More context is needed for a complete understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the schema already describes the only parameter as 'Launch ID. Must be a number (integer), not a string.' The description adds no extra semantic value beyond what the schema provides, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (apply), the resource (defect matcher rules to a launch), and the purpose (auto-triage unresolved test results). It distinguishes from sibling tools like create_defect or find_similar_failures by specifying this is an automated application of configured rules after launch completion.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Run this after a launch completes to auto-group failures', providing clear when-to-use guidance. It does not explicitly state when not to use or list alternative tools, but the context is sufficient for correct usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully convey behavioral traits. It fails to mention pagination behavior, return format, or any authorization needs. The parameters 'page' and 'size' hint at pagination, but the description ignores this, making the tool's behavior opaque.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that front-load the purpose and then provide a usage hint. No redundant words, every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity (3 params, no output schema, no nested objects), the description covers purpose and usage but lacks behavioral details like pagination or return structure. It is adequate but incomplete for an AI agent to fully understand the tool's behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all three parameters. The tool description adds no extra meaning beyond the schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and resource 'launches' with a specific defect context. It also provides a use case ('understand the blast radius'), which distinguishes it from siblings like list_launches or get_defect.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly recommends using the tool to assess blast radius and poses a guiding question. However, it does not mention when not to use it or suggest alternative tools, leaving some ambiguity for an AI agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It mentions required payload fields but does not describe side effects, idempotency, error handling, or prerequisites like defect existence.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words. The purpose is front-loaded and the description is appropriately sized for the tool's simplicity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description is adequate but incomplete. It does not explain return values, prerequisites (defect must exist), or the optional 'type' parameter.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description reiterates required fields (url, name) but adds no new meaning beyond the schema. It omits mentioning the optional 'type' field.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('link') and the resources ('external issue to a defect record'). It distinguishes from siblings like 'link_defect_to_test_results' and 'unlink_issue_from_defect'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear use case: 'associate a tracker ticket with a failure pattern'. However, it does not explicitly state when not to use it or mention alternatives like 'unlink_issue_from_defect'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so description must bear the burden. It mentions base64 encoding but fails to disclose side effects, permissions, or constraints (e.g., file size limits). Minimal behavioral info beyond schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences that are front-loaded with the action and essential detail. No wasted words; every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 required parameters and no output schema, the description offers minimal context beyond the base64 encoding. Lacks information on upload behavior (e.g., overwriting, result) and prerequisites, but is adequate for a basic operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers 100% of parameters with descriptions. The description adds no new meaning beyond repeating base64 encoding. Baseline score of 3 is appropriate due to high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states action (Upload) and resource (file attachment to a test case), including the base64 encoding requirement. This distinguishes it from sibling tools like delete, get, and list attachments.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description implies usage for attaching files but provides no explicit when-to-use, when-not-to-use, or alternatives. Lacks guidance on prerequisites (e.g., test case must exist) or size limits.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, and the description does not disclose behavioral traits such as authentication needs, error handling (e.g., if ID not found), or rate limits. This is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no superfluous words. It is front-loaded and directly states the purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (get by ID), the description is mostly sufficient. It could mention the return format or failure behavior, but the current brevity is acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (1 parameter described), and the description adds extra clarity: 'Must be a number (integer), not a string.' This goes beyond the schema's type definition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (get), the resource (test plan), and the distinguishing criteria (by ID). It effectively differentiates from sibling tools like list_test_plans.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when or when not to use the tool, nor mention of alternatives. However, for a simple retrieval tool, the use case is generally obvious.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. Only states it lists keys, but does not disclose potential behavior like authorization, rate limits, or whether all keys are returned. For a simple read-only tool, more could be said.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two efficient sentences. Front-loaded with action and resource. Every sentence provides value: first defines purpose, second gives usage guidance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and no output schema, description covers purpose and usage. Could mention that it returns a list of key names, but not essential for this simple tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has no parameters, so coverage is 100%. Description adds context about the nature of keys (test configuration context) but no parameter details needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'list' and resource 'environment variable definitions (keys)'. It distinguishes from siblings like list_env_var_values by specifying 'definitions (keys)' and provides example keys (browser, OS, etc.).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when to use: before constructing AQL filters in search_test_results or search_launches. Provides example usage with ev["browser"] = "Chrome". Does not explicitly exclude alternatives, but context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It states return value is 'candidates for linking' but does not disclose matching algorithm, side effects, or pagination behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place; first sentence front-loads purpose, no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a search tool: explains purpose, parameters, return. Missing explanation of pagination (page & size) and behavior details; no output schema, but description covers key aspects.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema descriptions cover 100% of parameters. Description adds value by explaining how parameters relate to the tool's purpose (scoping search, matching defects) and what returns, beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description states specific verb 'Find' and resource 'test results matching a defect pattern within a launch', and distinguishes from sibling tools by mentioning flaky detection and root cause grouping.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use: 'detect flaky tests and group failures by root cause' and explains optional parameter usage. Lacks explicit alternatives or when-not-to-use, but clear context is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears the burden of behavioral context. It discloses that the tool returns a list of previous attempts, but does not mention pagination behavior (despite page/size parameters), rate limits, or any side effects. The added context is useful but incomplete.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with only two sentences, the first stating the purpose and the second providing use guidance. Every word adds value, and the key information is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and no annotations, the description could be more complete. It does not describe the structure of each retry attempt or how to interpret the list. While the flaky test scenario adds some context, the overall completeness is moderate for a tool of this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed parameter descriptions (e.g., id must be a number). The description does not add extra meaning beyond the schema, so a baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool gets retry history for a test result, and distinguishes it by focusing on previous attempts for the same test in the same launch. This specificity differentiates it from siblings like get_test_result and get_test_result_history.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly positions the tool for flaky test detection with a concrete scenario (failing on attempt 1, passing on attempt 2). It advises assessing reliability across retries, providing clear use guidance. However, it does not explicitly state when not to use it or mention alternatives, missing full exclusion criteria.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It explains that defects have name, status, and matcher rules but does not disclose pagination behavior, required context (e.g., projectId or projectName may be needed despite schema showing none required), or error handling. The description adds some context but leaves gaps in behavioral details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loading the core purpose, then defining defects, and ending with a usage example. Every sentence adds value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description gives a good overview and a usage example, but lacks details on output format (no output schema), pagination defaults, and whether projectId or projectName is effectively required. This ambiguity affects completeness for agent usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with each parameter documented. The description does not add significant meaning to individual parameters beyond the schema; it provides high-level context about defects. Baseline is 3, and the description does not elevate it further.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists defect records for a project, explains what defects are (grouped failures by root cause), and provides a concrete usage example. It differentiates from sibling tools like get_defect and create_defect by positioning it as an entry point for failure pattern analysis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description advises using this tool as the entry point for failure pattern analysis and gives a natural language query. However, it does not explicitly state when not to use it or list alternative tools for specific operations (e.g., get_defect for a single defect, bulk_close_defects for updating status).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses that results are grouped by defect and includes failure counts and launch information, but lacks details on return format, pagination behavior, or side effects. The information is adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences. The first defines the tool, the second provides usage context. No unnecessary words or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 3 parameters, no output schema, and no annotations, the description gives enough context to understand purpose and typical use. It could be improved by specifying the return structure or pagination details, but it is not critically incomplete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description does not add new meaning beyond the schema for any parameter; it simply repeats the concept of 'defect' from the schema's id description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'test results grouped under a specific defect', distinguishing it from siblings like get_defect (defect info), get_defect_launches (launches), and list_test_results (ungrouped list).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a concrete use case: 'to see how many failures belong to this defect and in which launches they appeared.' It implicitly guides usage for aggregated defect-oriented analysis, though it does not explicitly exclude alternatives or provide when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full behavioral burden. It notes the return format matches get_test_case_steps, implying a read operation, but does not disclose error handling, authentication needs, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences long, front-loaded with the primary purpose, and contains no redundant or unnecessary information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simple nature of the tool (one parameter, no output schema), the description adequately explains its role in a larger workflow and mentions the return format. It is mostly complete but could add error scenarios or idempotency.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear description for the sole parameter 'id', including type enforcement. The description adds no additional parameter context beyond the schema, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns the full normalized scenario inside a shared step, and explicitly connects it to resolving sharedStepId references, distinguishing it from sibling tools like get_test_case_steps and get_shared_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells when to use the tool: when get_test_case_steps returns a step with sharedStepId. It provides clear context but does not mention when not to use it or list alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It clearly states it returns base64-encoded content with MIME type, indicating a read operation. However, it does not mention any side effects or permissions; it is adequate for a simple download.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no unnecessary words. Information is front-loaded and every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple 1-parameter download tool with no output schema, the description is complete enough. It covers purpose and return format. Could mention error conditions or file size limits, but not essential.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear description of attachmentId parameter. The description adds no additional parameter semantics beyond what the schema already provides, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Download the binary content of a test case attachment' with specific verb and resource, and distinguishes from siblings like list_test_case_attachments (list) and delete_test_case_attachment (delete).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives. It implicitly suggests downloading attachment content, but does not mention prerequisites like obtaining attachment ID from list_test_case_attachments or when not to use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description explains the return structure in detail (normalized scenario, root step, flat map of steps with specific fields), which is transparent. However, with no annotations provided, it does not explicitly state that it is a read-only operation or mention any potential side effects, though the name and context imply it is safe.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences: the first defines the purpose, the second details the return structure. It is concise, front-loaded, and contains no redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description adequately explains the return structure (root step, flat map, step fields). However, it lacks information about error handling, expected behavior for invalid IDs, or any constraints beyond the schema. Still, it is sufficiently complete for a single-parameter read tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% (the only parameter 'id' is fully described in the schema). The description does not add any extra meaning beyond the schema; it simply reiterates that the tool returns steps for a test case. Baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb (Get) and resource (manual scenario steps for a test case). It provides detailed information about the return structure, distinguishing it from sibling tools like get_test_case (which returns test case metadata) and get_shared_step_steps (for shared steps).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage (when you need steps of a test case), but does not explicitly state when to use this tool over alternatives like get_test_case or get_shared_step_steps. No usage guidance, exclusions, or alternative recommendations are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses that it returns distinct values across test runs, but with no annotations, the description should also mention any side effects or response format. It adequately describes behavior for a simple read tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences: first states purpose, second gives prerequisite, third provides use case. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Lacks explicit response format (e.g., array of strings) and error cases, but for a simple tool with one parameter and no output schema, it covers the essential context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema covers the parameter fully (100% coverage). The description reinforces the parameter's origin from list_env_vars but adds no new semantic details beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it lists recorded values for a specific environment variable key, with an example (browser versions). It distinguishes itself from siblings like list_env_vars (which lists keys) and suggest_env_var_values.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs to use envVarId from list_env_vars and highlights usefulness for building AQL filters. It implies when to use but does not explicitly state when not to.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It describes that the tool returns metadata and implies a read-only operation. However, it does not disclose potential side effects, auth requirements, or pagination limits beyond the schema parameters. Adequate but could be more informative.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loads the action and return values, and contains no redundant information. Every sentence is purposeful and aids understanding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description mentions return fields (id, name, contentType, size), which is helpful. It also references a sibling tool. Missing error conditions or handling of invalid testResultId, but overall fairly complete for a simple listing tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already provides descriptions for all 3 parameters (100% coverage), so the description adds little additional meaning beyond the schema. It does not elaborate on how to use page and size effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists attachments for a test result and specifies the returned metadata (id, name, contentType, size). It distinguishes itself from the sibling tool get_test_result_attachment_content by indicating it's for discovery before downloading content.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly advises using this tool before calling get_test_result_attachment_content, providing clear usage context. However, it does not explicitly state when not to use it or mention alternative tools like list_test_case_attachments for test cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must cover behavioral traits. It mentions search and optional scoping but does not disclose output format, pagination details, or rate limits. Adequate but minimal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, no redundant information. Front-loaded with the core action and optional scoping.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequately describes the tool for a search with multiple optional filters. Lacks details on return structure or pagination, but given schema covers parameters, it is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. Description adds context about partial text search and optional scoping but does not significantly enhance parameter understanding beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Search for recorded environment variable values by partial text' with optional scoping, distinguishing it from siblings like list_env_var_values and suggest_env_vars.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states 'Use this to autocomplete valid values for AQL ev[] filters before running a search', providing a clear use case. Lacks explicit exclusions or alternatives but sufficiently guides usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description carries full burden. It correctly implies a read operation with no destructive behavior, but does not disclose any specific behavioral traits like auth requirements, rate limits, or pagination details beyond the schema parameters.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first identifies action and resource, second explains purpose. No unnecessary words, efficient and informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Simple list tool with well-documented parameters. Description completely covers the tool's purpose and expected output (list of schema definitions). No missing context given lack of output schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions for all 4 parameters. The description adds minimal value beyond the schema, simply restating that the tool works 'for a project'. No further semantic enrichment.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it lists environment variable schema definitions for a project, explaining what schemas are and their role. Differentiates from sibling tools like list_env_vars by focusing on schemas, not actual values.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says to use this to understand configuration context captured per launch. Implicitly distinguishes from tools that list actual env var values, but does not provide explicit when-not-to-use or alternative names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses that steps are created after the test case, that projectId defaults to an environment variable, and how custom fields and steps are structured. This is good for a creation tool, though it could mention retention policies or error handling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness3/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, dense paragraph of 156 words. It front-loads the purpose but mixes field details without clear structure (e.g., bullets). While every sentence adds value, it could be more concise and better organized for quick scanning.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of the payload (nested object, multiple subfields) and lack of output schema, the description covers most input fields but omits other possible properties (e.g., tags, attachments) and does not describe the return value (e.g., created test case ID). This leaves some gaps for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% (only payload object without inner properties), but the description compensates fully. It explains each subfield: projectId defaults, precondition vs description distinction, customFields format, and steps array structure with expectedResult guidelines. This adds immense value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Create a new test case.' It explicitly identifies the verb (create) and resource (test case), distinguishing it from siblings like 'update_test_case' or 'create_shared_step'. The detailed field guidance reinforces the purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides thorough usage instructions for each field (e.g., payload.precondition vs payload.description, step expectedResult formatting). It explains defaults and format requirements. However, it does not explicitly tell when NOT to use this tool or mention alternatives like 'update_test_case' for modifications.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries the full burden. It discloses that the operation is read-only and specifies the return fields. It does not mention error conditions, authentication needs, or potential side effects, but the described behavior is accurate and sufficient for a simple get operation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that is front-loaded with purpose and includes essential return information. No extraneous content—every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description partially compensates by listing key return fields. It is sufficient for an agent to understand what to expect. It could mention that the defect must exist, but overall it is complete enough for the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the schema already documents the id parameter well (type, format). The description adds no new information about the parameter beyond restating that it is used to fetch a defect. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves a single defect by ID and lists the specific fields returned (name, status, description, matcher configuration). This distinguishes it from sibling tools like list_defects (which returns multiple) and update_defect (which modifies).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The context is clear: use when you have a specific defect ID and need details. However, it does not explicitly state when not to use it or mention alternatives like list_defects for querying multiple defects. The guidance is implied but adequate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It describes the purpose but does not disclose behavioral traits such as pagination behavior, rate limits, or required permissions. The description adds minimal behavioral context beyond the schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose and usage guidance. No extraneous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema, the description gives a reasonable hint about the return ('list of test cases that use a specific shared step'). For a list tool with pagination parameters, it could be more explicit about the return structure, but it is fairly complete for its purpose.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the schema already provides clear descriptions for each parameter (e.g., 'Shared step ID. Must be a number (integer), not a string.'). The description adds no additional meaning beyond the schema, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get' and the resource 'list of test cases that use a specific shared step', distinguishing it from siblings like get_shared_step, get_shared_step_steps, and archive_shared_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly provides usage context: 'Use this for impact analysis before editing or archiving a shared step: shows which test cases would be affected.' This tells when and why to use the tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must cover behavior. It accurately describes a read operation without side effects. However, it does not disclose potential rate limits or authentication requirements, which is acceptable for a simple get tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second gives usage guidance with example. Concise and front-loaded with no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a single-parameter read tool. Covers purpose and usage well. Lacks description of output format, but acceptable given no output schema and simple nature.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already covers testResultId with type and constraints. Description adds value by explaining that the returned env vars represent configuration context (browser, OS, etc.), enhancing understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it retrieves environment variable values for a specific test result, distinguishing it from general env var tools (list_env_vars, list_env_var_values) and other test result tools (get_test_result).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit usage scenario: investigating failures to understand environmental factors, with a concrete example. Lacks explicit when-not-to-use or alternative tool mentions, but the context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavior fully. It states the return type (keys and IDs) but does not mention pagination behavior, matching algorithm (exact vs partial), or order of results. With pagination parameters in the schema, some behavioral context is missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, no fluff. First sentence states the verb and resource. Second sentence provides usage guidance. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose and usage context well. For a simple search tool with pagination (3 params, no output schema), it provides adequate information. However, it could mention pagination behavior (e.g., default page size) or error handling for completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The main description adds no additional semantics beyond the schema; it only mentions 'by name' corresponding to the query parameter. No new meaning is provided for 'page' or 'size'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a clear verb 'Search' and specifies the resource 'environment variable definitions by name'. It distinguishes itself from siblings like 'list_env_vars' (list all) and 'suggest_env_var_values' (suggest values) by focusing on name-based search and returning IDs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states the primary use case: 'Use this to find the exact name of an env var key before querying values.' This provides clear guidance on when to invoke this tool and implies it is a preparatory step, distinguishing it from value querying tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so the description carries the full burden. It discloses AQL syntax, operators, fields, date formats, and a critical type nuance (id as integer). It does not mention pagination or side effects, but for a read-only search, this is sufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, front-loaded with the core purpose, and every sentence adds essential information. No redundant or filler content. The schema-level documentation is separate, keeping the description lean.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complex query language and multiple parameters, the description covers the key behavioral aspects (AQL syntax, id type). However, it does not explain the return structure (e.g., pagination or result format), which is a minor gap but acceptable due to the detailed schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 83% with detailed parameter descriptions. The description adds substantial value beyond the schema, especially the note about the 'id' field being an integer (important for downstream tools) and the extensive AQL syntax examples.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Search test results by AQL query', specifying the verb (search), resource (test results), and method (AQL query). It differentiates from siblings like get_test_result (single result) and list_test_results (likely without AQL).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives. It implies usage for searching via AQL but lacks exclusion criteria or alternative tool mentions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description bears full burden. States returns base64-encoded content with MIME type, and explains how to handle different content types. Does not mention rate limits or size constraints, but overall sufficient for a download tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with purpose, no redundant information. Each sentence adds value: purpose, source of ID, and usage examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given one parameter, no output schema, and no annotations, the description covers key aspects: return format, MIME type, and handling of different content types. Could mention potential errors, but overall complete for typical usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. Description adds context by specifying attachmentId comes from list_test_result_attachments and reinforces that it must be an integer. Provides meaningful guidance beyond schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Clear verb 'Download' and resource 'binary content of a test result attachment'. Distinguishes from sibling tools like list_test_result_attachments and get_test_case_attachment_content by referencing attachment ID source and providing usage context.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly tells to use attachment ID from list_test_result_attachments, and advises how to process the content (decode base64 for text, render for images). Lacks when-not-to-use or comparison with similar tools like get_test_case_attachment_content.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Describes outcome (archived steps readable but hidden from active library). No annotations provided, so description carries burden; lacks mention of permissions or side effects but covers core behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences: first defines action and outcome, second gives usage guideline. No redundant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple archive operation with one parameter and no output schema, the description fully covers purpose, behavior, and a critical usage check via sibling tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Single parameter 'id' with schema coverage 100%. Description does not add extra meaning beyond schema's own description of the parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states action (archive), resource (shared step), and effect (retire from active use, remain readable but hidden). Distinguishes from sibling tools like unarchive_shared_step.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly advises to run 'get_shared_step_usage' first to check impact, providing when-to-use and precondition guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![allure-testops-mcp MCP server](https://glama.ai/mcp/servers/iampopovich/allure-testops-mcp/badges/card.svg)](https://glama.ai/mcp/servers/iampopovich/allure-testops-mcp)

Score Badge

Copy to your README.md:

[![allure-testops-mcp MCP server](https://glama.ai/mcp/servers/iampopovich/allure-testops-mcp/badges/score.svg)](https://glama.ai/mcp/servers/iampopovich/allure-testops-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/iampopovich/allure-testops-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server