Xray MCP Server

Server Quality Checklist

Profile completionA complete profile improves this server's visibility in search results.

Latest release: v1.0.0
Disambiguation5/5
Every tool has a clearly distinct purpose targeting specific Xray entities (test cases, test plans, test sets, test executions) and actions (create, get, update, delete, add/remove tests, search). No ambiguity exists between tools, as each name precisely indicates its function and target resource.
Naming Consistency5/5
All tools follow a consistent verb_noun pattern with clear, descriptive names. The naming convention is uniform throughout, using snake_case and consistent verb choices (e.g., create_, get_, update_, delete_, search_, add_, remove_) paired with specific entity names.
Tool Count3/5
With 23 tools, the count is borderline high for a test management server, potentially feeling heavy. However, given the comprehensive coverage of CRUD operations, searches, and management actions across multiple entity types, it is somewhat justified but could be streamlined.
Completeness5/5
The tool set provides complete CRUD/lifecycle coverage for all core Xray entities (test cases, test plans, test sets, test executions), including search capabilities and specific operations like adding/removing tests and updating test run status. No obvious gaps exist for the domain.
Average 2.9/5 across 23 of 23 tools scored.
See the Tool Scores section below for per-tool breakdowns.
- No community issues in the last 6 months
- 0 commits in the last 12 weeks
- No stable releases found
- No critical vulnerability alerts
- No high-severity vulnerability alerts
- No code scanning findings
- CI status not available
This repository is licensed under ISC License.
This repository includes a README.md file.
No tool usage detected in the last 30 days. Usage tracking helps demonstrate server value.
Tip: use the "Try in Browser" feature on the server page to seed initial usage.
Add a glama.json file to provide metadata about your server.
If you are the author, simply .
If the server belongs to an organization, first add glama.json to the root of your repository:
```
{
  "$schema": "https://glama.ai/mcp/schemas/server.json",
  "maintainers": [
    "your-github-username"
  ]
}
```
Then . Browse examples.
Add related servers to improve discoverability.

How to sync the server with GitHub?

Servers are automatically synced at least once per day, but you can also sync manually at any time to instantly update the server profile.

To manually sync the server, click the "Sync Server" button in the MCP server admin interface.

How is the quality score calculated?

The overall quality score combines two components: Tool Definition Quality (70%) and Server Coherence (30%).

Tool Definition Quality measures how well each tool describes itself to AI agents. Every tool is scored 1–5 across six dimensions: Purpose Clarity (25%), Usage Guidelines (20%), Behavioral Transparency (20%), Parameter Semantics (15%), Conciseness & Structure (10%), and Contextual Completeness (10%). The server-level definition quality score is calculated as 60% mean TDQS + 40% minimum TDQS, so a single poorly described tool pulls the score down.

Server Coherence evaluates how well the tools work together as a set, scoring four dimensions equally: Disambiguation (can agents tell tools apart?), Naming Consistency, Tool Count Appropriateness, and Completeness (are there gaps in the tool surface?).

Tiers are derived from the overall score: A (≥3.5), B (≥3.0), C (≥2.0), D (≥1.0), F (<1.0). B and above is considered passing.

Tool Scores

Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states this is an update operation, implying mutation, but doesn't cover critical aspects like required permissions, whether changes are reversible, rate limits, or what the response looks like (since no output schema exists). This leaves significant gaps for an agent to understand the tool's behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, direct sentence with zero wasted words—'Update an existing test case' is front-loaded and efficiently conveys the core action. Every part of the description earns its place, making it highly concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of a mutation tool with 5 parameters, no annotations, and no output schema, the description is incomplete. It lacks information on behavioral traits (e.g., side effects, error handling), usage context, and output expectations, failing to compensate for the missing structured data and leaving the agent under-informed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, with all 5 parameters clearly documented in the schema itself (e.g., testKey as the identifier, summary as the title). The description adds no additional parameter semantics beyond what the schema provides, so it meets the baseline score of 3 for high schema coverage without adding value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose3/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Update an existing test case' clearly states the verb ('update') and resource ('test case'), making the purpose understandable. However, it doesn't differentiate this tool from its sibling 'update_test_run_status' or other update-like operations, nor does it specify what aspects of a test case can be updated beyond what the schema reveals.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'create_test_case' for new entries or 'delete_test_case' for removal. It doesn't mention prerequisites (e.g., needing an existing test case key) or contextual cues for selection among sibling tools, leaving usage entirely implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. While 'add' implies a mutation operation, it doesn't disclose important behavioral traits like whether this requires specific permissions, if it's idempotent, what happens when adding duplicate tests, or what the expected response format is. The description is minimal and lacks necessary context for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise - a single sentence that directly states the tool's purpose with zero wasted words. It's appropriately sized for a simple tool and front-loaded with the essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what happens after adding tests, whether there are success/failure conditions, what permissions are required, or how it interacts with the broader test management system. Given the complexity implied by the sibling tools, more context is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both parameters clearly documented in the schema. The description doesn't add any meaningful parameter semantics beyond what's already in the schema. It doesn't explain what constitutes valid test plan or test issue IDs, format requirements, or constraints on the array size.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('add tests') and target resource ('to an existing test plan'), providing a specific verb+resource combination. However, it doesn't differentiate from the sibling tool 'add_tests_to_test_set' which has a similar purpose but targets a different resource type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (like needing an existing test plan), when not to use it, or how it differs from similar sibling tools like 'add_tests_to_test_set' or 'create_test_plan'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It states this is an 'add' operation which implies mutation, but doesn't disclose permission requirements, whether this is reversible, rate limits, error conditions, or what happens if tests are already in the set. For a mutation tool with zero annotation coverage, this is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero wasted words. It's appropriately sized for a straightforward tool and front-loads the essential information immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no annotations and no output schema, the description is inadequate. It doesn't explain what the tool returns, error conditions, or behavioral characteristics. Given the complexity of modifying test sets and the lack of structured metadata, the description should provide more operational context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already fully documents both parameters. The description doesn't add any parameter semantics beyond what's in the schema - it doesn't explain what constitutes valid test set issue IDs versus test issue IDs, or provide examples. Baseline 3 is appropriate when schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Add tests') and target resource ('to an existing test set'), providing a specific verb+resource combination. However, it doesn't differentiate from the sibling tool 'add_tests_to_test_plan' which has a similar structure but different target resource, missing explicit sibling differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'add_tests_to_test_plan' or 'create_test_set'. It mentions 'existing test set' which implies a prerequisite but doesn't specify when this tool is appropriate versus creating a new test set or using other test management tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states 'Create' which implies a write/mutation operation, but doesn't mention permissions required, whether the creation is idempotent, error handling, or what happens on success (e.g., returns a test case ID). For a mutation tool with zero annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with zero waste—it directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded, making it easy for an agent to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given this is a mutation tool with no annotations and no output schema, the description is incomplete. It doesn't explain behavioral aspects like permissions, side effects, or return values, which are critical for safe and effective use. The high schema coverage helps with parameters, but overall context is lacking for a creation operation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, meaning all parameters are documented in the input schema. The description adds no additional parameter information beyond what's in the schema, such as examples or constraints. With high schema coverage, the baseline score of 3 is appropriate as the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create a new test case') and resource ('in Xray Cloud'), which is specific and unambiguous. However, it doesn't differentiate this tool from its sibling 'update_test_case' or explain when to create versus update, missing the sibling distinction needed for a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'update_test_case' or other creation tools (e.g., 'create_test_plan'). There's no mention of prerequisites, such as needing an existing project, or exclusions, leaving the agent to infer usage from context alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. While 'Create' implies a write/mutation operation, the description doesn't disclose important behavioral traits like required permissions, whether this creates a draft or immediately executes tests, what happens to existing test executions, or any rate limits. It provides minimal context beyond the basic action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise - a single sentence that gets straight to the point without any unnecessary words. It's front-loaded with the core purpose and efficiently communicates the essential action. Every word earns its place in this minimal description.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no annotations and no output schema, the description is inadequate. It doesn't explain what happens after creation, what the tool returns, error conditions, or how this fits into the broader test execution workflow. Given the complexity of test execution systems and the lack of structured behavioral information, this description leaves significant gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 5 parameters thoroughly. The description doesn't add any meaningful parameter semantics beyond what's in the schema - it doesn't explain relationships between parameters, provide examples of valid combinations, or clarify edge cases. The baseline score of 3 reflects adequate but minimal value addition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create a new test execution') and the resource ('in Xray Cloud to run tests'), providing a specific verb+resource combination. However, it doesn't distinguish this tool from sibling tools like 'create_test_case', 'create_test_plan', or 'create_test_set' which also create different test artifacts in the same system.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance about when to use this tool versus alternatives. It doesn't mention when you would create a test execution versus a test plan or test set, nor does it specify prerequisites or contextual constraints for using this tool effectively.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. While 'Create' implies a write/mutation operation, it doesn't specify permission requirements, whether the creation is idempotent, what happens with duplicate names, or what the response contains. For a creation tool with zero annotation coverage, this leaves significant behavioral gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized and front-loaded with the core functionality.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a creation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what a successful creation returns, error conditions, or how the created test plan integrates with the broader Xray ecosystem. The description should provide more context about the operation's outcome and implications.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, providing complete documentation of all 4 parameters. The description doesn't add any parameter-specific information beyond what's already in the schema, so it meets the baseline expectation but doesn't enhance understanding of parameter usage or relationships.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create a new test plan') and resource ('in Xray Cloud to organize tests'), providing a specific verb+resource combination. However, it doesn't distinguish this tool from its sibling 'create_test_set' which also creates organizational structures for tests, missing explicit differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'create_test_set' or 'create_test_execution', nor does it mention prerequisites or context for creating test plans. It simply states what the tool does without usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action ('Delete') but doesn't clarify if this is permanent, requires specific permissions, affects related data (e.g., test executions), or has side effects like audit logging. For a destructive operation with zero annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise—a single, direct sentence with no wasted words. It front-loads the core action ('Delete') and resource, making it immediately scannable and efficient. Every word earns its place by conveying essential information without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's destructive nature, lack of annotations, and absence of an output schema, the description is incomplete. It doesn't address critical aspects like success/failure responses, error conditions, or the impact of deletion on the system. For a mutation tool with no structured safety hints, more context is needed to guide safe usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, with the single parameter 'testKey' fully documented in the schema (including an example). The description adds no additional parameter information beyond what the schema provides, such as format constraints or validation rules. This meets the baseline for high schema coverage but doesn't enhance understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Delete') and resource ('a test case'), making the purpose immediately understandable. It distinguishes itself from sibling tools like 'update_test_case' or 'get_test_case' by specifying the destructive action. However, it doesn't mention the specific system or context (e.g., Jira, test management platform), which prevents a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing test case), consequences (e.g., irreversible deletion), or when to choose other tools like 'update_test_case' for modifications instead. This leaves the agent with minimal contextual direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('Get') but doesn't describe whether this is a read-only operation, how results are returned (e.g., pagination, sorting), error conditions, or performance implications. This leaves significant gaps in understanding the tool's behavior beyond the basic action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is front-loaded and wastes no space, making it easy for an agent to parse quickly and understand the core functionality.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of annotations and output schema, the description is incomplete for a tool that retrieves data. It doesn't explain what 'all test cases' entails (e.g., format, fields returned, limitations) or how results are handled (e.g., list structure, pagination). For a data-fetching tool with no structured output information, this leaves the agent underinformed about what to expect.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, with clear descriptions for both parameters (projectKey and maxResults). The description adds no additional semantic context beyond what the schema provides, such as explaining the relationship between parameters or usage nuances. This meets the baseline for adequate but unenhanced parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Get') and resource ('all test cases for a specific project'), making the purpose understandable. However, it doesn't distinguish this tool from its sibling 'search_test_cases', which could also retrieve test cases, leaving some ambiguity about when to use one over the other.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'search_test_cases' or 'get_test_case'. The description implies usage for retrieving all test cases in a project, but it doesn't specify prerequisites, exclusions, or contextual best practices, leaving the agent to infer usage scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves data ('Get'), implying a read-only operation, but doesn't specify whether it requires authentication, has rate limits, returns paginated results, or what format the output takes. This leaves significant behavioral aspects unclear.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It is appropriately sized and front-loaded, making it easy to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of annotations and output schema, the description is incomplete for a tool that likely returns a list of test executions. It doesn't explain the return format, pagination behavior, error conditions, or how it differs from similar tools, leaving gaps in contextual understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, with clear documentation for both parameters in the input schema. The description adds no additional parameter semantics beyond what the schema already provides, such as clarifying the scope of 'projectKey' or usage of 'maxResults'. This meets the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get') and resource ('test executions for a specific project'), making the purpose immediately understandable. However, it doesn't distinguish this tool from sibling tools like 'search_test_executions' or 'get_test_execution', which limits its differentiation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'search_test_executions' or 'get_test_execution'. It lacks context about use cases, prerequisites, or exclusions, leaving the agent to infer usage from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It states this is a 'Get' operation, implying read-only behavior, but doesn't mention any constraints like pagination (despite the 'maxResults' parameter), rate limits, authentication requirements, or what happens if the project doesn't exist. The description adds minimal behavioral context beyond the basic action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized for a simple retrieval tool and front-loads the key information ('Get all test sets').
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'test sets' are in this context, how results are structured, or any error conditions. For a tool with two parameters and no structured output documentation, more context is needed to help an agent use it effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 100% description coverage, with clear documentation for both parameters ('projectKey' and 'maxResults'). The description doesn't add any meaningful parameter semantics beyond what's already in the schema—it mentions 'for a specific project' which aligns with 'projectKey' but provides no additional context about parameter usage or interactions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Get') and resource ('all test sets for a specific project'), making the purpose understandable. However, it doesn't explicitly distinguish this tool from sibling tools like 'get_test_set' (singular) or 'search_test_sets', leaving some ambiguity about when to use this specific list-retrieval tool versus alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'get_test_set' (for a single test set) or 'search_test_sets' (for filtered searches). It mentions 'for a specific project' but doesn't clarify if this is the only way to get test sets or when other tools might be more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. While 'Get details' suggests a read operation, it doesn't specify whether this requires authentication, what format/details are returned, if there are rate limits, or error conditions. For a tool with zero annotation coverage, this is inadequate behavioral transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that gets straight to the point with no wasted words. It's appropriately sized for a simple lookup tool and front-loads the essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with no annotations, no output schema, and multiple sibling tools that perform similar functions, the description is insufficiently complete. It doesn't explain what 'details' are returned, how this differs from other test case retrieval tools, or provide any context about authentication requirements or error handling.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% with the single parameter 'testKey' well-documented in the schema. The description adds minimal value beyond the schema by mentioning 'by key' which is already implied. With high schema coverage, the baseline score of 3 is appropriate as the description doesn't significantly enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get details') and resource ('specific test case by key'), making the purpose immediately understandable. However, it doesn't differentiate from siblings like 'search_test_cases' or 'get_project_test_cases' that also retrieve test case information, so it doesn't reach the highest clarity level.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'search_test_cases' or 'get_project_test_cases'. It mentions 'specific test case by key' which implies you need the exact key, but doesn't explicitly state this as a distinguishing factor or mention when other tools would be more appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It implies a read-only operation ('Get details'), but doesn't specify whether it requires authentication, has rate limits, returns paginated results, or what format the details include (e.g., JSON structure). For a tool with no annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. It directly states what the tool does ('Get details of a specific test plan by key, including all tests'), making it easy to parse and understand quickly. Every part of the sentence earns its place by specifying the action and scope.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of retrieving test plan details with no annotations and no output schema, the description is incomplete. It doesn't explain what 'details' include, how 'all tests' are represented, or potential error conditions. For a tool in a context with many sibling tools and no structured output, more completeness is needed to guide effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds minimal value beyond the input schema, which has 100% coverage for the single parameter 'testPlanKey.' It mentions 'by key' but doesn't elaborate on key format or validation beyond what the schema provides ('e.g., "PROJ-789"'). With high schema coverage, the baseline is 3, as the description doesn't compensate with additional semantic context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get details of a specific test plan by key, including all tests.' It specifies the verb ('Get details'), resource ('test plan'), and scope ('including all tests'), which is more specific than just restating the name. However, it doesn't explicitly distinguish this tool from sibling tools like 'get_test_case' or 'get_project_test_plans,' which prevents a score of 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_project_test_plans' (for listing multiple test plans) or 'search_test_plans' (for filtered searches), nor does it specify prerequisites or exclusions. This lack of contextual guidance leaves the agent to infer usage from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. It indicates a read operation ('Get details'), which implies it is non-destructive, but does not address permissions, rate limits, error handling, or response format. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior and constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It is front-loaded with the core action and includes essential scope information ('including all tests'), making it easy to parse and understand quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the lack of annotations and output schema, the description is insufficient for a complete understanding. It does not explain what 'details' include, how 'all tests' are structured in the response, or any behavioral aspects like error cases. For a tool with no structured metadata, more context is needed to guide effective usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description mentions retrieving details 'by key,' which aligns with the single parameter 'testSetKey' in the schema. Since schema description coverage is 100%, the schema already documents the parameter adequately. The description adds minimal value beyond the schema, such as clarifying that it fetches 'all tests,' but does not provide additional syntax or format details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get details of a specific test set by key, including all tests.' It specifies the verb ('Get'), resource ('test set'), and scope ('by key, including all tests'), which is specific and actionable. However, it does not explicitly differentiate from sibling tools like 'get_project_test_sets' or 'search_test_sets,' which might offer broader retrieval options.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It does not mention prerequisites, such as needing a valid test set key, or compare it to siblings like 'get_project_test_sets' for listing multiple sets or 'search_test_sets' for filtered searches. Without such context, an agent may struggle to select the appropriate tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool removes tests from a test plan, implying a destructive mutation, but doesn't address critical aspects like whether this requires specific permissions, if the removal is reversible, what happens to the removed tests (e.g., deletion vs. disassociation), or any rate limits. For a mutation tool with zero annotation coverage, this is a significant gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without any unnecessary words. It's appropriately sized and front-loaded, with every word earning its place. No structural issues or redundancy are present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity as a destructive mutation operation, the absence of annotations, and no output schema, the description is incomplete. It doesn't address behavioral traits (e.g., permissions, reversibility), output expectations, or error conditions. The agent lacks sufficient context to use this tool safely and effectively beyond basic parameter passing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema description coverage is 100%, with both parameters clearly documented in the input schema. The description doesn't add any semantic information beyond what's already in the schema (e.g., it doesn't clarify format examples, constraints, or relationships between parameters). According to the rules, when schema coverage is high (>80%), the baseline score is 3 even without parameter details in the description.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Remove tests') and target resource ('from an existing test plan'), providing a specific verb+resource combination. It distinguishes from sibling tools like 'remove_tests_from_test_set' by specifying the target is a test plan rather than a test set. However, it doesn't fully differentiate from other removal operations like 'delete_test_case' beyond the resource type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing test plan), exclusions, or compare with similar tools like 'remove_tests_from_test_set' or 'delete_test_case'. The agent must infer usage solely from the tool name and sibling list.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states a mutation action ('Remove') but doesn't clarify if this is destructive (e.g., permanently deletes tests or just unlinks them), requires specific permissions, has side effects (e.g., affects test executions), or provides any response details. This leaves critical behavioral traits unspecified for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's action without unnecessary words. It's front-loaded with the core purpose, and every part of the sentence earns its place by specifying the resource ('tests from an existing test set'). No fluff or redundancy is present.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given this is a mutation tool with no annotations and no output schema, the description is incomplete. It lacks details on behavioral traits (e.g., destructiveness, permissions), response format, error handling, or how it fits within the broader test management context. For a tool that modifies data, this leaves significant gaps for an AI agent to operate safely and effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both parameters ('testSetIssueId', 'testIssueIds') clearly documented in the schema. The description adds no additional meaning beyond what the schema provides (e.g., no examples, format clarifications, or constraints). Baseline 3 is appropriate as the schema does the heavy lifting, but the description doesn't enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Remove') and resource ('tests from an existing test set'), making the purpose immediately understandable. It distinguishes from siblings like 'remove_tests_from_test_plan' by specifying 'test set' rather than 'test plan'. However, it doesn't specify what type of tests (e.g., test cases) or how removal affects the test set, leaving some ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing an existing test set), exclusions (e.g., cannot remove all tests), or compare with siblings like 'delete_test_case' or 'update_test_case'. The agent must infer usage from the name and context alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but only states the action without behavioral details. It doesn't disclose whether this is read-only, potential rate limits, authentication needs, or what the return format looks like (e.g., pagination, error handling).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero waste. It's appropriately sized and front-loaded, clearly stating the tool's purpose without unnecessary elaboration.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations and no output schema, the description is incomplete for a search tool. It lacks details on behavioral traits, return values, or error handling, leaving significant gaps for an AI agent to understand how to use it effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters. The description adds no additional meaning beyond implying JQL usage, which is already covered in the schema's example. Baseline 3 is appropriate as the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Search') and resource ('test cases') with the method ('using JQL'). It distinguishes from siblings like 'get_project_test_cases' by specifying search functionality, though it doesn't explicitly contrast with 'search_test_executions' or other search siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'get_project_test_cases' or other search tools (e.g., 'search_test_executions'). The description implies usage for JQL-based searches but lacks explicit context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but lacks behavioral details. It doesn't disclose whether this is read-only, its rate limits, authentication needs, or what happens on execution (e.g., pagination, error handling). The mention of JQL adds some context but is insufficient for a mutation-aware tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero waste. It front-loads the core action and method, making it easy to parse quickly without unnecessary elaboration.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, no output schema, and a search tool with potential complexity (JQL queries), the description is incomplete. It doesn't explain return values, error cases, or behavioral traits, leaving gaps for an AI agent to infer usage safely.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds no additional meaning beyond implying JQL usage, which is covered in the schema's example. Baseline 3 is appropriate as the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Search for') and resource ('test executions'), specifying the method ('using JQL'). It distinguishes from siblings like 'get_project_test_executions' by emphasizing search functionality, though it doesn't explicitly contrast with 'search_test_cases' or 'search_test_plans'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'get_project_test_executions' or other search siblings. The description implies usage for JQL-based queries but offers no context on prerequisites, exclusions, or comparative scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It states the search uses JQL but doesn't describe what the tool returns (e.g., list of test plans, pagination behavior, error handling), whether it's read-only or has side effects, or any limitations like rate limits or authentication needs. This is a significant gap for a tool with potential complexity in JQL queries.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero waste. It's front-loaded with the core action and resource, making it easy to parse. Every word earns its place by specifying the search method (JQL).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of JQL-based search, no annotations, and no output schema, the description is incomplete. It doesn't explain return values, error cases, or behavioral traits like pagination. For a tool that could involve complex queries and varied results, more context is needed to guide effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters ('jql' and 'maxResults'). The description adds no additional meaning beyond what's in the schema, such as JQL syntax examples or constraints on 'maxResults'. The baseline score of 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb ('Search') and resource ('test plans'), and specifies the query language ('using JQL'). It distinguishes itself from siblings like 'get_project_test_plans' by emphasizing search functionality. However, it doesn't explicitly differentiate from other search tools like 'search_test_cases' or 'search_test_sets' beyond the resource name.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to prefer this over 'get_project_test_plans' (which likely retrieves all test plans in a project) or other search tools like 'search_test_cases'. There are no exclusions, prerequisites, or context for usage beyond the basic action.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions JQL usage but fails to describe key traits: whether this is a read-only operation, potential rate limits, authentication needs, or what the output looks like (e.g., pagination, error handling). For a search tool with zero annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence with zero wasted words. It front-loads the core purpose ('Search for test sets') and adds necessary detail ('using JQL') without redundancy. This is appropriately sized and well-structured for clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the complexity of a search operation with no annotations and no output schema, the description is incomplete. It doesn't explain return values, error conditions, or behavioral constraints, leaving the agent under-informed. For a tool with 2 parameters and significant sibling tools, more context is needed to ensure proper usage.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters ('jql' and 'maxResults'). The description adds no additional semantic context beyond implying JQL is used for searching, which is already covered in the schema. This meets the baseline of 3 when the schema handles parameter documentation effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Search for') and resource ('test sets'), and specifies the query language ('using JQL'). It distinguishes from siblings like 'get_project_test_sets' by indicating a search function rather than a direct retrieval. However, it doesn't explicitly differentiate from other search tools (e.g., 'search_test_cases'), making it a 4 rather than a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to prefer this over 'get_project_test_sets' or other search tools, nor does it specify prerequisites or exclusions. This lack of contextual direction leaves the agent with minimal usage cues.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It implies a mutation ('Update'), but doesn't specify permissions required, whether changes are reversible, rate limits, or what happens on success/failure. The example statuses ('PASS or FAIL') are helpful but don't cover the full enum range or behavioral outcomes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose. Every word earns its place, with no redundant information. The parenthetical example ('e.g., mark as PASS or FAIL') adds useful clarification without verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness2/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with no annotations and no output schema, the description is incomplete. It doesn't address behavioral aspects like error conditions, side effects, or response format. Given the complexity of updating test run statuses (which could affect workflows), more context on implications and usage would be beneficial.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters (testRunId and status with enum). The description adds minimal value beyond the schema—it mentions status examples ('PASS or FAIL') but doesn't explain parameter interactions or provide additional context like format requirements for testRunId. Baseline 3 is appropriate given high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Update the status') and resource ('a specific test run'), with examples of status values ('PASS or FAIL'). It distinguishes from siblings like 'update_test_case' by focusing on status updates rather than general test case modifications. However, it doesn't explicitly differentiate from all mutation tools like 'create_test_execution'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a test run ID from execution details), exclusions, or relationships with sibling tools like 'create_test_execution' or 'update_test_case'. The description only states what it does, not when it's appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It states it 'creates' a new test set, implying a write/mutation operation, but doesn't disclose behavioral traits such as permissions required, whether it's idempotent, rate limits, or what happens on failure. For a creation tool with zero annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose. Every word earns its place with no waste, making it highly concise and well-structured for quick understanding.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no annotations, no output schema, and a creation tool with 4 parameters, the description is minimally adequate. It states what the tool does but lacks details on behavior, output, or usage context. It's complete enough to understand the basic purpose but insufficient for safe or effective use without external knowledge.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents all 4 parameters. The description adds no additional meaning beyond what's in the schema (e.g., it doesn't explain parameter interactions or provide examples). Baseline score of 3 is appropriate as the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Create a new test set') and resource ('in Xray Cloud'), with the purpose 'to group related tests'. It distinguishes from siblings like 'create_test_plan' or 'create_test_case' by specifying it's for test sets, but doesn't explicitly contrast with them. The purpose is specific and actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'create_test_plan' or 'add_tests_to_test_set'. The description implies usage for grouping tests, but lacks explicit context, prerequisites, or exclusions. It's a basic statement without operational guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden but only states the basic action without behavioral details. It doesn't mention whether this is a read-only operation, if it requires specific permissions, how results are ordered or paginated, or what the output format looks like (e.g., list of objects with fields). For a tool with no annotation coverage, this leaves significant gaps in understanding its behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that directly states the tool's purpose without unnecessary words. It's appropriately sized for a simple retrieval tool and front-loaded with the core action, making it easy to parse quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (simple retrieval with 2 parameters) and 100% schema coverage, the description is minimally adequate but lacks context about output format, pagination, or behavioral traits. Without annotations or an output schema, the description should ideally mention what the return value contains (e.g., list of test plan objects) to be more complete, but it's not critically incomplete for this use case.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema fully documents both parameters (projectKey and maxResults). The description adds no additional meaning beyond implying the tool uses 'projectKey' to filter results, which is already clear from the schema. This meets the baseline of 3 when the schema does the heavy lifting, but the description doesn't enhance parameter understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get') and resource ('all test plans for a specific project'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'get_test_plan' (singular) or 'search_test_plans', which could cause confusion about when to use this specific list-retrieval tool versus alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like 'get_test_plan' (for a single test plan) or 'search_test_plans' (for filtered searches). It mentions 'for a specific project' but doesn't clarify if this is the only way to get test plans by project or if other tools serve similar purposes with different scopes or filters.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden of behavioral disclosure. While it indicates this is a read operation ('Get details'), it doesn't cover critical aspects like authentication needs, rate limits, error handling, or what 'details' specifically include (e.g., format, depth). For a tool with no annotation coverage, this is a significant gap in transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. It directly states what the tool does and includes a key detail ('including all test runs'), making it appropriately sized and well-structured for quick comprehension.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness3/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (1 parameter, no nested objects) and high schema coverage (100%), the description is minimally adequate. However, with no annotations and no output schema, it fails to fully compensate for missing behavioral and output details. A more complete description would clarify the return format or usage context, but it meets basic requirements for a simple lookup tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters3/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description mentions 'by key', which aligns with the single parameter 'testExecutionKey' in the schema. Since schema description coverage is 100% (the parameter is fully documented in the schema), the description adds minimal value beyond what's already structured. The baseline score of 3 reflects adequate but not enhanced parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose4/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get details of a specific test execution by key, including all test runs'. It specifies the verb ('Get'), resource ('test execution'), and scope ('by key, including all test runs'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'get_project_test_executions' or 'search_test_executions', which prevents a perfect score.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_project_test_executions' (for listing multiple executions) or 'search_test_executions' (for filtered searches), nor does it specify prerequisites or exclusions. This lack of context leaves the agent to infer usage from the tool name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

GitHub Badge

Glama performs regular codebase and documentation scans to:

Confirm that the MCP server is working as expected.
Confirm that there are no obvious security issues.
Evaluate tool definition quality.

Our badge communicates server capabilities, safety, and installation instructions.

Card Badge

Copy to your README.md:

[![xray-mcp MCP server](https://glama.ai/mcp/servers/c4m3lblue-star/xray-mcp/badges/card.svg)](https://glama.ai/mcp/servers/c4m3lblue-star/xray-mcp)

Score Badge

Copy to your README.md:

[![xray-mcp MCP server](https://glama.ai/mcp/servers/c4m3lblue-star/xray-mcp/badges/score.svg)](https://glama.ai/mcp/servers/c4m3lblue-star/xray-mcp)

Latest Blog Posts

Who's Calling? MCP Hosts Are an Identity Blind Spot (And the Spec Knows It)
By Om-Shree-0709 on July 25, 2026.
mcp
Agent Identity
OAuth 2.1
Your AI Chatbot Just Exposed Your CEO's Salary to an Intern
By Om-Shree-0709 on July 2, 2026.
Agent Identity
MCP Security
OAuth Delegation
Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)
By Om-Shree-0709 on June 30, 2026.
Agentic Ai
Prompt Injection
WebAssembly

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/c4m3lblue-star/xray-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server