Xray MCP
Integrates with Jira and Xray Test Management to manage tests, test executions, test plans, test sets, and import/export test results.
Allows import of Cucumber JSON test results and export of Cucumber feature files from Xray.
Allows import of Robot Framework XML test results into Xray.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Xray MCPList test executions in project PROJ"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Xray MCP
A Model Context Protocol (MCP) server that integrates Cursor IDE with Xray Test Management for Jira, allowing you to manage tests, test executions, and test plans directly from your development environment.
Features
Tests
List Tests: Fetch all tests from your Jira project with filtering options
Get Test Details: Retrieve detailed test information including test steps via GraphQL āØ
Get Test With Steps: NEW! Reliably fetch test details with steps using the GraphQL
getTestsquery šCreate Test: Create new manual or automated tests
Update Test: Modify existing tests
Test Executions
List Test Executions: View all test executions in a project
Get Test Execution Details: See detailed execution results including test runs
Create Test Execution: Create new test execution sessions
Update Test Run: Record test results (PASS, FAIL, TODO, EXECUTING, ABORTED)
Test Plans
List Test Plans: View all test plans in a project
Get Test Plan Details: See test plan information with associated tests
Create Test Plan: Create new test plans
Add Tests to Test Plan: Associate tests with existing plans
Test Sets
List Test Sets: View all test sets in a project
Get Test Set Details: See test set information with associated tests
Import Operations (CI/CD Integration) š NEW!
Import Xray JSON Results: Import test execution results in Xray JSON format
Import Cucumber Results: Import Cucumber JSON test results
Import JUnit Results: Import JUnit XML test results
Import TestNG Results: Import TestNG XML test results
Import NUnit Results: Import NUnit XML test results
Import Robot Framework Results: Import Robot Framework XML test results
Import Behave Results: Import Behave JSON test results
Import Feature Files: Import Cucumber .feature files (BDD scenarios)
Export Operations š¤ NEW!
Export Cucumber Features: Export Cucumber feature files from Xray
Related MCP server: mcp-zephyr-scale
Prerequisites
Node.js 18 or higher
npm or yarn
Jira Cloud account with Xray Test Management plugin installed
Jira API token for authentication
Installation
Option 1: Install from npm (Recommended)
The package is available on npm and can be used directly with npx:
npx @korfu/xray-mcpNo installation required! Continue to Generate Jira API Token section.
Option 2: Local Development Setup
If you want to contribute or modify the code:
1. Clone the Repository
git clone https://github.com/Korfu/mcp-xray.git
cd mcp-xray2. Install Dependencies
npm install3. Build the Project
npm run buildSetup
Generate Jira API Token
Click "Create API token"
Give it a descriptive name (e.g., "Xray MCP Integration")
Copy the generated token - you won't be able to see it again!
4. Configure Environment Variables
You'll need the following environment variables:
Required:
JIRA_BASE_URL: Your Jira instance URL (e.g.,https://your-domain.atlassian.net)JIRA_EMAIL: Your Jira account emailJIRA_API_TOKEN: The API token from step 3
Optional (for test steps):
XRAY_CLIENT_ID: Your Xray Cloud API client IDXRAY_CLIENT_SECRET: Your Xray Cloud API client secret
4.1. Get Xray Cloud API Credentials (Optional - for Test Steps)
To fetch test steps from Xray, you need Xray Cloud API credentials:
Log in to your Jira instance
Go to Settings (āļø) ā Apps ā Manage apps
In the left sidebar, find Xray section
Click on API Keys (or Cloud API)
Click Create API Key
Give it a name (e.g., "MCP Integration")
Copy the Client ID and Client Secret
Save these securely - you won't be able to see the secret again!
Note: Without Xray Cloud API credentials, the MCP will still work for listing and managing tests, but test steps won't be fetched.
5. Add to Cursor MCP Configuration
Add the following configuration to your Cursor MCP settings file (usually at ~/.cursor/mcp.json or similar):
For npm installation (recommended):
{
"mcpServers": {
"xray": {
"command": "npx",
"args": ["-y", "@korfu/xray-mcp"],
"env": {
"JIRA_BASE_URL": "https://your-domain.atlassian.net",
"JIRA_EMAIL": "your-email@example.com",
"JIRA_API_TOKEN": "your-api-token-here",
"XRAY_CLIENT_ID": "your-xray-client-id",
"XRAY_CLIENT_SECRET": "your-xray-client-secret"
}
}
}
}Important:
Replace the values with your actual credentials
XRAY_CLIENT_IDandXRAY_CLIENT_SECRETare optional - omit them if you don't need test step details
6. Restart Cursor
After adding the configuration, restart Cursor IDE to load the MCP server.
Usage
Once configured, you can use the following tools in Cursor:
Tests
list_tests- List all tests in a projectget_test- Get detailed test information (may not always return steps)get_test_with_steps- NEW! Get detailed test information with steps (uses reliable GraphQL query)create_test- Create a new testupdate_test- Update an existing test
Test Executions
list_test_executions- List test executionsget_test_execution- Get test execution detailscreate_test_execution- Create a new test executionupdate_test_run- Update test run status
Test Plans
list_test_plans- List all test plansget_test_plan- Get test plan detailscreate_test_plan- Create a new test planadd_tests_to_test_plan- Add tests to a plan
Test Sets
list_test_sets- List all test setsget_test_set- Get test set details
Import Operations š NEW!
import_execution_results- Import Xray JSON format resultsimport_cucumber_results- Import Cucumber JSON resultsimport_junit_results- Import JUnit XML resultsimport_testng_results- Import TestNG XML resultsimport_nunit_results- Import NUnit XML resultsimport_robot_results- Import Robot Framework XML resultsimport_behave_results- Import Behave JSON resultsimport_feature_file- Import Cucumber .feature files
Export Operations š¤ NEW!
export_cucumber_features- Export Cucumber features as .feature files
Examples
Basic Operations
Get Test with Steps (RECOMMENDED)
Get test EXM-123 with stepsThis uses the new get_test_with_steps tool which reliably fetches test steps.
Output includes test steps:
**Test: EXM-123**
**Summary:** Verify user login functionality
**Test Type:** Manual
**Test Steps:**
**Step 1:**
- **Action:** Navigate to the login page
- **Data:** URL: https://example.com/login
- **Expected Result:** Login page is displayed
**Step 2:**
- **Action:** Enter valid credentials and click Submit
- **Data:** Username: testuser, Password: ********
- **Expected Result:** User is successfully authenticated
**Step 3:**
- **Action:** Verify user dashboard is displayed
- **Data:** N/A
- **Expected Result:** Dashboard shows user profile and navigation menuCI/CD Integration Examples š
Import JUnit Results
Import JUnit test results from file test-results/junit.xmlImport Cucumber Results
Import Cucumber results from file cucumber-report.jsonExport Feature Files
Export Cucumber features for tests EXM-1,EXM-2,EXM-3Legacy Examples
List Tests in a Project
List all tests in project PROJ with label "regression"Create a Test
Create a new manual test in project PROJ with summary "Login validation test" and description "Verify user can login with valid credentials"Update Test Run Status
Update test run for test PROJ-123 in execution PROJ-456 with status PASS and comment "All assertions passed"Create Test Execution
Create a test execution in project PROJ with summary "Sprint 5 Regression" and add tests PROJ-123, PROJ-124, PROJ-125Development
Running in Development Mode
npm run devBuilding
npm run buildProject Structure
XRayMCP/
āāā src/
ā āāā index.ts # Main MCP server
ā āāā types.ts # TypeScript interfaces
ā āāā services/
ā ā āāā XrayCloudService.ts # Xray Cloud API service (GraphQL + REST)
ā āāā tools/
ā āāā tests/ # Test operations
ā ā āāā listTests.ts
ā ā āāā getTest.ts
ā ā āāā createTest.ts
ā ā āāā updateTest.ts
ā āāā test-executions/ # Test execution operations
ā ā āāā listTestExecutions.ts
ā ā āāā getTestExecution.ts
ā ā āāā createTestExecution.ts
ā ā āāā updateTestRun.ts
ā āāā test-plans/ # Test plan operations
ā ā āāā listTestPlans.ts
ā ā āāā getTestPlan.ts
ā ā āāā createTestPlan.ts
ā ā āāā addTestsToTestPlan.ts
ā āāā test-sets/ # Test set operations
ā ā āāā listTestSets.ts
ā ā āāā getTestSet.ts
ā āāā import/ # Import operations (NEW!)
ā ā āāā importExecutionResults.ts
ā ā āāā importCucumberResults.ts
ā ā āāā importJUnitResults.ts
ā ā āāā importTestNGResults.ts
ā ā āāā importNUnitResults.ts
ā ā āāā importRobotResults.ts
ā ā āāā importBehaveResults.ts
ā ā āāā importFeatureFile.ts
ā āāā export/ # Export operations (NEW!)
ā āāā exportCucumberFeatures.ts
āāā dist/ # Compiled JavaScript output
āāā XRAY_CLOUD_API_V2_ENDPOINTS.md # API documentation
āāā XRAY_CLOUD_V2_IMPLEMENTATION.md # Implementation details
āāā package.json
āāā tsconfig.json
āāā README.mdAPI Architecture
Xray Cloud API v2 - Dual Approach
This MCP server uses two complementary APIs provided by Xray Cloud:
1. GraphQL API (for Queries)
Endpoint:
https://xray.cloud.getxray.app/api/v2/graphqlUsed For: Fetching test details with steps
Why: Xray Cloud exposes test steps only via GraphQL, not REST
query {
getTest(issueId: "EXM-123") {
issueId
steps {
id
action
data
result
}
}
}2. REST API v2 (for Import/Export)
Base URL:
https://xray.cloud.getxray.app/api/v2Used For: Importing test results, exporting features
Endpoints:
POST /import/execution/*- Import results (multiple formats)POST /import/feature- Import BDD featuresGET /export/cucumber- Export BDD features
Authentication
All operations use JWT Bearer tokens obtained via:
POST /api/v2/authenticate
Body: { client_id, client_secret }
Response: "eyJhbGciOiJIUzI1NiI..."Tokens are:
Cached for 50 minutes
Automatically refreshed
Stored in memory only (not persisted)
Troubleshooting
Authentication Issues
Verify your Jira email and API token are correct
Ensure the API token has the correct permissions
Check that your Jira base URL is correct (should not end with a slash)
"Test issue type not found" Error
Ensure Xray is installed in your Jira instance
Verify the project has Xray enabled
Check that you have permission to create issues in the project
Connection Issues
Verify your internet connection
Check if your Jira instance is accessible
Ensure the MCP server is running (check Cursor's MCP logs)
Test Not Found
Verify the test key is correct (e.g., PROJ-123)
Ensure you have access to the test
Check that the test exists in Jira
MCP Server Not Loading
Verify the path to
dist/index.jsis correct in your Cursor configurationEnsure Node.js is installed and accessible from the command line
Check Cursor's MCP server logs for error messages
Make sure you've run
npm run buildto compile the TypeScript code
Test Steps Not Appearing
Ensure you've configured
XRAY_CLIENT_IDandXRAY_CLIENT_SECRETin your MCP configurationVerify your Xray Cloud API credentials are correct
Check that your Xray license includes API access
The test must be a Manual test type (Cucumber and Generic tests have different step formats)
Check Cursor's MCP logs for authentication errors
API Endpoints Used
This MCP uses the following Jira and Xray API endpoints:
Jira REST API v3:
/rest/api/3/search/jql,/rest/api/3/issue/*for general issue operationsXray Cloud API v2:
https://xray.cloud.getxray.app/api/v2/*for test steps and detailed test information (optional)
Security Notes
Keep your Jira API token secure and never commit it to version control
Keep your Xray Cloud API credentials (Client ID and Secret) secure
Use environment variables or secure configuration management for credentials
The API token should only have the minimum required permissions
Consider rotating your API tokens regularly
Xray API credentials are separate from Jira credentials and provide access to test data
Contributing
Fork the repository
Create a feature branch
Make your changes
Add tests if applicable
Submit a pull request
Related Resources
License
MIT License - see LICENSE file for details
Author
Korfu
Acknowledgments
Built following the Model Context Protocol specification
Inspired by the BitbucketMCP implementation pattern
Uses Xray Test Management for Jira by Xblend
Available Tools
24 toolsadd_tests_to_test_planB
Add tests to an existing test plan
| Name | Required | Description | Default |
|---|---|---|---|
| test_plan_key | Yes | Test Plan issue key (e.g., PROJ-789) | |
| test_keys | Yes | Comma-separated test keys to add |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, and the description only states 'Add tests' without disclosing behavioral traits like whether it appends or overwrites, error handling, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise (one sentence) but lacks structure; it could include more detail without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema and the description does not explain return values, prerequisites, or behavior after execution, leaving the agent with incomplete context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with both parameters described clearly (e.g., example format). The description adds no extra meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (add tests) and the target (existing test plan), distinguishing it from sibling tools like create_test_plan which creates a new plan.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives, no prerequisites (e.g., test plan must exist) or conditions for adding tests.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_testC
Create a new test in Jira with Xray
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| summary | Yes | Test summary/title | |
| description | No | Test description (optional) | |
| test_type | No | Test type: Manual, Cucumber, or Generic (default: Manual) | Manual |
| labels | No | Comma-separated labels (optional) | |
| priority | No | Priority name (e.g., High, Medium, Low) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations supplied, the description must fully convey behavioral traits. It only states the action without mentioning required permissions, side effects (e.g., data creation), success/failure responses, or any other behaviors. This is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The single-sentence description is concise but too brief; it lacks necessary usage and behavioral details. While not verbose, it could be expanded to be more helpful without being wasteful.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters, no output schema, and no annotations, the description should provide more context about return values, side effects, and appropriate use. It only offers a minimal statement, leaving significant gaps for an AI agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter has a schema description. The description adds no additional meaning beyond the schema, meeting the baseline of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Create a new test in Jira with Xray' clearly identifies the action (create) and the resource (test), and distinguishes from sibling tools like create_test_plan or create_test_execution by specifying 'test' and referencing Xray.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives such as creating a test execution or importing results. It lacks context about prerequisites, scenarios, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_test_executionC
Create a new test execution in Jira with Xray
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| summary | Yes | Test execution summary/title | |
| description | No | Test execution description (optional) | |
| test_plan_key | No | Test plan key to associate with (optional) | |
| test_environments | No | Comma-separated test environments (optional) | |
| tests | No | Comma-separated test keys to add to execution (optional) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description only indicates creation, which implies mutation, but provides no details on side effects, permissions required, or return behavior. With no annotations, the description fails to disclose any behavioral traits beyond the basic action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with a single sentence, but lacks structure such as bullet points or sections. While short, it may be under-specified for a tool with 6 parameters.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters and no output schema or annotations, the description fails to provide context on return values, behavior, or usage workflow. It only gives the basic purpose, leaving significant gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 6 parameters have descriptions in the input schema (100% coverage), so the schema already provides meaning. The description adds no additional context to the parameters, matching the baseline of 3.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (Create) and the resource (test execution in Jira with Xray). It distinguishes itself from sibling tools like create_test and create_test_plan by specifying 'test execution'. However, it could be more specific about the scope and purpose of a test execution.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives such as import_execution_results or add_tests_to_test_plan. There is no mention of prerequisites, context, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_test_planB
Create a new test plan in Jira with Xray
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| summary | Yes | Test plan summary/title | |
| description | No | Test plan description (optional) | |
| tests | No | Comma-separated test keys to add to plan (optional) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully convey behavioral traits. While it implies mutation, it omits details like required permissions, idempotency, side effects, or confirmation of success. The basic verb 'create' is insufficient.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence with no redundancy or fluff. It avoids tautology and earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a creation tool with 4 parameters and no output schema, the description is minimal. It fails to explain return values, error handling, or required context, leaving the agent underinformed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, providing adequate descriptions for all parameters. The description adds no additional meaning beyond what the schema already states, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Create a new test plan in Jira with Xray' clearly states the action (create), resource (test plan), and context (Jira/Xray). It effectively distinguishes from sibling tools like get_test_plan (read) and add_tests_to_test_plan (modify).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives like create_test or create_test_execution. No prerequisites, exclusions, or context for when to choose this tool are given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
export_cucumber_featuresA
Export Cucumber feature files from Xray. Can export all features or specific test keys.
| Name | Required | Description | Default |
|---|---|---|---|
| test_keys | No | Optional: Comma-separated list of test keys to export (e.g., "EXM-1,EXM-2"). If not provided, exports all features from the project. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must cover behavioral traits. It merely states the export action without disclosing whether it is read-only, the output format, or any side effects. This leaves the agent uncertain about the operation's safety.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description consists of two short sentences with no redundant information. It is front-loaded with the core purpose and efficiently covers the main usage options.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simple parameters and no output schema, the description should explain what the tool returns (e.g., feature file content or URIs) and whether it is synchronous. It omits these essential details, leaving the agent with an incomplete picture.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The parameter test_keys is fully described in the schema with a clear description. The tool description adds no additional meaning beyond what the schema already provides, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Export' and the resource 'Cucumber feature files from Xray', distinguishing between exporting all features or specific test keys. This differentiates it from sibling tools like import_feature_file.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains that the tool can export all features or specific test keys, providing clear usage context. However, it lacks explicit when-not-to-use guidance or alternatives, such as noting that import_feature_file is the appropriate tool for importing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_testA
Get detailed information about a specific test by its key
| Name | Required | Description | Default |
|---|---|---|---|
| test_key | Yes | Test issue key (e.g., PROJ-123) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavior. It states 'detailed information' but does not specify what fields or structure are returned, leaving ambiguity for the agent.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence of 11 words with no redundancy. All information is front-loaded and concise.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simplicity (1 required param, no output schema), the description is minimally sufficient but does not describe the return format, leaving the agent without full expectations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the parameter description is adequate. The description repeats the schema's point that the test is identified by key, adding no new meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Get', the resource 'test', and the scope 'by its key'. This distinguishes it from siblings like list_tests or create_test.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when a test key is available, but does not provide explicit when-to-use or when-not-to-use guidance compared to alternative tools like get_test_with_steps.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_executionB
Get detailed information about a test execution including test run results
| Name | Required | Description | Default |
|---|---|---|---|
| test_execution_key | Yes | Test Execution issue key (e.g., PROJ-456) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It mentions returning 'detailed information' and 'test run results', but omits aspects like read-only nature, potential errors, or any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence, concise and directly to the point. However, it could be more structurally informative with bullet points or explicit callouts.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a simple 1-parameter tool with no output schema, the description is adequate but not thorough. It mentions 'test run results' but does not elaborate on the output structure, which would help agents.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters with descriptions (e.g., 'Test Execution issue key (e.g., PROJ-456)'). The description does not add additional meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Get') and the resource ('test execution'), and specifies the output includes 'test run results'. It distinguishes from sibling tools like get_test or get_test_plan by focusing on executions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as list_test_executions or get_test. The description lacks context about prerequisites or preferred scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_planA
Get detailed information about a test plan including associated tests
| Name | Required | Description | Default |
|---|---|---|---|
| test_plan_key | Yes | Test Plan issue key (e.g., PROJ-789) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It mentions retrieving detailed information but does not clarify side effects, required permissions, data limits, or the exact structure of the output. This lack of transparency is a gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence with no redundant information. It is appropriately front-loaded and concise, earning its place without wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description gives the core purpose and a hint about output (including associated tests). However, with no annotations, more context about the return value structure would be beneficial. It is minimally adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides a clear description for test_plan_key (e.g., PROJ-789) with 100% coverage. The description adds that the output includes associated tests, but this is more about output than parameter semantics. The schema already fully documents the parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly uses the verb 'Get' with the resource 'test plan' and specifies the scope 'detailed information including associated tests'. This distinguishes it from sibling tools like list_test_plans, which suggest a listing function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for obtaining detailed information about a specific test plan, but does not explicitly state when to use this tool vs alternatives like list_test_plans. The guidance is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_setA
Get detailed information about a test set including associated tests
| Name | Required | Description | Default |
|---|---|---|---|
| test_set_key | Yes | Test Set issue key (e.g., PROJ-999) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided; description only implies read-only behavior via 'Get detailed information'. Lacks disclosure on side effects, authentication, or other behavioral constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence clearly conveys purpose without waste. Front-loaded with essential information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description adequately hints at return content ('detailed information', 'associated tests'), though more specifics could help.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with one parameter; description adds no extra meaning beyond the schema's own description. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves detailed information about a test set including associated tests, distinguishing it from sibling tools like get_test, get_test_execution, etc.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage from the description: when a user needs test set details. No explicit when-to-use or when-not-to-use guidance, nor alternatives mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_with_stepsA
Get detailed test information with test steps using the reliable GraphQL getTests query. This tool always fetches test steps if they exist.
| Name | Required | Description | Default |
|---|---|---|---|
| test_key | Yes | Test issue key (e.g., EXM-123) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses a key behavioral trait: 'always fetches test steps if they exist'. However, it does not cover error handling, performance, or what happens when the test key is invalid. Since no annotations are provided, the description partially fulfills transparency but lacks completeness.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise with two sentences that front-load the purpose and follow with a behavioral note. Every sentence adds value, and there is no redundant or irrelevant content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one required parameter, no output schema, no annotations), the description provides minimal context. It does not explain the return format or behavior when steps are absent. While adequate for basic use, it could be more complete for an agent to fully understand outcomes.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter 'test_key', which is well-documented in the schema with an example. The description adds no additional semantic information about the parameter, so it meets the baseline for high schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Get detailed test information with test steps'. It explicitly mentions the resource (test info) and action (get with steps), and distinguishes from sibling tools like 'get_test' by specifying the inclusion of test steps.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when test steps are needed by stating 'always fetches test steps if they exist', but it does not explicitly advise when to use this tool versus alternatives like 'get_test'. No exclusions or context are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_behave_resultsA
Import Behave JSON test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| behave_json | Yes | Behave JSON results as a string or JSON array |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry full burden. It mentions side effects (creates, updates) but lacks details on permissions, reversibility, or error conditions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short, front-loaded sentences with no unnecessary words. Efficient and clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Adequate for a simple single-parameter import tool, but lacks details on expected JSON structure, error handling, or prerequisites. No output schema, so return value expectations are omitted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter is well-described in the schema. The description adds minimal value beyond restating the resource type ('Behave JSON test results').
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'import', the resource 'Behave JSON test results', and the outcomes ('creates test execution and updates test statuses'). It distinguishes from sibling tools like import_cucumber_results by specifying the framework.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for Behave framework results but provides no explicit when-to-use or when-not-to-use guidance. No alternatives are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_cucumber_resultsA
Import Cucumber JSON test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| cucumber_json | Yes | Cucumber JSON results as a string or JSON array |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses creation of test execution and status updates, which is beyond the schema. No annotations are provided, so the description carries full burden. However, it omits details like what prevents duplicate executions or required test case existence.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with action and outcome. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-param import tool, the description is reasonably complete. It covers purpose and side effects, but lacks error conditions or behavior on repeated calls. No output schema, so return value is implicit.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear description for the single parameter. The tool description adds no additional parameter meaning beyond what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the verb 'import', the resource 'Cucumber JSON test results', and the target 'Xray'. Also describes side effects (creates test execution, updates statuses). Distinguishes from siblings by specifying Cucumber format.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this vs. alternatives. The name implies it's for Cucumber JSON, but there is no mention of prerequisites, limitations, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_execution_resultsA
Import test execution results in Xray JSON format. Creates test executions and test runs with results.
| Name | Required | Description | Default |
|---|---|---|---|
| results_json | Yes | Xray JSON format execution results as a JSON string. Example: {"testExecutionKey": "EXM-789", "tests": [{"testKey": "EXM-1", "status": "PASS"}]} |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must disclose behavioral traits. It only states creation of test executions and runs, but omits details like whether the import is additive or overwrites existing data, any required permissions, or validation behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two concise sentences that front-load the purpose. No redundant information; every word contributes.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity (one parameter, no output schema), the description covers the essential purpose and format. However, it lacks details like the effect on existing data or error handling, but overall is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for the single parameter. The description adds an example and specifies the Xray JSON format, enhancing understanding beyond the schema's description. While the schema already describes the parameter, the description provides practical context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: importing test execution results in Xray JSON format and creating test executions and runs. It uses a specific verb 'Import' and resource 'execution results', distinguishing it from sibling tools like import_junit_results or import_cucumber_results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for Xray JSON format but does not explicitly state when to use vs alternatives, nor does it mention any exclusions or prerequisites. Usage is implied but not robust.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_feature_fileA
Import Cucumber .feature file to Xray. Creates or updates BDD tests from Gherkin scenarios.
| Name | Required | Description | Default |
|---|---|---|---|
| feature_content | Yes | Content of the .feature file (Gherkin syntax) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions mutation ('Creates or updates') but does not explain idempotency, whether existing tests are overwritten, what happens to associated artifacts, or authorization requirements. The agent is left to infer behavior from the minimal description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is highly concise: two sentences front-loaded with the main action and then the effect. No unnecessary words or repetition. Every sentence serves a clear purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with 1 parameter and no output schema, the description provides the core purpose but lacks details on expected return values (e.g., test IDs, success/failure indicators) and edge cases (e.g., handling multiple scenarios). Given the tool's simplicity, a baseline of 3 is reasonable, but additional behavioral context would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (single parameter 'feature_content' described as 'Content of the .feature file (Gherkin syntax)'). The description adds context by stating the effect on BDD tests, but does not provide additional details on the parameter format or constraints beyond what the schema already provides. Baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Import', the resource 'Cucumber .feature file', and the target 'Xray'. It explains the effect: 'Creates or updates BDD tests from Gherkin scenarios.' This distinguishes it from sibling tools like export_cucumber_features (opposite direction) and import_cucumber_results (imports results, not features).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for importing feature files to create/update BDD tests, but it provides no explicit guidance on when to use this tool versus alternatives. For example, it does not differentiate from import_cucumber_results (which imports execution results) or export_cucumber_features. An explicit 'use when you have .feature content' and 'not for importing test results' would improve clarity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_junit_resultsA
Import JUnit XML test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| junit_xml | Yes | JUnit XML results as a string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses side effects (creates test execution, updates statuses) but no details on error handling, authentication needs, or whether existing results are overwritten. No annotations provided, so description carries full burden.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences with no superfluous information. Each sentence adds value: first states primary action, second clarifies side effects.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given low complexity (1 param, no output schema, no annotations), the description covers purpose and behavior adequately. Could mention return value (e.g., execution ID) but not essential for selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with description 'JUnit XML results as a string'. Description repeats the format but adds no extra semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the verb 'Import', the resource 'JUnit XML test results', and the target 'Xray'. Also specifies automatic creation of test execution and status updates, distinguishing from sibling import tools (e.g., import_cucumber_results).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implied usage through format-specific naming but lacks explicit guidance on when to use this tool vs alternatives (e.g., for other formats) or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_nunit_resultsA
Import NUnit XML test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| nunit_xml | Yes | NUnit XML results as a string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses key side effects: automatically creates test execution and updates test statuses. However, with no annotations, it fails to cover permissions, reversibility, or error behavior. Adds some context beyond the bare schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single, well-structured sentence that immediately conveys the tool's purpose and side effects. No extraneous words or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with one parameter and no output schema, the description covers the essential function and side effects. Lacks details on required permissions or error handling, but meets the minimum for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% description coverage for the single parameter 'nunit_xml'. The description's mention of 'NUnit XML test results' adds minimal meaning beyond the schema's 'NUnit XML results as a string'. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the verb 'Import', the resource 'NUnit XML test results', and the target 'Xray'. It distinguishes from similar sibling tools like import_junit_results by specifying the NUnit format.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives (e.g., import_junit_results, import_cucumber_results). It lacks prerequisites, context for selection, or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_robot_resultsA
Import Robot Framework XML test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| robot_xml | Yes | Robot Framework XML results (output.xml) as a string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses side effects (creates execution, updates statuses) but no annotations exist. Lacks details on permissions, data overwrite, or other behavioral traits, leaving moderate gap.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences front-loading the purpose and behavior, no wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-param tool with no output schema, the description covers the main function and side effects adequately, though could mention prerequisites like project context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with a clear parameter description. The description does not add extra meaning beyond the schema, meeting the baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the verb 'Import', the resource 'Robot Framework XML test results', and the effects 'creates test execution and updates test statuses', distinguishing it from sibling import tools like import_cucumber_results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use this tool vs alternatives. While the name implies Robot Framework results, the description does not provide usage context or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
import_testng_resultsB
Import TestNG XML test results to Xray. Automatically creates test execution and updates test statuses.
| Name | Required | Description | Default |
|---|---|---|---|
| testng_xml | Yes | TestNG XML results as a string |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavioral traits. It mentions automatic creation of test execution and status updates, but lacks details on side effects (e.g., what if XML is invalid, idempotency, required permissions, or whether existing data is overwritten). This is a significant gap for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences and no wasted words. It could be slightly improved by structuring with bullet points, but it remains concise and readable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given no output schema and only one parameter, the description should cover expected outcomes on success/failure and any constraints (e.g., file size). It only states basic behavior, leaving important context unclear.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the single parameter is described as 'TestNG XML results as a string'. The description adds no extra meaning beyond the schema, so a baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'Import' and the resource 'TestNG XML test results to Xray', and specifies two key outcomes: creating a test execution and updating test statuses. This effectively distinguishes it from sibling tools like import_cucumber_results or import_junit_results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for TestNG result import but provides no explicit when-to-use guidance or alternatives. Among many import siblings, it would benefit from stating that this is specifically for TestNG XML rather than other formats.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_executionsB
List test executions in a Jira project
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| test_plan_key | No | Filter by test plan key (optional) | |
| test_key | No | Filter by test key (optional) | |
| max_results | No | Maximum number of results (default: 50, max: 100) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description must disclose behavioral traits. It states 'list' (implying read-only) but omits details like pagination behavior, potential performance impact, or required permissions. The max_results parameter suggests pagination, but not explicitly stated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single focused sentence with no wasted words. It is front-loaded with the action and resource, though it could benefit from including a usage context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 4 parameters, no output schema, and no annotations, the description is too minimal. It doesn't explain return format, default behavior, or date range implications. Sibling tools like 'export_cucumber_features' have richer descriptions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the input schema already explains each parameter. The description adds no extra meaning beyond 'List test executions in a Jira project'. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'List test executions in a Jira project' is a specific verb+resource+scope statement. It clearly distinguishes from siblings like 'create_test_execution' and 'get_test_execution' by focusing on listing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'get_test_execution' for a single execution or 'create_test_execution' for creation. The description provides no exclusions or context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_plansB
List all test plans in a Jira project
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| max_results | No | Maximum number of results (default: 50, max: 100) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden but only says 'list all test plans'. It does not disclose pagination behavior (though max_results hints at it), authentication needs, permission implications, or ordering. The term 'all' may be misleading without scope clarification.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that directly states the purpose. It is concise, front-loaded, and contains no superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool lacks output schema, yet the description does not clarify what is returned (e.g., list of objects, IDs, names). It does not address edge cases like empty results or invalid project_key. Given the complexity of the sibling set, more context is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the input schema already documents both parameters well. The description adds no additional semantic value beyond what is in the schema, thus baseline score 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'List all test plans in a Jira project', using a specific verb and resource. It distinguishes from siblings like 'get_test_plan' (single retrieval) and 'create_test_plan' (creation).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives like 'get_test_plan' or 'list_tests'. There is no mention of context, prerequisites, or exclusions, which is a significant gap given the number of siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_testsB
List all tests in a Jira project using JQL query
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| labels | No | Comma-separated labels to filter tests (optional) | |
| component | No | Component name to filter tests (optional) | |
| max_results | No | Maximum number of tests to return (default: 50, max: 100) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations provided, so description must disclose behavioral traits. It does not mention pagination (though max_results implies it), JQL syntax expectations, or that the operation is read-only. Minimal disclosure beyond the basic action.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, clear sentence that is concise and front-loaded with the core action. It is well-structured but could earn its place better by including brief usage context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Lacks information about return values (no output schema), pagination behavior, or how JQL queries are applied. Given moderate complexity (4 parameters, JQL), the description is insufficient for full understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with descriptions, so the baseline is 3. The description adds no additional meaning or context beyond the schema; it does not explain parameter interactions or usage nuances.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (List), resource (tests in a Jira project), and method (using JQL query). It distinguishes from sibling tools like get_test (single test) and list_test_executions (executions) by specifying the resource and query mechanism.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives (e.g., get_test, list_test_executions). There are no exclusion criteria or explicit context for proper use, which is needed given the many sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_setsB
List all test sets in a Jira project
| Name | Required | Description | Default |
|---|---|---|---|
| project_key | Yes | Jira project key (e.g., PROJ) | |
| max_results | No | Maximum number of results (default: 50, max: 100) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must fully disclose behavior. It only says 'list' without mentioning pagination (though max_results parameter hints), return format, ordering, or rate limits. This is insufficient for an agent to understand side effects or output structure.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is a single sentence, efficiently conveying the core purpose. It is front-loaded and concise, though could benefit from slightly more detail without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with no output schema and no annotations, the description lacks critical details like return structure, default ordering, and pagination behavior. It is minimal but not fully complete for an agent to invoke confidently.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% as both parameters (project_key, max_results) have descriptions. The tool description adds no further semantic context; baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'List all test sets in a Jira project', specifying verb, resource, and scope. This distinguishes it from siblings like list_test_plans, list_tests, and get_test_set, which target different resources.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for listing test sets but provides no explicit guidance on when to use this tool versus alternatives (e.g., get_test_set for a single test set) or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_testB
Update an existing test
| Name | Required | Description | Default |
|---|---|---|---|
| test_key | Yes | Test issue key (e.g., PROJ-123) | |
| summary | No | New test summary (optional) | |
| description | No | New test description (optional) | |
| labels | No | Comma-separated labels to set (optional) | |
| priority | No | Priority name (e.g., High, Medium, Low) (optional) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It only indicates a mutation operation ('Update') but omits details on permissions, side effects, or constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single clear sentence with no superfluous words. However, it sacrifices completeness for brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a mutation tool with 5 parameters and no output schema or annotations, the description is too minimal. It fails to explain return values, error handling, or behavioral nuances.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, and the description adds no additional meaning beyond the schema's parameter descriptions. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The name 'update_test' and description 'Update an existing test' clearly specify the verb (update) and resource (test). It is distinct from sibling tools like 'create_test' and 'get_test'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives such as 'create_test' or 'update_test_run'. No prerequisites or exclusions are stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_test_runC
Update the result of a test run within a test execution
| Name | Required | Description | Default |
|---|---|---|---|
| test_execution_key | Yes | Test Execution issue key (e.g., PROJ-456) | |
| test_key | Yes | Test issue key (e.g., PROJ-123) | |
| status | Yes | Test run status | |
| comment | No | Comment about the test run (optional) | |
| defects | No | Comma-separated defect keys (optional) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description bears full behavioral burden. It only states 'update' but does not disclose whether updates overwrite or merge, required permissions, error handling, or idempotency. Critical behavioral traits are missing.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, well-structured sentence that front-loads the core action. However, it is too terse and could be more informative without losing conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 5 parameters, no output schema, and no annotations, the description is insufficient. It does not explain return values, side effects, or error scenarios, and fails to provide enough context for an AI agent to use it reliably.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema; it does not explain parameter relationships, provide examples, or clarify optional vs. required usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool as updating a test run's result within a test execution, distinguishing it from siblings like create_test or import_* tools. However, it uses the vague term 'result' rather than explicitly listing the updatable fields (status, comment, defects).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., importing results via import_* tools). There is no mention of prerequisites, use cases, or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
24 tool updates
v1.0.2- First observed
add_tests_to_test_plan - First observed
create_test - First observed
create_test_execution - First observed
create_test_plan - First observed
export_cucumber_features - First observed
get_test - First observed
get_test_execution - First observed
get_test_plan - First observed
get_test_set - First observed
get_test_with_steps - First observed
import_behave_results - First observed
import_cucumber_results - First observed
import_execution_results - First observed
import_feature_file - First observed
import_junit_results - First observed
import_nunit_results - First observed
import_robot_results - First observed
import_testng_results - First observed
list_test_executions - First observed
list_test_plans - First observed
list_test_sets - First observed
list_tests - First observed
update_test - First observed
update_test_run
TDQS
Scored across 24 tools
Each tool targets a distinct Xray entity or action (test, test plan, test execution, test set, import format), with clear naming and descriptions preventing confusion. The only potential overlap (get_test vs get_test_with_steps) is resolved by explicit distinction in descriptions.
All tools follow a consistent verb_noun pattern in snake_case (e.g., create_test, get_test_plan, import_cucumber_results). The pattern is predictable and matches standard CRUD operations plus imports/exports.
24 tools is slightly high but justified for a comprehensive test management server covering multiple entity types and import/export formats. Each tool serves a distinct purpose, and the count aligns with the domain's complexity.
The tool set covers essential CRUD for tests, test plans, test executions, and test sets, plus imports/exports for various formats. However, delete operations for test plans, executions, and test sets are missing, leaving a notable gap in lifecycle coverage.
Maintenance
Related MCP Connectors
An MCP server that provides access to Testiny projects, test cases and test runs
Run, debug, and triage tests from your IDE using natural language, no dashboard switching, no manual data transfers. The TestMu AI (formerly LambdaTest) MCP Server is a single remote server exposing four tool suites: HyperExecute ā analyze your project, generate YAML configs and test runner commands, then monitor jobs and sessions. Automation ā pull a TestID's details plus command, network, and console logs into one chat for instant root-cause analysis. Includes mobile app upload. SmartUI ā explain pixel, layout, DOM, and perceptual changes in a visual regression run, with context-aware React/HTML/CSS fixes. Accessibility ā audit any public URL or a local React app against WCAG and get ready-to-apply remediation steps. Connects over https://mcp.lambdatest.com/mcp using OAuth 2.1 ā no API keys in your config. One-click install in Cursor; works with Claude, GitHub Copilot, Cline, and any MCP client. Tests execute on the TestMu AI cloud: 3,000+ browsers and 10,000+ real devices.
MCP server for AI access to SmartBear tools, including BugSnag, Reflect, Swagger, PactFlow, QTM4J.
MCP Server for JFrog, providing tools for development and artifact management.
Related MCP Servers
- AlicenseBqualityDmaintenanceA Model Context Protocol server that integrates JIRA directly into Cursor IDE, allowing users to view assigned issues, get detailed information on specific tickets, and convert JIRA issues into local tasks without leaving their editor.146 npm6TypeScriptMIT
- AlicenseNot gradedqualityFmaintenanceMCP server for Zephyr Scale test management tool, enabling test case, test plan, test cycle, and test execution management.18 npm2MIT
- AlicenseBqualityAmaintenanceAn MCP server for integrating with Atlassian products including Confluence, Jira, Bitbucket, and Xray for Jira, enabling AI-powered operations like searching, creating, and updating issues, pages, and test results.7227MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server that connects AI assistants to Xray Cloud for Jira test management, enabling natural language interaction with tests, test plans, executions, and CI result imports.MIT