Skip to main content
Glama

Xray MCP

A Model Context Protocol (MCP) server that integrates Cursor IDE with Xray Test Management for Jira, allowing you to manage tests, test executions, and test plans directly from your development environment.

Features

Tests

  • List Tests: Fetch all tests from your Jira project with filtering options

  • Get Test Details: Retrieve detailed test information including test steps via GraphQL ✨

  • Get Test With Steps: NEW! Reliably fetch test details with steps using the GraphQL getTests query šŸš€

  • Create Test: Create new manual or automated tests

  • Update Test: Modify existing tests

Test Executions

  • List Test Executions: View all test executions in a project

  • Get Test Execution Details: See detailed execution results including test runs

  • Create Test Execution: Create new test execution sessions

  • Update Test Run: Record test results (PASS, FAIL, TODO, EXECUTING, ABORTED)

Test Plans

  • List Test Plans: View all test plans in a project

  • Get Test Plan Details: See test plan information with associated tests

  • Create Test Plan: Create new test plans

  • Add Tests to Test Plan: Associate tests with existing plans

Test Sets

  • List Test Sets: View all test sets in a project

  • Get Test Set Details: See test set information with associated tests

Import Operations (CI/CD Integration) šŸš€ NEW!

  • Import Xray JSON Results: Import test execution results in Xray JSON format

  • Import Cucumber Results: Import Cucumber JSON test results

  • Import JUnit Results: Import JUnit XML test results

  • Import TestNG Results: Import TestNG XML test results

  • Import NUnit Results: Import NUnit XML test results

  • Import Robot Framework Results: Import Robot Framework XML test results

  • Import Behave Results: Import Behave JSON test results

  • Import Feature Files: Import Cucumber .feature files (BDD scenarios)

Export Operations šŸ“¤ NEW!

  • Export Cucumber Features: Export Cucumber feature files from Xray

Related MCP server: mcp-zephyr-scale

Prerequisites

  • Node.js 18 or higher

  • npm or yarn

  • Jira Cloud account with Xray Test Management plugin installed

  • Jira API token for authentication

Installation

The package is available on npm and can be used directly with npx:

npx @korfu/xray-mcp

No installation required! Continue to Generate Jira API Token section.

Option 2: Local Development Setup

If you want to contribute or modify the code:

1. Clone the Repository

git clone https://github.com/Korfu/mcp-xray.git
cd mcp-xray

2. Install Dependencies

npm install

3. Build the Project

npm run build

Setup

Generate Jira API Token

  1. Go to Atlassian Account Settings

  2. Click "Create API token"

  3. Give it a descriptive name (e.g., "Xray MCP Integration")

  4. Copy the generated token - you won't be able to see it again!

4. Configure Environment Variables

You'll need the following environment variables:

Required:

  • JIRA_BASE_URL: Your Jira instance URL (e.g., https://your-domain.atlassian.net)

  • JIRA_EMAIL: Your Jira account email

  • JIRA_API_TOKEN: The API token from step 3

Optional (for test steps):

  • XRAY_CLIENT_ID: Your Xray Cloud API client ID

  • XRAY_CLIENT_SECRET: Your Xray Cloud API client secret

4.1. Get Xray Cloud API Credentials (Optional - for Test Steps)

To fetch test steps from Xray, you need Xray Cloud API credentials:

  1. Log in to your Jira instance

  2. Go to Settings (āš™ļø) → Apps → Manage apps

  3. In the left sidebar, find Xray section

  4. Click on API Keys (or Cloud API)

  5. Click Create API Key

  6. Give it a name (e.g., "MCP Integration")

  7. Copy the Client ID and Client Secret

  8. Save these securely - you won't be able to see the secret again!

Note: Without Xray Cloud API credentials, the MCP will still work for listing and managing tests, but test steps won't be fetched.

5. Add to Cursor MCP Configuration

Add the following configuration to your Cursor MCP settings file (usually at ~/.cursor/mcp.json or similar):

For npm installation (recommended):

{
  "mcpServers": {
    "xray": {
      "command": "npx",
      "args": ["-y", "@korfu/xray-mcp"],
      "env": {
        "JIRA_BASE_URL": "https://your-domain.atlassian.net",
        "JIRA_EMAIL": "your-email@example.com",
        "JIRA_API_TOKEN": "your-api-token-here",
        "XRAY_CLIENT_ID": "your-xray-client-id",
        "XRAY_CLIENT_SECRET": "your-xray-client-secret"
      }
    }
  }
}

Important:

  • Replace the values with your actual credentials

  • XRAY_CLIENT_ID and XRAY_CLIENT_SECRET are optional - omit them if you don't need test step details

6. Restart Cursor

After adding the configuration, restart Cursor IDE to load the MCP server.

Usage

Once configured, you can use the following tools in Cursor:

Tests

  • list_tests - List all tests in a project

  • get_test - Get detailed test information (may not always return steps)

  • get_test_with_steps - NEW! Get detailed test information with steps (uses reliable GraphQL query)

  • create_test - Create a new test

  • update_test - Update an existing test

Test Executions

  • list_test_executions - List test executions

  • get_test_execution - Get test execution details

  • create_test_execution - Create a new test execution

  • update_test_run - Update test run status

Test Plans

  • list_test_plans - List all test plans

  • get_test_plan - Get test plan details

  • create_test_plan - Create a new test plan

  • add_tests_to_test_plan - Add tests to a plan

Test Sets

  • list_test_sets - List all test sets

  • get_test_set - Get test set details

Import Operations šŸš€ NEW!

  • import_execution_results - Import Xray JSON format results

  • import_cucumber_results - Import Cucumber JSON results

  • import_junit_results - Import JUnit XML results

  • import_testng_results - Import TestNG XML results

  • import_nunit_results - Import NUnit XML results

  • import_robot_results - Import Robot Framework XML results

  • import_behave_results - Import Behave JSON results

  • import_feature_file - Import Cucumber .feature files

Export Operations šŸ“¤ NEW!

  • export_cucumber_features - Export Cucumber features as .feature files

Examples

Basic Operations

Get test EXM-123 with steps

This uses the new get_test_with_steps tool which reliably fetches test steps.

Output includes test steps:

**Test: EXM-123**
**Summary:** Verify user login functionality

**Test Type:** Manual

**Test Steps:**

**Step 1:**
- **Action:** Navigate to the login page
- **Data:** URL: https://example.com/login
- **Expected Result:** Login page is displayed

**Step 2:**
- **Action:** Enter valid credentials and click Submit
- **Data:** Username: testuser, Password: ********
- **Expected Result:** User is successfully authenticated

**Step 3:**
- **Action:** Verify user dashboard is displayed
- **Data:** N/A
- **Expected Result:** Dashboard shows user profile and navigation menu

CI/CD Integration Examples šŸš€

Import JUnit Results

Import JUnit test results from file test-results/junit.xml

Import Cucumber Results

Import Cucumber results from file cucumber-report.json

Export Feature Files

Export Cucumber features for tests EXM-1,EXM-2,EXM-3

Legacy Examples

List Tests in a Project

List all tests in project PROJ with label "regression"

Create a Test

Create a new manual test in project PROJ with summary "Login validation test" and description "Verify user can login with valid credentials"

Update Test Run Status

Update test run for test PROJ-123 in execution PROJ-456 with status PASS and comment "All assertions passed"

Create Test Execution

Create a test execution in project PROJ with summary "Sprint 5 Regression" and add tests PROJ-123, PROJ-124, PROJ-125

Development

Running in Development Mode

npm run dev

Building

npm run build

Project Structure

XRayMCP/
ā”œā”€ā”€ src/
│   ā”œā”€ā”€ index.ts                  # Main MCP server
│   ā”œā”€ā”€ types.ts                  # TypeScript interfaces
│   ā”œā”€ā”€ services/
│   │   └── XrayCloudService.ts   # Xray Cloud API service (GraphQL + REST)
│   └── tools/
│       ā”œā”€ā”€ tests/                # Test operations
│       │   ā”œā”€ā”€ listTests.ts
│       │   ā”œā”€ā”€ getTest.ts
│       │   ā”œā”€ā”€ createTest.ts
│       │   └── updateTest.ts
│       ā”œā”€ā”€ test-executions/      # Test execution operations
│       │   ā”œā”€ā”€ listTestExecutions.ts
│       │   ā”œā”€ā”€ getTestExecution.ts
│       │   ā”œā”€ā”€ createTestExecution.ts
│       │   └── updateTestRun.ts
│       ā”œā”€ā”€ test-plans/           # Test plan operations
│       │   ā”œā”€ā”€ listTestPlans.ts
│       │   ā”œā”€ā”€ getTestPlan.ts
│       │   ā”œā”€ā”€ createTestPlan.ts
│       │   └── addTestsToTestPlan.ts
│       ā”œā”€ā”€ test-sets/            # Test set operations
│       │   ā”œā”€ā”€ listTestSets.ts
│       │   └── getTestSet.ts
│       ā”œā”€ā”€ import/               # Import operations (NEW!)
│       │   ā”œā”€ā”€ importExecutionResults.ts
│       │   ā”œā”€ā”€ importCucumberResults.ts
│       │   ā”œā”€ā”€ importJUnitResults.ts
│       │   ā”œā”€ā”€ importTestNGResults.ts
│       │   ā”œā”€ā”€ importNUnitResults.ts
│       │   ā”œā”€ā”€ importRobotResults.ts
│       │   ā”œā”€ā”€ importBehaveResults.ts
│       │   └── importFeatureFile.ts
│       └── export/               # Export operations (NEW!)
│           └── exportCucumberFeatures.ts
ā”œā”€ā”€ dist/                         # Compiled JavaScript output
ā”œā”€ā”€ XRAY_CLOUD_API_V2_ENDPOINTS.md     # API documentation
ā”œā”€ā”€ XRAY_CLOUD_V2_IMPLEMENTATION.md    # Implementation details
ā”œā”€ā”€ package.json
ā”œā”€ā”€ tsconfig.json
└── README.md

API Architecture

Xray Cloud API v2 - Dual Approach

This MCP server uses two complementary APIs provided by Xray Cloud:

1. GraphQL API (for Queries)

  • Endpoint: https://xray.cloud.getxray.app/api/v2/graphql

  • Used For: Fetching test details with steps

  • Why: Xray Cloud exposes test steps only via GraphQL, not REST

query {
  getTest(issueId: "EXM-123") {
    issueId
    steps {
      id
      action
      data
      result
    }
  }
}

2. REST API v2 (for Import/Export)

  • Base URL: https://xray.cloud.getxray.app/api/v2

  • Used For: Importing test results, exporting features

  • Endpoints:

    • POST /import/execution/* - Import results (multiple formats)

    • POST /import/feature - Import BDD features

    • GET /export/cucumber - Export BDD features

Authentication

All operations use JWT Bearer tokens obtained via:

POST /api/v2/authenticate
Body: { client_id, client_secret }
Response: "eyJhbGciOiJIUzI1NiI..."

Tokens are:

  • Cached for 50 minutes

  • Automatically refreshed

  • Stored in memory only (not persisted)

Troubleshooting

Authentication Issues

  • Verify your Jira email and API token are correct

  • Ensure the API token has the correct permissions

  • Check that your Jira base URL is correct (should not end with a slash)

"Test issue type not found" Error

  • Ensure Xray is installed in your Jira instance

  • Verify the project has Xray enabled

  • Check that you have permission to create issues in the project

Connection Issues

  • Verify your internet connection

  • Check if your Jira instance is accessible

  • Ensure the MCP server is running (check Cursor's MCP logs)

Test Not Found

  • Verify the test key is correct (e.g., PROJ-123)

  • Ensure you have access to the test

  • Check that the test exists in Jira

MCP Server Not Loading

  • Verify the path to dist/index.js is correct in your Cursor configuration

  • Ensure Node.js is installed and accessible from the command line

  • Check Cursor's MCP server logs for error messages

  • Make sure you've run npm run build to compile the TypeScript code

Test Steps Not Appearing

  • Ensure you've configured XRAY_CLIENT_ID and XRAY_CLIENT_SECRET in your MCP configuration

  • Verify your Xray Cloud API credentials are correct

  • Check that your Xray license includes API access

  • The test must be a Manual test type (Cucumber and Generic tests have different step formats)

  • Check Cursor's MCP logs for authentication errors

API Endpoints Used

This MCP uses the following Jira and Xray API endpoints:

  • Jira REST API v3: /rest/api/3/search/jql, /rest/api/3/issue/* for general issue operations

  • Xray Cloud API v2: https://xray.cloud.getxray.app/api/v2/* for test steps and detailed test information (optional)

Security Notes

  • Keep your Jira API token secure and never commit it to version control

  • Keep your Xray Cloud API credentials (Client ID and Secret) secure

  • Use environment variables or secure configuration management for credentials

  • The API token should only have the minimum required permissions

  • Consider rotating your API tokens regularly

  • Xray API credentials are separate from Jira credentials and provide access to test data

Contributing

  1. Fork the repository

  2. Create a feature branch

  3. Make your changes

  4. Add tests if applicable

  5. Submit a pull request

License

MIT License - see LICENSE file for details

Author

Korfu

Acknowledgments

  • Built following the Model Context Protocol specification

  • Inspired by the BitbucketMCP implementation pattern

  • Uses Xray Test Management for Jira by Xblend

Available Tools

24 tools
add_tests_to_test_planB

Add tests to an existing test plan

ParametersJSON Schema
NameRequiredDescriptionDefault
test_plan_keyYesTest Plan issue key (e.g., PROJ-789)
test_keysYesComma-separated test keys to add

TDQS

B3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, and the description only states 'Add tests' without disclosing behavioral traits like whether it appends or overwrites, error handling, or side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise (one sentence) but lacks structure; it could include more detail without becoming verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and the description does not explain return values, prerequisites, or behavior after execution, leaving the agent with incomplete context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with both parameters described clearly (e.g., example format). The description adds no extra meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (add tests) and the target (existing test plan), distinguishing it from sibling tools like create_test_plan which creates a new plan.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives, no prerequisites (e.g., test plan must exist) or conditions for adding tests.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_testC

Create a new test in Jira with Xray

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
summaryYesTest summary/title
descriptionNoTest description (optional)
test_typeNoTest type: Manual, Cucumber, or Generic (default: Manual)Manual
labelsNoComma-separated labels (optional)
priorityNoPriority name (e.g., High, Medium, Low)

TDQS

C2.8/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations supplied, the description must fully convey behavioral traits. It only states the action without mentioning required permissions, side effects (e.g., data creation), success/failure responses, or any other behaviors. This is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The single-sentence description is concise but too brief; it lacks necessary usage and behavioral details. While not verbose, it could be expanded to be more helpful without being wasteful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 6 parameters, no output schema, and no annotations, the description should provide more context about return values, side effects, and appropriate use. It only offers a minimal statement, leaving significant gaps for an AI agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so each parameter has a schema description. The description adds no additional meaning beyond the schema, meeting the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Create a new test in Jira with Xray' clearly identifies the action (create) and the resource (test), and distinguishes from sibling tools like create_test_plan or create_test_execution by specifying 'test' and referencing Xray.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives such as creating a test execution or importing results. It lacks context about prerequisites, scenarios, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_test_executionC

Create a new test execution in Jira with Xray

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
summaryYesTest execution summary/title
descriptionNoTest execution description (optional)
test_plan_keyNoTest plan key to associate with (optional)
test_environmentsNoComma-separated test environments (optional)
testsNoComma-separated test keys to add to execution (optional)

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description only indicates creation, which implies mutation, but provides no details on side effects, permissions required, or return behavior. With no annotations, the description fails to disclose any behavioral traits beyond the basic action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise with a single sentence, but lacks structure such as bullet points or sections. While short, it may be under-specified for a tool with 6 parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 6 parameters and no output schema or annotations, the description fails to provide context on return values, behavior, or usage workflow. It only gives the basic purpose, leaving significant gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All 6 parameters have descriptions in the input schema (100% coverage), so the schema already provides meaning. The description adds no additional context to the parameters, matching the baseline of 3.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (Create) and the resource (test execution in Jira with Xray). It distinguishes itself from sibling tools like create_test and create_test_plan by specifying 'test execution'. However, it could be more specific about the scope and purpose of a test execution.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives such as import_execution_results or add_tests_to_test_plan. There is no mention of prerequisites, context, or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_test_planB

Create a new test plan in Jira with Xray

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
summaryYesTest plan summary/title
descriptionNoTest plan description (optional)
testsNoComma-separated test keys to add to plan (optional)

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description must fully convey behavioral traits. While it implies mutation, it omits details like required permissions, idempotency, side effects, or confirmation of success. The basic verb 'create' is insufficient.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, concise sentence with no redundancy or fluff. It avoids tautology and earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a creation tool with 4 parameters and no output schema, the description is minimal. It fails to explain return values, error handling, or required context, leaving the agent underinformed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, providing adequate descriptions for all parameters. The description adds no additional meaning beyond what the schema already states, so the baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'Create a new test plan in Jira with Xray' clearly states the action (create), resource (test plan), and context (Jira/Xray). It effectively distinguishes from sibling tools like get_test_plan (read) and add_tests_to_test_plan (modify).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives like create_test or create_test_execution. No prerequisites, exclusions, or context for when to choose this tool are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

export_cucumber_featuresA

Export Cucumber feature files from Xray. Can export all features or specific test keys.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_keysNoOptional: Comma-separated list of test keys to export (e.g., "EXM-1,EXM-2"). If not provided, exports all features from the project.

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must cover behavioral traits. It merely states the export action without disclosing whether it is read-only, the output format, or any side effects. This leaves the agent uncertain about the operation's safety.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description consists of two short sentences with no redundant information. It is front-loaded with the core purpose and efficiently covers the main usage options.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple parameters and no output schema, the description should explain what the tool returns (e.g., feature file content or URIs) and whether it is synchronous. It omits these essential details, leaving the agent with an incomplete picture.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The parameter test_keys is fully described in the schema with a clear description. The tool description adds no additional meaning beyond what the schema already provides, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Export' and the resource 'Cucumber feature files from Xray', distinguishing between exporting all features or specific test keys. This differentiates it from sibling tools like import_feature_file.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains that the tool can export all features or specific test keys, providing clear usage context. However, it lacks explicit when-not-to-use guidance or alternatives, such as noting that import_feature_file is the appropriate tool for importing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_testA

Get detailed information about a specific test by its key

ParametersJSON Schema
NameRequiredDescriptionDefault
test_keyYesTest issue key (e.g., PROJ-123)

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavior. It states 'detailed information' but does not specify what fields or structure are returned, leaving ambiguity for the agent.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence of 11 words with no redundancy. All information is front-loaded and concise.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simplicity (1 required param, no output schema), the description is minimally sufficient but does not describe the return format, leaving the agent without full expectations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the parameter description is adequate. The description repeats the schema's point that the test is identified by key, adding no new meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Get', the resource 'test', and the scope 'by its key'. This distinguishes it from siblings like list_tests or create_test.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when a test key is available, but does not provide explicit when-to-use or when-not-to-use guidance compared to alternative tools like get_test_with_steps.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_executionB

Get detailed information about a test execution including test run results

ParametersJSON Schema
NameRequiredDescriptionDefault
test_execution_keyYesTest Execution issue key (e.g., PROJ-456)

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It mentions returning 'detailed information' and 'test run results', but omits aspects like read-only nature, potential errors, or any side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence, concise and directly to the point. However, it could be more structurally informative with bullet points or explicit callouts.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given a simple 1-parameter tool with no output schema, the description is adequate but not thorough. It mentions 'test run results' but does not elaborate on the output structure, which would help agents.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema covers 100% of parameters with descriptions (e.g., 'Test Execution issue key (e.g., PROJ-456)'). The description does not add additional meaning beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Get') and the resource ('test execution'), and specifies the output includes 'test run results'. It distinguishes from sibling tools like get_test or get_test_plan by focusing on executions.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as list_test_executions or get_test. The description lacks context about prerequisites or preferred scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_planA

Get detailed information about a test plan including associated tests

ParametersJSON Schema
NameRequiredDescriptionDefault
test_plan_keyYesTest Plan issue key (e.g., PROJ-789)

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It mentions retrieving detailed information but does not clarify side effects, required permissions, data limits, or the exact structure of the output. This lack of transparency is a gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence with no redundant information. It is appropriately front-loaded and concise, earning its place without wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description gives the core purpose and a hint about output (including associated tests). However, with no annotations, more context about the return value structure would be beneficial. It is minimally adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides a clear description for test_plan_key (e.g., PROJ-789) with 100% coverage. The description adds that the output includes associated tests, but this is more about output than parameter semantics. The schema already fully documents the parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly uses the verb 'Get' with the resource 'test plan' and specifies the scope 'detailed information including associated tests'. This distinguishes it from sibling tools like list_test_plans, which suggest a listing function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for obtaining detailed information about a specific test plan, but does not explicitly state when to use this tool vs alternatives like list_test_plans. The guidance is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_setA

Get detailed information about a test set including associated tests

ParametersJSON Schema
NameRequiredDescriptionDefault
test_set_keyYesTest Set issue key (e.g., PROJ-999)

TDQS

A3.6/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided; description only implies read-only behavior via 'Get detailed information'. Lacks disclosure on side effects, authentication, or other behavioral constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence clearly conveys purpose without waste. Front-loaded with essential information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description adequately hints at return content ('detailed information', 'associated tests'), though more specifics could help.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with one parameter; description adds no extra meaning beyond the schema's own description. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves detailed information about a test set including associated tests, distinguishing it from sibling tools like get_test, get_test_execution, etc.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implied usage from the description: when a user needs test set details. No explicit when-to-use or when-not-to-use guidance, nor alternatives mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_with_stepsA

Get detailed test information with test steps using the reliable GraphQL getTests query. This tool always fetches test steps if they exist.

ParametersJSON Schema
NameRequiredDescriptionDefault
test_keyYesTest issue key (e.g., EXM-123)

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses a key behavioral trait: 'always fetches test steps if they exist'. However, it does not cover error handling, performance, or what happens when the test key is invalid. Since no annotations are provided, the description partially fulfills transparency but lacks completeness.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise with two sentences that front-load the purpose and follow with a behavioral note. Every sentence adds value, and there is no redundant or irrelevant content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one required parameter, no output schema, no annotations), the description provides minimal context. It does not explain the return format or behavior when steps are absent. While adequate for basic use, it could be more complete for an agent to fully understand outcomes.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter 'test_key', which is well-documented in the schema with an example. The description adds no additional semantic information about the parameter, so it meets the baseline for high schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Get detailed test information with test steps'. It explicitly mentions the resource (test info) and action (get with steps), and distinguishes from sibling tools like 'get_test' by specifying the inclusion of test steps.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when test steps are needed by stating 'always fetches test steps if they exist', but it does not explicitly advise when to use this tool versus alternatives like 'get_test'. No exclusions or context are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_behave_resultsA

Import Behave JSON test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
behave_jsonYesBehave JSON results as a string or JSON array

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry full burden. It mentions side effects (creates, updates) but lacks details on permissions, reversibility, or error conditions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short, front-loaded sentences with no unnecessary words. Efficient and clear.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Adequate for a simple single-parameter import tool, but lacks details on expected JSON structure, error handling, or prerequisites. No output schema, so return value expectations are omitted.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter is well-described in the schema. The description adds minimal value beyond restating the resource type ('Behave JSON test results').

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the verb 'import', the resource 'Behave JSON test results', and the outcomes ('creates test execution and updates test statuses'). It distinguishes from sibling tools like import_cucumber_results by specifying the framework.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for Behave framework results but provides no explicit when-to-use or when-not-to-use guidance. No alternatives are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_cucumber_resultsA

Import Cucumber JSON test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
cucumber_jsonYesCucumber JSON results as a string or JSON array

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses creation of test execution and status updates, which is beyond the schema. No annotations are provided, so the description carries full burden. However, it omits details like what prevents duplicate executions or required test case existence.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with action and outcome. Every word earns its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-param import tool, the description is reasonably complete. It covers purpose and side effects, but lacks error conditions or behavior on repeated calls. No output schema, so return value is implicit.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear description for the single parameter. The tool description adds no additional parameter meaning beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the verb 'import', the resource 'Cucumber JSON test results', and the target 'Xray'. Also describes side effects (creates test execution, updates statuses). Distinguishes from siblings by specifying Cucumber format.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this vs. alternatives. The name implies it's for Cucumber JSON, but there is no mention of prerequisites, limitations, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_execution_resultsA

Import test execution results in Xray JSON format. Creates test executions and test runs with results.

ParametersJSON Schema
NameRequiredDescriptionDefault
results_jsonYesXray JSON format execution results as a JSON string. Example: {"testExecutionKey": "EXM-789", "tests": [{"testKey": "EXM-1", "status": "PASS"}]}

TDQS

A3.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must disclose behavioral traits. It only states creation of test executions and runs, but omits details like whether the import is additive or overwrites existing data, any required permissions, or validation behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two concise sentences that front-load the purpose. No redundant information; every word contributes.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity (one parameter, no output schema), the description covers the essential purpose and format. However, it lacks details like the effect on existing data or error handling, but overall is adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% for the single parameter. The description adds an example and specifies the Xray JSON format, enhancing understanding beyond the schema's description. While the schema already describes the parameter, the description provides practical context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: importing test execution results in Xray JSON format and creating test executions and runs. It uses a specific verb 'Import' and resource 'execution results', distinguishing it from sibling tools like import_junit_results or import_cucumber_results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for Xray JSON format but does not explicitly state when to use vs alternatives, nor does it mention any exclusions or prerequisites. Usage is implied but not robust.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_feature_fileA

Import Cucumber .feature file to Xray. Creates or updates BDD tests from Gherkin scenarios.

ParametersJSON Schema
NameRequiredDescriptionDefault
feature_contentYesContent of the .feature file (Gherkin syntax)

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions mutation ('Creates or updates') but does not explain idempotency, whether existing tests are overwritten, what happens to associated artifacts, or authorization requirements. The agent is left to infer behavior from the minimal description.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is highly concise: two sentences front-loaded with the main action and then the effect. No unnecessary words or repetition. Every sentence serves a clear purpose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with 1 parameter and no output schema, the description provides the core purpose but lacks details on expected return values (e.g., test IDs, success/failure indicators) and edge cases (e.g., handling multiple scenarios). Given the tool's simplicity, a baseline of 3 is reasonable, but additional behavioral context would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (single parameter 'feature_content' described as 'Content of the .feature file (Gherkin syntax)'). The description adds context by stating the effect on BDD tests, but does not provide additional details on the parameter format or constraints beyond what the schema already provides. Baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Import', the resource 'Cucumber .feature file', and the target 'Xray'. It explains the effect: 'Creates or updates BDD tests from Gherkin scenarios.' This distinguishes it from sibling tools like export_cucumber_features (opposite direction) and import_cucumber_results (imports results, not features).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for importing feature files to create/update BDD tests, but it provides no explicit guidance on when to use this tool versus alternatives. For example, it does not differentiate from import_cucumber_results (which imports execution results) or export_cucumber_features. An explicit 'use when you have .feature content' and 'not for importing test results' would improve clarity.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_junit_resultsA

Import JUnit XML test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
junit_xmlYesJUnit XML results as a string

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses side effects (creates test execution, updates statuses) but no details on error handling, authentication needs, or whether existing results are overwritten. No annotations provided, so description carries full burden.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences with no superfluous information. Each sentence adds value: first states primary action, second clarifies side effects.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given low complexity (1 param, no output schema, no annotations), the description covers purpose and behavior adequately. Could mention return value (e.g., execution ID) but not essential for selection.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with description 'JUnit XML results as a string'. Description repeats the format but adds no extra semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the verb 'Import', the resource 'JUnit XML test results', and the target 'Xray'. Also specifies automatic creation of test execution and status updates, distinguishing from sibling import tools (e.g., import_cucumber_results).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implied usage through format-specific naming but lacks explicit guidance on when to use this tool vs alternatives (e.g., for other formats) or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_nunit_resultsA

Import NUnit XML test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
nunit_xmlYesNUnit XML results as a string

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses key side effects: automatically creates test execution and updates test statuses. However, with no annotations, it fails to cover permissions, reversibility, or error behavior. Adds some context beyond the bare schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, well-structured sentence that immediately conveys the tool's purpose and side effects. No extraneous words or repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with one parameter and no output schema, the description covers the essential function and side effects. Lacks details on required permissions or error handling, but meets the minimum for effective use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has 100% description coverage for the single parameter 'nunit_xml'. The description's mention of 'NUnit XML test results' adds minimal meaning beyond the schema's 'NUnit XML results as a string'. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the verb 'Import', the resource 'NUnit XML test results', and the target 'Xray'. It distinguishes from similar sibling tools like import_junit_results by specifying the NUnit format.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives (e.g., import_junit_results, import_cucumber_results). It lacks prerequisites, context for selection, or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_robot_resultsA

Import Robot Framework XML test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
robot_xmlYesRobot Framework XML results (output.xml) as a string

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses side effects (creates execution, updates statuses) but no annotations exist. Lacks details on permissions, data overwrite, or other behavioral traits, leaving moderate gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences front-loading the purpose and behavior, no wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-param tool with no output schema, the description covers the main function and side effects adequately, though could mention prerequisites like project context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with a clear parameter description. The description does not add extra meaning beyond the schema, meeting the baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the verb 'Import', the resource 'Robot Framework XML test results', and the effects 'creates test execution and updates test statuses', distinguishing it from sibling import tools like import_cucumber_results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool vs alternatives. While the name implies Robot Framework results, the description does not provide usage context or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_testng_resultsB

Import TestNG XML test results to Xray. Automatically creates test execution and updates test statuses.

ParametersJSON Schema
NameRequiredDescriptionDefault
testng_xmlYesTestNG XML results as a string

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavioral traits. It mentions automatic creation of test execution and status updates, but lacks details on side effects (e.g., what if XML is invalid, idempotency, required permissions, or whether existing data is overwritten). This is a significant gap for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences and no wasted words. It could be slightly improved by structuring with bullet points, but it remains concise and readable.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no output schema and only one parameter, the description should cover expected outcomes on success/failure and any constraints (e.g., file size). It only states basic behavior, leaving important context unclear.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single parameter is described as 'TestNG XML results as a string'. The description adds no extra meaning beyond the schema, so a baseline of 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'Import' and the resource 'TestNG XML test results to Xray', and specifies two key outcomes: creating a test execution and updating test statuses. This effectively distinguishes it from sibling tools like import_cucumber_results or import_junit_results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for TestNG result import but provides no explicit when-to-use guidance or alternatives. Among many import siblings, it would benefit from stating that this is specifically for TestNG XML rather than other formats.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_executionsB

List test executions in a Jira project

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
test_plan_keyNoFilter by test plan key (optional)
test_keyNoFilter by test key (optional)
max_resultsNoMaximum number of results (default: 50, max: 100)

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description must disclose behavioral traits. It states 'list' (implying read-only) but omits details like pagination behavior, potential performance impact, or required permissions. The max_results parameter suggests pagination, but not explicitly stated.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single focused sentence with no wasted words. It is front-loaded with the action and resource, though it could benefit from including a usage context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 4 parameters, no output schema, and no annotations, the description is too minimal. It doesn't explain return format, default behavior, or date range implications. Sibling tools like 'export_cucumber_features' have richer descriptions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema already explains each parameter. The description adds no extra meaning beyond 'List test executions in a Jira project'. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description 'List test executions in a Jira project' is a specific verb+resource+scope statement. It clearly distinguishes from siblings like 'create_test_execution' and 'get_test_execution' by focusing on listing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as 'get_test_execution' for a single execution or 'create_test_execution' for creation. The description provides no exclusions or context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_plansB

List all test plans in a Jira project

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
max_resultsNoMaximum number of results (default: 50, max: 100)

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries full burden but only says 'list all test plans'. It does not disclose pagination behavior (though max_results hints at it), authentication needs, permission implications, or ordering. The term 'all' may be misleading without scope clarification.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single sentence that directly states the purpose. It is concise, front-loaded, and contains no superfluous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool lacks output schema, yet the description does not clarify what is returned (e.g., list of objects, IDs, names). It does not address edge cases like empty results or invalid project_key. Given the complexity of the sibling set, more context is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the input schema already documents both parameters well. The description adds no additional semantic value beyond what is in the schema, thus baseline score 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states 'List all test plans in a Jira project', using a specific verb and resource. It distinguishes from siblings like 'get_test_plan' (single retrieval) and 'create_test_plan' (creation).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives like 'get_test_plan' or 'list_tests'. There is no mention of context, prerequisites, or exclusions, which is a significant gap given the number of siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_testsB

List all tests in a Jira project using JQL query

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
labelsNoComma-separated labels to filter tests (optional)
componentNoComponent name to filter tests (optional)
max_resultsNoMaximum number of tests to return (default: 50, max: 100)

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so description must disclose behavioral traits. It does not mention pagination (though max_results implies it), JQL syntax expectations, or that the operation is read-only. Minimal disclosure beyond the basic action.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, clear sentence that is concise and front-loaded with the core action. It is well-structured but could earn its place better by including brief usage context.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Lacks information about return values (no output schema), pagination behavior, or how JQL queries are applied. Given moderate complexity (4 parameters, JQL), the description is insufficient for full understanding.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions, so the baseline is 3. The description adds no additional meaning or context beyond the schema; it does not explain parameter interactions or usage nuances.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (List), resource (tests in a Jira project), and method (using JQL query). It distinguishes from sibling tools like get_test (single test) and list_test_executions (executions) by specifying the resource and query mechanism.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives (e.g., get_test, list_test_executions). There are no exclusion criteria or explicit context for proper use, which is needed given the many sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_setsB

List all test sets in a Jira project

ParametersJSON Schema
NameRequiredDescriptionDefault
project_keyYesJira project key (e.g., PROJ)
max_resultsNoMaximum number of results (default: 50, max: 100)

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It only says 'list' without mentioning pagination (though max_results parameter hints), return format, ordering, or rate limits. This is insufficient for an agent to understand side effects or output structure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is a single sentence, efficiently conveying the core purpose. It is front-loaded and concise, though could benefit from slightly more detail without becoming verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with no output schema and no annotations, the description lacks critical details like return structure, default ordering, and pagination behavior. It is minimal but not fully complete for an agent to invoke confidently.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% as both parameters (project_key, max_results) have descriptions. The tool description adds no further semantic context; baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states 'List all test sets in a Jira project', specifying verb, resource, and scope. This distinguishes it from siblings like list_test_plans, list_tests, and get_test_set, which target different resources.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for listing test sets but provides no explicit guidance on when to use this tool versus alternatives (e.g., get_test_set for a single test set) or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_testB

Update an existing test

ParametersJSON Schema
NameRequiredDescriptionDefault
test_keyYesTest issue key (e.g., PROJ-123)
summaryNoNew test summary (optional)
descriptionNoNew test description (optional)
labelsNoComma-separated labels to set (optional)
priorityNoPriority name (e.g., High, Medium, Low) (optional)

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It only indicates a mutation operation ('Update') but omits details on permissions, side effects, or constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single clear sentence with no superfluous words. However, it sacrifices completeness for brevity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with 5 parameters and no output schema or annotations, the description is too minimal. It fails to explain return values, error handling, or behavioral nuances.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, and the description adds no additional meaning beyond the schema's parameter descriptions. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The name 'update_test' and description 'Update an existing test' clearly specify the verb (update) and resource (test). It is distinct from sibling tools like 'create_test' and 'get_test'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus alternatives such as 'create_test' or 'update_test_run'. No prerequisites or exclusions are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_test_runC

Update the result of a test run within a test execution

ParametersJSON Schema
NameRequiredDescriptionDefault
test_execution_keyYesTest Execution issue key (e.g., PROJ-456)
test_keyYesTest issue key (e.g., PROJ-123)
statusYesTest run status
commentNoComment about the test run (optional)
defectsNoComma-separated defect keys (optional)

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description bears full behavioral burden. It only states 'update' but does not disclose whether updates overwrite or merge, required permissions, error handling, or idempotency. Critical behavioral traits are missing.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, well-structured sentence that front-loads the core action. However, it is too terse and could be more informative without losing conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters, no output schema, and no annotations, the description is insufficient. It does not explain return values, side effects, or error scenarios, and fails to provide enough context for an AI agent to use it reliably.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3. The description adds no additional meaning beyond the schema; it does not explain parameter relationships, provide examples, or clarify optional vs. required usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool as updating a test run's result within a test execution, distinguishing it from siblings like create_test or import_* tools. However, it uses the vague term 'result' rather than explicitly listing the updatable fields (status, comment, defects).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives (e.g., importing results via import_* tools). There is no mention of prerequisites, use cases, or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 24 tool updatesv1.0.2
    • First observedadd_tests_to_test_plan
    • First observedcreate_test
    • First observedcreate_test_execution
    • First observedcreate_test_plan
    • First observedexport_cucumber_features
    • First observedget_test
    • First observedget_test_execution
    • First observedget_test_plan
    • First observedget_test_set
    • First observedget_test_with_steps
    • First observedimport_behave_results
    • First observedimport_cucumber_results
    • First observedimport_execution_results
    • First observedimport_feature_file
    • First observedimport_junit_results
    • First observedimport_nunit_results
    • First observedimport_robot_results
    • First observedimport_testng_results
    • First observedlist_test_executions
    • First observedlist_test_plans
    • First observedlist_test_sets
    • First observedlist_tests
    • First observedupdate_test
    • First observedupdate_test_run

TDQS

A3.5/5.0

Scored across 24 tools

Disambiguation5/5

Each tool targets a distinct Xray entity or action (test, test plan, test execution, test set, import format), with clear naming and descriptions preventing confusion. The only potential overlap (get_test vs get_test_with_steps) is resolved by explicit distinction in descriptions.

Naming Consistency5/5

All tools follow a consistent verb_noun pattern in snake_case (e.g., create_test, get_test_plan, import_cucumber_results). The pattern is predictable and matches standard CRUD operations plus imports/exports.

Tool Count4/5

24 tools is slightly high but justified for a comprehensive test management server covering multiple entity types and import/export formats. Each tool serves a distinct purpose, and the count aligns with the domain's complexity.

Completeness3/5

The tool set covers essential CRUD for tests, test plans, test executions, and test sets, plus imports/exports for various formats. However, delete operations for test plans, executions, and test sets are missing, leaving a notable gap in lifecycle coverage.

Maintenance

ActivityInactive
ResponsivenessNo issues

Related MCP Connectors

  • An MCP server that provides access to Testiny projects, test cases and test runs

  • Run, debug, and triage tests from your IDE using natural language, no dashboard switching, no manual data transfers. The TestMu AI (formerly LambdaTest) MCP Server is a single remote server exposing four tool suites: HyperExecute — analyze your project, generate YAML configs and test runner commands, then monitor jobs and sessions. Automation — pull a TestID's details plus command, network, and console logs into one chat for instant root-cause analysis. Includes mobile app upload. SmartUI — explain pixel, layout, DOM, and perceptual changes in a visual regression run, with context-aware React/HTML/CSS fixes. Accessibility — audit any public URL or a local React app against WCAG and get ready-to-apply remediation steps. Connects over https://mcp.lambdatest.com/mcp using OAuth 2.1 — no API keys in your config. One-click install in Cursor; works with Claude, GitHub Copilot, Cline, and any MCP client. Tests execute on the TestMu AI cloud: 3,000+ browsers and 10,000+ real devices.

  • MCP server for AI access to SmartBear tools, including BugSnag, Reflect, Swagger, PactFlow, QTM4J.

  • MCP Server for JFrog, providing tools for development and artifact management.

Related MCP Servers

  • A
    license
    B
    quality
    D
    maintenance
    A Model Context Protocol server that integrates JIRA directly into Cursor IDE, allowing users to view assigned issues, get detailed information on specific tickets, and convert JIRA issues into local tasks without leaving their editor.
    14
    6 npm
    6
    TypeScript
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    An MCP server that connects AI assistants to Xray Cloud for Jira test management, enabling natural language interaction with tests, test plans, executions, and CI result imports.
    MIT