Skip to main content
Glama
gracexiaowork

testrail-mcp-server

TestRail MCP Server

A Model Context Protocol (MCP) server for TestRail integration, providing comprehensive access to TestRail's test management functionality through MCP tools.

Features

This MCP server provides tools for:

  • Projects: List and retrieve TestRail projects

  • Test Cases: Create, read, update, and delete test cases

  • Test Runs: Manage test runs (create, update, close, delete)

  • Test Results: Add and retrieve test results (single and bulk operations)

  • Sections: Organize test cases with sections

Related MCP server: TestRail MCP Server

Prerequisites

  • Node.js 18 or higher

  • A TestRail instance with API access

  • TestRail API key (generate from your TestRail account settings)

Installation

  1. Clone or download this repository

  2. Install dependencies:

npm install
  1. Build the project:

npm run build
  1. Create a .env file in the project root with your TestRail credentials:

cp .env.example .env

Edit .env with your credentials:

TESTRAIL_URL=https://your-instance.testrail.io
TESTRAIL_USERNAME=your-email@example.com
TESTRAIL_API_KEY=your-api-key

Configuration for Claude Code

To use this MCP server with Claude Code, add it to your MCP settings file:

macOS/Linux: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "testrail": {
      "command": "node",
      "args": ["/absolute/path/to/testrail-mcp-server/dist/index.js"],
      "env": {
        "TESTRAIL_URL": "https://your-instance.testrail.io",
        "TESTRAIL_USERNAME": "your-email@example.com",
        "TESTRAIL_API_KEY": "your-api-key"
      }
    }
  }
}

Or if you prefer to use the .env file:

{
  "mcpServers": {
    "testrail": {
      "command": "node",
      "args": ["/absolute/path/to/testrail-mcp-server/dist/index.js"]
    }
  }
}

Available Tools

Projects

  • testrail_get_projects - Get all projects

  • testrail_get_project - Get a specific project by ID

Test Cases

  • testrail_get_cases - Get test cases from a project (optionally filtered by suite)

  • testrail_get_case - Get a specific test case by ID

  • testrail_add_case - Create a new test case in a section

  • testrail_update_case - Update an existing test case

  • testrail_delete_case - Delete a test case

Test Runs

  • testrail_get_runs - Get all test runs in a project

  • testrail_get_run - Get a specific test run by ID

  • testrail_add_run - Create a new test run

  • testrail_update_run - Update an existing test run

  • testrail_close_run - Close a test run

  • testrail_delete_run - Delete a test run

Test Results

  • testrail_get_results - Get test results for a specific test

  • testrail_get_results_for_case - Get test results for a test case in a run

  • testrail_get_results_for_run - Get all test results for a test run

  • testrail_add_result - Add a test result for a specific test

  • testrail_add_result_for_case - Add a test result for a case in a test run

  • testrail_add_results - Add multiple test results (bulk operation)

  • testrail_add_results_for_cases - Add multiple test results for cases (bulk operation)

Sections

  • testrail_get_sections - Get sections from a project (optionally filtered by suite)

  • testrail_get_section - Get a specific section by ID

  • testrail_add_section - Create a new section

  • testrail_update_section - Update an existing section

  • testrail_delete_section - Delete a section

Usage Examples

Once configured, you can use these tools through Claude Code. Here are some example prompts:

Getting Projects

List all TestRail projects

Creating a Test Case

Create a new test case in section 123 with title "Test user login" and priority High

Adding Test Results

Add a passed test result for test case 456 in run 789 with comment "All checks passed"

Creating a Test Run

Create a new test run called "Sprint 24 Testing" for project 1

TestRail Status IDs

When adding test results, use these status IDs:

  • 1 - Passed

  • 2 - Blocked

  • 3 - Untested (retest)

  • 4 - Retest

  • 5 - Failed

TestRail Priority IDs

When creating or updating test cases, use these priority IDs:

  • 1 - Low

  • 2 - Medium

  • 3 - High

  • 4 - Critical

Development

Project Structure

testrail-mcp-server/
├── src/
│   ├── index.ts              # Main MCP server implementation
│   └── testrail-client.ts    # TestRail API client
├── dist/                     # Compiled JavaScript output
├── package.json
├── tsconfig.json
├── .env.example
└── README.md

Building

npm run build

Watching for Changes

npm run watch

Running Locally

npm start

Troubleshooting

Connection Issues

  1. Verify your TestRail credentials are correct

  2. Check that your TestRail instance URL is accessible

  3. Ensure your API key has the necessary permissions

  4. Check the Claude Code logs for detailed error messages

API Errors

  • 401 Unauthorized: Invalid credentials or API key

  • 403 Forbidden: Insufficient permissions for the requested operation

  • 404 Not Found: Resource (project, case, run, etc.) not found

  • 429 Too Many Requests: Rate limit exceeded, wait and retry

API Reference

For more details on TestRail's API, see the official TestRail API documentation.

License

MIT

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

Security

Never commit your .env file or expose your TestRail API credentials. Always use environment variables or secure credential storage.

Available Tools

25 tools
testrail_add_caseC

Create a new test case in a section

ParametersJSON Schema
NameRequiredDescriptionDefault
refsNoOptional references (e.g., ticket IDs)
titleYesThe title of the test case
type_idNoOptional type ID
estimateNoOptional estimate (e.g., "30s", "1m", "2h")
section_idYesThe ID of the section to add the case to
priority_idNoOptional priority ID (1=Low, 2=Medium, 3=High, 4=Critical)
template_idNoOptional template ID
custom_stepsNoOptional test steps
custom_expectedNoOptional expected result
custom_precondsNoOptional preconditions

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the entire burden of behavioral disclosure. Beyond 'create', it says nothing about required permissions, whether titles must be unique within a section, whether the operation is reversible, or what the response contains. This is a mutation tool, so those omissions matter.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single efficient clause with zero waste and the action front-loaded. It is appropriately sized, though its brevity reflects under-specification rather than tight editing of rich content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter mutation tool with no annotations and no output schema, the description is too thin. An agent gets no sense of preconditions, side effects, or how the two required fields interact with the eight optional ones.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all ten parameters are already documented in the schema (priority_id even includes the 1-4 mapping). The description adds no parameter meaning beyond that, which is the expected baseline when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Create a new test case') plus the placement scope ('in a section'), which distinguishes it from testrail_add_section and testrail_update_case. It stops short of explicitly naming those siblings or their boundary conditions, so it is clear but not differentiating.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to create a case versus updating one, no prerequisites (e.g., the section must already exist, or that testrail_get_sections should be called first), and no mention of alternatives. Usage is only implied by the verb.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_resultB

Add a test result for a specific test

ParametersJSON Schema
NameRequiredDescriptionDefault
commentNoOptional comment
defectsNoOptional defects/bugs (comma-separated)
elapsedNoOptional elapsed time (e.g., "30s", "1m", "2h")
test_idYesThe ID of the test
versionNoOptional version tested
status_idYesStatus ID (1=Passed, 2=Blocked, 3=Untested, 4=Retest, 5=Failed)
assignedto_idNoOptional user ID to assign for retest

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It does not disclose that this is a mutation, whether it requires auth, whether it overwrites or appends, or what the response contains. For a write tool with zero annotation coverage this is a significant omission.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with zero redundancy. Appropriately sized for the tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter mutation tool with no annotations and no output schema, the description is thin. The schema covers parameters well, but behavioral context (auth, append semantics, response shape) is entirely absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all 7 parameters including the status_id enum mapping are already documented in the schema. The description adds nothing beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: 'Add a test result for a specific test.' This clearly distinguishes it from read siblings like testrail_get_results, but it does not differentiate from the very close sibling testrail_add_result_for_case, leaving ambiguity about which add-tool applies.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance, no prerequisites, and no mention of alternatives such as testrail_add_result_for_case or testrail_add_results. The agent is left to infer that this is the single-result-by-test-id variant.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_result_for_caseC

Add a test result for a case in a specific test run

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run
case_idYesThe ID of the test case
commentNoOptional comment
defectsNoOptional defects/bugs
elapsedNoOptional elapsed time
versionNoOptional version tested
status_idYesStatus ID (1=Passed, 2=Blocked, 3=Untested, 4=Retest, 5=Failed)
assignedto_idNoOptional user ID to assign

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure for a write operation. It does not say whether the target run must be open, whether prior results are overwritten, what permissions are needed, or that status_id drives pass/fail semantics — all material for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with zero padding, and the scope constraint is front-loaded. Nothing needs trimming.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an eight-parameter mutation tool with no annotations and no output schema, the description is too thin: it omits run-state prerequisites, overwrite behavior, and any distinction from the three sibling result-adding tools. It leans entirely on the schema without compensating for the missing behavioral context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all eight parameters (including the status_id code mapping) are already documented in the schema. The description adds only the run-scoping framing, which the schema's run_id entry already conveys; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Add a test result') plus the scoping context ('for a case in a specific test run'), so the agent knows exactly what is being created. It does not, however, distinguish itself from close siblings like testrail_add_result, testrail_add_results, or testrail_add_results_for_cases, which differ only in whether they target a case or a test.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no exclusions, and no mention of the alternative tools (e.g., testrail_add_result for a test ID, or the bulk testrail_add_results_for_cases). The agent must infer the case-vs-test distinction purely from the tool name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_resultsB

Add multiple test results for a test run (bulk operation)

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run
resultsYesArray of results to add

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, yet it only states that the operation is bulk. For a mutation tool it says nothing about required permissions, whether partial failures are tolerated, whether re-adding results duplicates them, or what state change occurs on the run; only the 'bulk' nature is disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence with no filler, and the key scope cue ('bulk') is placed at the end where it reads clearly. It is efficient, though it is arguably terse for a bulk mutation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The schema is fully documented and there is no output schema to explain, so the structured data covers most of the calling mechanics. However, a bulk-mutation tool with no annotations leaves behavioral and side-effect context undescribed, so the definition is only minimally complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents run_id and the results array with its nested fields. The description adds no syntax, format, or meaning beyond what the schema provides, so the baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb (Add) and resource (test results) plus a scope qualifier (multiple / bulk operation) and ties it to a test run. This distinguishes it reasonably well from the singular testrail_add_result sibling, but it never names the sibling or explicitly contrasts the bulk vs single variants, so an agent must infer the routing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'bulk operation' implicitly signals that this is the multi-result variant rather than the single-result tool, but there is no explicit when-to-use/when-not-to-use statement and no mention of alternatives like testrail_add_result or testrail_add_results_for_cases. Usage is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_results_for_casesC

Add multiple test results for cases in a test run (bulk operation)

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run
resultsYesArray of results to add

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. It reveals only that this is a bulk write; it says nothing about permissions required, whether earlier results are overwritten or appended, partial-failure behavior across the array, or rate limits on large batches.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no filler, and the bulk qualifier is front-loaded. Nothing could be trimmed without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a mutation tool with no annotations, no output schema, and a nested results array that can fail in many ways; the description provides no error, overwrite, or response semantics. For a two-parameter bulk write, it is under-specified relative to what an agent needs to call it safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so run_id and the results array shape are already documented in the schema. The description adds no syntax, ordering, or field-level meaning beyond restating that results go into a run, which is the expected baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('add multiple test results for cases in a test run') and flags the bulk nature, which separates it from the singular testrail_add_result. It does not, however, name or differentiate itself from the closely-related siblings add_results and add_result_for_case, so an agent still has to infer which bulk variant to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this versus testrail_add_result, testrail_add_results, or testrail_add_result_for_case, despite three near-identical siblings in the list. The parenthetical '(bulk operation)' hints at scale but is not an explicit selection rule.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_runC

Create a new test run

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesThe name of the test run
case_idsNoOptional array of case IDs to include (when include_all is false)
suite_idNoOptional suite ID
project_idYesThe ID of the project
descriptionNoOptional description
include_allNoWhether to include all test cases (default: true)
milestone_idNoOptional milestone ID
assignedto_idNoOptional user ID to assign the run to

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, and it says nothing about permissions, side effects, or what the created run contains. Notably it omits the important defaulting behavior of include_all=true, which is a meaningful behavioral trait for this mutation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with zero waste. However, for an 8-parameter mutation the terseness borders on under-specification rather than true conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 8-parameter write tool with no annotations and no output schema, a four-word description leaves major gaps: no note that project_id and name are required, no include_all default behavior, and no indication of what the created run returns.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents all eight parameters including the include_all/case_ids relationship. The description adds no parameter meaning beyond that, which is the expected baseline when the schema does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Create') and resource ('test run'), which clearly separates it from get_run, update_run, close_run and delete_run by action. It is clear but gives no scope detail (e.g. project/run composition) that would fully differentiate it from other creation tools like add_case or add_result.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance, no prerequisites, and no mention of alternatives or follow-up tools (e.g. add_result after creating a run). The agent must infer everything from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_add_sectionC

Create a new section in a project

ParametersJSON Schema
NameRequiredDescriptionDefault
nameYesThe name of the section
suite_idNoOptional suite ID
parent_idNoOptional parent section ID (for nested sections)
project_idYesThe ID of the project
descriptionNoOptional description

TDQS

C2.6/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden of behavioral disclosure, yet it only restates the action. It omits permissions required, side effects, whether nested sections are supported, error behavior, and what is returned. This is inadequate for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single front-loaded sentence with no wasted words. However, it is too sparse for a five-parameter mutation tool, leaving critical behavioral context unaddressed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (five parameters, a mutation operation, no output schema, and no annotations), the description is incomplete. It does not explain how sections relate to suites or projects, the effect of parent_id, or any return or error information.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the input schema already documents all five parameters clearly. The description adds no additional meaning beyond what the schema provides, which is the baseline expectation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Create a new section in a project'), making the core action clear. However, it does not differentiate from sibling tools like update_section or delete_section beyond the verb, and no alternative is named.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no guidance on when to use this tool versus alternatives, nor any prerequisites or context. It simply states the action without explaining the conditions under which an agent should select it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_close_runC

Close a test run

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run to close

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It does not state whether closing is irreversible, what happens to the run's existing results, what permissions are required, or whether the run can be reopened.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single four-word phrase with zero waste and the action front-loaded. It is efficient, though arguably too terse to be genuinely helpful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no annotations and no output schema, the definition is missing key context: irreversibility, effect on results, and expected response. The parameter is documented by the schema, but the behavioral picture is not.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single run_id parameter is fully documented in the schema. The description adds no syntax or format detail beyond that, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ("Close") and resource ("a test run"), so the operation is unambiguous on its own. However, it does nothing to distinguish itself from adjacent siblings like testrail_update_run or testrail_delete_run, which an agent must choose between.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to close a run versus updating or deleting it, nor any prerequisites. The agent must infer usage entirely from the name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_delete_caseC

Delete a test case

ParametersJSON Schema
NameRequiredDescriptionDefault
case_idYesThe ID of the test case to delete

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. 'Delete' implies irreversibility, but the description never confirms whether deletion is permanent, whether it requires elevated permissions or project-admin rights, or what happens to case IDs already referenced by results. With zero annotation coverage, a destructive tool needs more than this.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler, which is structurally sound for such a simple tool. It is arguably too terse given the destructive nature, but there is no wasted wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no annotations and no output schema, the description omits the information an agent most needs: permanence of the delete, permission requirements, and any side effects on related results. Complete for parameter passing, inadequate for safe invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter and schema description coverage is 100%, so the schema already explains case_id. The description adds nothing beyond it, which is the expected baseline when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Delete) and resource (test case), and the singular 'case' plus the required case_id make it distinguishable from sibling bulk/destructive run/section tools. It stops short of explicitly differentiating itself from testrail_delete_run or testrail_delete_section, but the resource noun is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance, no prerequisite context, and no mention of alternatives (e.g. update_case for a softer change). For a destructive operation this is a real gap – an agent gets no signal about when deletion is appropriate versus updating.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_delete_runC

Delete a test run

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run to delete

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. 'Delete' does imply a destructive operation, but the description never states whether it is permanent or recoverable, what happens to associated results/cases, or what permissions are required.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence that is front-loaded and free of filler. It is arguably under-specified rather than bloated, so it earns credit for efficiency without reaching the top of the scale.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive single-parameter tool with no annotations and no output schema, the description should at minimum clarify the permanence of the operation and the effect on related data. It omits all of this, leaving real gaps an agent would need to call it safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with run_id fully documented in the schema itself. The description adds no syntax, format, or range detail beyond that, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Delete a test run'), so the agent knows exactly what the tool does. However, it does nothing to distinguish this from closely related siblings like testrail_close_run or testrail_update_run, which matters because 'close' and 'delete' are easy to conflate on a run resource.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no mention of when deletion is appropriate versus closing a run, and no prerequisites or alternatives named. The agent must infer usage entirely from the tool name and sibling list.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_delete_sectionC

Delete a section

ParametersJSON Schema
NameRequiredDescriptionDefault
section_idYesThe ID of the section to delete

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, and it discloses almost nothing: it does not say whether the deletion is permanent, what happens to child sections or test cases under the section, whether the response is empty, or what permissions are required. 'Delete' implies destruction, but the operational consequences are entirely unspecified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single four-word sentence with the action front-loaded and zero filler. It is efficient, though its brevity is closer to under-specification than polished concision.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive mutation tool with no annotations and no output schema, the description omits the essentials an agent needs: irreversibility, cascade effects on contained cases/sections, and auth or permission requirements. The one parameter is covered by the schema, but the behavioral picture is not complete enough to call this safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% (the single section_id is documented as 'The ID of the section to delete'), and the description adds no further syntax, format, or sourcing details. Baseline 3 is appropriate when the schema already does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb (Delete) and resource (a section), which is enough for an agent to know exactly what the tool does. It does not, however, differentiate itself from siblings like testrail_delete_case or testrail_update_section, or note what a 'section' deletion implies.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no note that this is irreversible, and no pointer to alternatives such as testrail_update_section for non-destructive edits. The agent is given nothing about the conditions under which this tool should be chosen.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_caseC

Get a specific test case by ID

ParametersJSON Schema
NameRequiredDescriptionDefault
case_idYesThe ID of the test case

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden, yet it says nothing about authentication needs, behavior on a missing/invalid case_id, or what the response contains. Only the trivial fact that this is a retrieval is conveyed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with zero filler and no redundancy. It is efficient, though its brevity is partly a symptom of under-specification rather than deliberate tightness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple single-resource getter with a fully documented parameter and no output schema, the description is minimally sufficient. It omits any mention of return contents or failure modes, which would be needed for a higher score.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single case_id parameter is fully documented in the schema, so the description adds no meaning beyond it. Baseline 3 applies when the schema does the heavy lifting.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get a specific test case by ID'), which is unmistakably distinct from the sibling list tool testrail_get_cases. It does not name the sibling explicitly, but the singular 'specific ... by ID' framing makes the distinction clear enough for selection.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this versus testrail_get_cases, nor any note about required permissions or the error behavior when an ID does not exist. Usage is only implied by the word 'specific'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_casesC

Get test cases from a project (optionally filtered by suite)

ParametersJSON Schema
NameRequiredDescriptionDefault
suite_idNoOptional suite ID to filter cases
project_idYesThe ID of the project

TDQS

C2.7/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations provided, so the description carries the full burden of behavioral disclosure. It states nothing about pagination, rate limits, return format, permissions, or the fact that this is a read operation. This is a retrieval tool with zero behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Single sentence, front-loaded with the core operation and the optional filter. No waste, no repetition, appropriately sized for a simple two-parameter tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a retrieval tool with no annotations and no output schema, the description is insufficient. It says nothing about what is returned (list of cases with fields?), whether results are paginated, or how it differs from sibling retrieval tools like testrail_get_case. An agent cannot confidently select this over alternatives.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema fully documents both parameters. The description's mention of the suite filter is redundant with the schema and adds no new meaning. Baseline 3 is appropriate when schema does all the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the verb+resource (Get test cases) and scope (from a project, optionally by suite). Distinguishable from testrail_get_case (singular) by the plural noun, but no explicit differentiation from testrail_get_results_for_case or other retrieval siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this versus testrail_get_case (single case) or how to retrieve a specific case. The optional suite filter is mentioned but no context about when a suite filter is needed or applicable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_projectB

Get a specific project by ID

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesThe ID of the project

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the burden. 'Get' implies a read-only, non-destructive, idempotent lookup, which is adequate behavioral signal for a trivial getter, but it discloses nothing about permissions, error behavior for missing IDs, or return shape.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single terse sentence with the verb and scoping qualifier front-loaded and zero waste. It is appropriately sized, though it is bare enough that 'conciseness' borders on under-specification rather than polished brevity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter lookup tool this is close to sufficient, but with no output schema the description could have said what a project object contains or that a missing ID errors. As written it is minimally adequate rather than complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single required parameter (project_id), so the schema already documents everything needed. The description adds no format or constraint detail beyond 'by ID', which is the correct baseline when the schema does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get a specific project by ID'), and the phrase 'a specific project' implicitly distinguishes it from the sibling testrail_get_projects (list). Clear but it never explicitly names that alternative, so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer 'call this when you have a project ID and need that one project.' There is no explicit when-to-use statement or routing guidance toward testrail_get_projects for listing, and no prerequisites mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_projectsB

Get all projects from TestRail

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, yet it discloses nothing about read-only safety, pagination, ordering, or result size. 'Get' implies a read, but that is inference rather than disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with the resource front-loaded and zero filler. It is efficient, though its brevity shades into under-specification for anything beyond the verb-resource pair.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter read tool with no output schema and no annotations, the definition is minimally viable. It omits anything about return shape, pagination, or the sibling distinction, which an agent would need to call it confidently alongside testrail_get_project.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, which is the baseline-4 case; there is nothing for the description to clarify or compensate for. No parameter meaning is needed or missing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get') and resource ('projects'), and 'all' implicitly contrasts with the singular sibling testrail_get_project. However, it never names or routes against that sibling explicitly, so an agent must infer the plural-vs-singular distinction.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use, when-not-to-use, or prerequisite guidance, and no alternative tool is named. The only signal is the word 'all', which implies a listing/discovery role but is never stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_resultsC

Get test results for a specific test

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoOptional limit on the number of results
test_idYesThe ID of the test

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and discloses almost nothing. It does not say the operation is read-only, what the results contain, whether they are ordered, how pagination interacts with the limit parameter, or whether authentication/project access is required.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single, front-loaded sentence with no filler. It is efficient, though its brevity shades into under-specification rather than crispness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and no annotations mean the description must cover more ground, but it stops at restating the resource. It omits what a result record looks like, pagination behavior for limit, and how this differs from the case/run variants.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both test_id and limit are already documented in the schema. The description adds no format, range, or default detail beyond the schema, so the baseline 3 is correct here.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a clear verb+resource ('Get test results') and scopes it to 'a specific test', which implicitly separates it from the sibling tools testrail_get_results_for_case and testrail_get_results_for_run. However, it never makes that distinction explicit, so the agent must infer what a 'test' is versus a case or run.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use or when-not-to-use guidance. The phrase 'for a specific test' hints at the scoping, but the description never names the alternatives (get_results_for_case, get_results_for_run) or the condition that should route the agent to them, despite three near-identical siblings existing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_results_for_caseB

Get test results for a specific test case in a run

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoOptional limit on the number of results
run_idYesThe ID of the test run
case_idYesThe ID of the test case

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It says nothing about ordering, pagination behavior for the limit parameter, permissions required, or whether the read is live or cached, and gives no hint of the return shape for a results-list operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with the verb and scope stated immediately and zero wasted words. It is tight but so short that it veers toward under-specification rather than true conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read tool with a fully documented schema and no output schema, the description is minimally adequate. However, with no annotations and no output schema, it should at least mention result ordering or pagination semantics to be genuinely complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so all three parameters (run_id, case_id, limit) are already documented in the schema. The description restates only the run/case relationship and adds no format, default, or value-range detail beyond what the schema provides; baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Get) plus a scoped resource (test results for a specific test case in a run). The 'for a specific test case in a run' qualifier distinguishes it from sibling testrail_get_results_for_run and testrail_get_results, though it does not name those siblings explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'for a specific test case in a run' implies when this tool applies (you have both a run_id and a case_id), but it never states when to prefer it over testrail_get_results or testrail_get_results_for_run. Usage must be inferred from the parameter scope.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_results_for_runC

Get all test results for a test run

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNoOptional limit on the number of results
run_idYesThe ID of the test run

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden, yet it only implies a read. It says nothing about pagination, whether results are capped, ordering, or the effect of the limit parameter, so an agent cannot predict the response shape or volume.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no filler and the key scope ('for a test run') front-loaded. It is efficient, though it is terse to the point of under-specifying behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema and no annotations mean the description should explain the returned result shape and pagination, but it does not. For a retrieval tool with a limit parameter, this leaves the agent unable to anticipate what comes back.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both run_id and limit are already documented in the schema. The description adds no syntax, defaults, or format detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Get) and resource (test results) scoped to a test run, which implicitly separates it from testrail_get_results_for_case. However, it largely restates the tool name and never explicitly names a sibling to differentiate against.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this versus testrail_get_results, testrail_get_results_for_case, or testrail_get_run. The agent must infer routing from the name alone, with no stated conditions or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_runC

Get a specific test run by ID

ParametersJSON Schema
NameRequiredDescriptionDefault
run_idYesThe ID of the test run

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full disclosure burden. 'Get' implies a safe read, but nothing is said about permission requirements, whether it errors on a nonexistent or deleted run, or what the response contains — all of which matter given there is no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no padding and the resource front-loaded after the verb. It is efficient, though it is terse to the point of omitting useful context rather than being a model of front-loaded structure.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This is a simple one-parameter getter with a fully documented parameter, so minimal description is defensible. However, with no output schema the agent gets no indication of what a run object looks like or how failures surface, leaving a genuine gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% and the single run_id parameter is fully documented in the schema, so the baseline of 3 applies. The description adds no syntax or format detail beyond restating that the ID identifies a test run.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Get) and resource (test run) with a scoping qualifier ('by ID'), which implicitly distinguishes it from the list-style sibling testrail_get_runs and the mutating siblings (add/update/close/delete_run). It does not name those siblings explicitly, so it is clear but not fully differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives no when-to-use or when-not-to-use guidance and never points to alternatives such as testrail_get_runs for listing or testrail_get_results_for_run for results. The single-retrieval intent is only inferable from the phrase 'a specific test run'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_runsB

Get all test runs in a project

ParametersJSON Schema
NameRequiredDescriptionDefault
project_idYesThe ID of the project

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations are absent, so the description carries the full disclosure burden. It implies a read via 'Get' but says nothing about pagination, result ordering, volume of runs returned, or whether inactive/closed runs are included — meaningful gaps for a list endpoint.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single tight sentence with the resource and scope front-loaded and no filler. Nothing is wasted, though the terseness is partly the source of the transparency gaps.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter read with no output schema and full schema coverage, the description is minimally viable. It omits filtering, pagination, and inclusion semantics for closed runs, which an agent might need given the large sibling set around runs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single required project_id is fully documented in the schema, so the description cannot add syntax value here. Baseline 3 applies since the description merely restates the project scoping already conveyed by the parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get all test runs') scoped to a project, which reads clearly as a list operation. It never names the sibling testrail_get_run (single run) or testrail_get_projects, so an agent must infer the distinction from the plural 'all' rather than being told.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No indication of when to use this versus testrail_get_run, testrail_get_results_for_run, or testrail_get_projects. There are no prerequisites, exclusions, or alternative routing guidance of any kind.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_sectionB

Get a specific section by ID

ParametersJSON Schema
NameRequiredDescriptionDefault
section_idYesThe ID of the section

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden. 'Get' implies a read, but it says nothing about permissions, error behavior for missing/invalid IDs, or what the returned section contains. Minimal disclosure for a tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler. Appropriately sized for a trivial one-parameter lookup.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple, the schema fully documents the sole parameter, and no output schema exists. The description is nearly sufficient, though it omits any note on error handling for nonexistent section IDs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% with a single documented parameter, so the baseline of 3 applies. The description adds nothing beyond confirming that the ID identifies a section.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get') and resource ('section by ID'), which is enough to distinguish it from the list-style sibling testrail_get_sections. It does not explicitly name that sibling as the alternative, so it falls short of the 5-level differentiation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this versus testrail_get_sections or other read tools. The retrieval-by-ID intent is implied by the name but no context or exclusions are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_get_sectionsB

Get sections from a project (optionally filtered by suite)

ParametersJSON Schema
NameRequiredDescriptionDefault
suite_idNoOptional suite ID to filter sections
project_idYesThe ID of the project

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden of behavioral disclosure. It conveys a read operation by implication but does not state permissions required, whether results are paginated, or what the return structure looks like, which is a notable gap for a list tool with zero annotation coverage.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single efficient sentence that front-loads the core action and tucks the optional filter into a parenthetical. Nothing is wasted, though it is minimal rather than rich.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter read tool with no output schema, the description is minimally adequate. It covers the resource and the filter but omits return-format or pagination context, leaving an agent with just enough to call it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters are already documented in the schema, making 3 the baseline. The description's 'optionally filtered by suite' adds mild semantic clarification that suite_id narrows the result set, but nothing beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Get') and resource ('sections') scoped to a project, with an optional suite filter. This distinguishes it from the singular sibling testrail_get_section, though it never names that sibling explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The parenthetical 'optionally filtered by suite' implies when the suite_id matters, giving mild usage context. However, there is no explicit guidance on when to use this list tool versus testrail_get_section or testrail_get_cases, so usage is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_update_caseC

Update an existing test case

ParametersJSON Schema
NameRequiredDescriptionDefault
refsNoOptional references
titleNoOptional new title
case_idYesThe ID of the test case to update
estimateNoOptional estimate
priority_idNoOptional priority ID
custom_stepsNoOptional test steps
custom_expectedNoOptional expected result
custom_precondsNoOptional preconditions

TDQS

C2.8/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden. It never states that this is a partial update (only case_id is required, all other fields optional), whether omitted fields are preserved, what permissions are required, or what the response contains — all material for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The single sentence is front-loaded and free of waste, but at four words it is under-specified rather than genuinely concise for an 8-parameter mutation tool. Brevity here reflects a missing description more than disciplined editing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no annotations, no output schema, and eight parameters, the description omits update semantics (full replace vs partial), permissions, and failure modes. What exists is accurate but far short of what an agent needs to call this safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter is already documented in the schema and the baseline is 3. The description adds no syntax, format, or constraint detail (e.g. accepted priority_id values, estimate format) beyond what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (update) and resource (test case), so the agent knows exactly what domain operation this is. It does not differentiate from siblings such as testrail_update_section or testrail_update_run, but the resource noun is specific enough to disambiguate in practice.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this versus testrail_add_case or testrail_get_case, no mention that the case must already exist, and no prerequisites or permissions noted. The only implicit cue is the word 'existing', which hints at a precondition but states nothing actionable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_update_runC

Update an existing test run

ParametersJSON Schema
NameRequiredDescriptionDefault
nameNoOptional new name
run_idYesThe ID of the test run to update
descriptionNoOptional new description
milestone_idNoOptional milestone ID

TDQS

C2.9/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden and adds nothing beyond the name. It does not say whether this is a partial or full replacement update, whether omitted fields are left untouched, whether permissions are required, or whether the change is reversible.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short, front-loaded sentence with no filler. It is efficient, though arguably over-terse for a mutation tool with four parameters.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A mutation tool with no annotations and no output schema should disclose more: which fields can be modified, partial-vs-full update semantics, and required permissions. The description supplies none of this, leaving the agent under-informed about a state-changing operation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and each parameter already carries its own description ('Optional new name', 'The ID of the test run to update'), so the schema does the heavy lifting. The description adds no syntax, format, or constraint detail beyond that.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Update) and resource (an existing test run), so the operation is unambiguous. However, it does nothing to distinguish itself from siblings like testrail_add_run, testrail_close_run, or testrail_delete_run beyond the verb.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No indication of when to use this tool versus testrail_close_run or testrail_add_run, and no preconditions (e.g., run must exist, must be open) are stated. The agent must infer usage entirely from the name.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

testrail_update_sectionC

Update an existing section

ParametersJSON Schema
NameRequiredDescriptionDefault
nameNoOptional new name
section_idYesThe ID of the section to update
descriptionNoOptional new description

TDQS

C2.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full behavioral burden, and it discloses nothing: not whether omitted fields are left unchanged (partial update semantics), not whether the operation requires specific permissions, and not whether it is reversible. For a mutation tool with zero annotation coverage this is a significant gap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness3/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is a single short sentence with no wasted words and the action is front-loaded. But the brevity reflects under-specification rather than disciplined conciseness, since nothing useful beyond the title is conveyed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a three-parameter mutation tool with no annotations and no output schema, the description is far too thin. It should at minimum clarify partial-update behavior and confirm the target resource, none of which is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% – each of the three parameters is documented in the schema, including the optional new name and description and the required section_id. The description adds no meaning beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Update an existing section'), so the action is unambiguous. However, it offers no differentiation from sibling mutation tools such as testrail_update_case, testrail_update_run, or testrail_update_project, leaving the agent to infer scope purely from the name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to use this versus testrail_add_section or testrail_delete_section, nor any prerequisite or context. The agent must infer usage entirely from the tool name and parameter schema.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 25 tool updatesv1.0.0
    • First observedtestrail_add_case
    • First observedtestrail_add_result
    • First observedtestrail_add_result_for_case
    • First observedtestrail_add_results
    • First observedtestrail_add_results_for_cases
    • First observedtestrail_add_run
    • First observedtestrail_add_section
    • First observedtestrail_close_run
    • First observedtestrail_delete_case
    • First observedtestrail_delete_run
    • First observedtestrail_delete_section
    • First observedtestrail_get_case
    • First observedtestrail_get_cases
    • First observedtestrail_get_project
    • First observedtestrail_get_projects
    • First observedtestrail_get_results
    • First observedtestrail_get_results_for_case
    • First observedtestrail_get_results_for_run
    • First observedtestrail_get_run
    • First observedtestrail_get_runs
    • First observedtestrail_get_section
    • First observedtestrail_get_sections
    • First observedtestrail_update_case
    • First observedtestrail_update_run
    • First observedtestrail_update_section

TDQS

B3.2/5.0

Scored across 25 tools

Disambiguation4/5

Most tools are clearly distinct by resource and action, but the result-related tools (testrail_get_results, testrail_get_results_for_case, testrail_get_results_for_run, testrail_add_result, testrail_add_result_for_case, testrail_add_results, testrail_add_results_for_cases) have subtle differences that may cause confusion. The bulk vs single and for_case vs for_run distinctions are not immediately obvious from names alone.

Naming Consistency5/5

All tools follow a consistent pattern: testrail_<verb>_<resource> (e.g., testrail_get_project, testrail_add_case). The prefix is uniform, verbs are consistent (get, add, update, delete), and resources are clear. Minor variations like 'get_results_for_case' are still predictable.

Tool Count3/5

25 tools is on the higher end and might feel heavy, but it covers a broad domain (projects, cases, runs, results, sections). However, some tools could be consolidated (e.g., result retrieval and addition variants) to reduce redundancy. The count is borderline but justified by the comprehensive CRUD operations.

Completeness4/5

The surface covers core entities (projects, cases, runs, results, sections) with full CRUD for cases, runs, and sections. However, projects only have get operations (no create/update/delete), and there is no tool for retrieving a single test result by ID. These are minor gaps that agents can work around by using list operations.

Maintenance

ActivityInactive
ResponsivenessNo issues

Related MCP Connectors

  • An MCP server that provides access to Testiny projects, test cases and test runs

  • Official MCP server for Qase — manage test cases, runs, suites, defects via AI tools.

  • Run, debug, and triage tests from your IDE using natural language, no dashboard switching, no manual data transfers. The TestMu AI (formerly LambdaTest) MCP Server is a single remote server exposing four tool suites: HyperExecute — analyze your project, generate YAML configs and test runner commands, then monitor jobs and sessions. Automation — pull a TestID's details plus command, network, and console logs into one chat for instant root-cause analysis. Includes mobile app upload. SmartUI — explain pixel, layout, DOM, and perceptual changes in a visual regression run, with context-aware React/HTML/CSS fixes. Accessibility — audit any public URL or a local React app against WCAG and get ready-to-apply remediation steps. Connects over https://mcp.lambdatest.com/mcp using OAuth 2.1 — no API keys in your config. One-click install in Cursor; works with Claude, GitHub Copilot, Cline, and any MCP client. Tests execute on the TestMu AI cloud: 3,000+ browsers and 10,000+ real devices.

  • MCP server for AI access to SmartBear tools, including BugSnag, Reflect, Swagger, PactFlow, QTM4J.

Related MCP Servers

  • A
    license
    B
    quality
    D
    maintenance
    Enables AI assistants to interact with TestRail test management systems through comprehensive API integration. Supports retrieving and updating test cases, projects, suites, runs, and results, plus adding attachments and managing test data through natural language commands.
    18
    12 npm
    MIT
  • A
    license
    A
    quality
    F
    maintenance
    Enables management of TestRail projects, test cases, runs, and results directly through MCP-supported clients. It provides a comprehensive set of tools to interact with the TestRail API for seamless test cycle management within AI environments.
    42
    1,101 npm
    44
    MIT
  • A
    license
    B
    quality
    D
    maintenance
    Enables AI assistants to interact with TestRail test management system, supporting full CRUD operations on projects, suites, sections, test cases, runs, results, plans, and milestones.
    35
    1,504 npm
    1
    MIT