Supports running configured Appium mobile test suites and parsing their JUnit XML reports to summarize pass/fail results and surface failure details.
Files Jira Cloud bugs with test evidence through the Jira Cloud REST API, falling back to draft files when credentials are not configured.
Supports running configured pytest suites and parsing their JUnit XML output to summarize test results and failures.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Test Automation MCP Server Run the checkout suite and file bugs for any real defects"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A Model Context Protocol server that gives AI assistants (Claude, GitHub Copilot, Cursor) safe, structured access to a test automation stack. From a chat, an assistant can run a suite, read why tests failed, pull test data, and file a Jira bug with the evidence.
"Run the checkout suite and file bugs for anything that's a real defect."
assistant ──▶ run_tests(sample-e2e) 6 tests: 3 passed, 2 failed, 1 skipped
──▶ get_failure_details(promo) Expected "$39.20" Received "$49.00" · API: PROMO_EXPIRED
──▶ run_tests(filter: "order") still times out → flaky / environment, no bug
──▶ query_test_data(promoCodes) SPRING10 is marked expired in test data → test data issue
──▶ create_jira_bug(...) only for confirmed product defectsTool | What it does |
| Lists configured suites, their commands and report paths |
| Runs a suite (optionally filtered to matching tests) and returns pass/fail counts plus each failure |
| Summarizes the latest JUnit report without re-running |
| Error message, stack trace and captured output for a failing test |
| Looks up accounts, products, promo codes, etc. from a JSON fixture file |
| Files a Jira Cloud bug; writes a draft file instead when no credentials are set |
Prompt: triage_failures walks the assistant through run → inspect → re-run to rule out flakiness → classify (product defect, test issue, environment) → file bugs only for real defects.
Related MCP server: TestPilot MCP
Playwright, TestNG, Maven Surefire, JUnit 5, pytest and most CI tools all produce JUnit XML, so one parser covers web and mobile suites. See examples/mcp-tests.playwright-and-testng.json for a Playwright web suite next to an Appium + TestNG iOS suite.
{
"suites": [
{
"name": "web-e2e",
"command": "npx playwright test --reporter=junit",
"cwd": "../my-web-app",
"junitReport": "test-results/junit.xml",
"grepFlag": "--grep"
}
],
"testDataFile": "data/test-data.json",
"jira": { "projectKey": "QA", "issueType": "Bug" }
}npm install
npm test # 10 tests: JUnit parsing + every tool end to end over an in-memory MCP transport
npm run buildConnect it to Claude Desktop or Claude Code (claude_desktop_config.json / .mcp.json):
{
"mcpServers": {
"test-automation": {
"command": "node",
"args": ["/absolute/path/to/test-automation-mcp-server/dist/index.js", "/absolute/path/to/mcp-tests.config.json"],
"env": {
"JIRA_BASE_URL": "https://your-company.atlassian.net",
"JIRA_EMAIL": "you@company.com",
"JIRA_API_TOKEN": "..."
}
}
}
}Leave the Jira variables out to keep create_jira_bug in draft mode (drafts go to reports/jira-drafts/).
The model never gets a raw shell. It can only run suites defined in the config file. Test filters are shell-quoted, so a test name cannot inject commands.
Small, model-friendly outputs. Tools return counts and one line per failure; details come only when asked. That keeps context windows small on large suites.
Errors are tool results, not crashes. A bad suite name or a missing report comes back as isError: true with a clear message the assistant can act on.
Safe by default. Jira writes are drafts until credentials are provided, and read-only tools are annotated readOnlyHint.
TypeScript · MCP TypeScript SDK · Zod · Vitest · JUnit XML · Jira Cloud REST API
MIT
File a bug for a confirmed product defect. Include reproduction steps, expected vs actual result and the failing test name. Without Jira credentials this writes a draft file instead of creating a real issue.
| Name | Required | Description | Default |
|---|---|---|---|
| labels | No | ||
| summary | Yes | ||
| priority | No | ||
| description | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full load and does disclose a meaningful side effect: without Jira credentials it writes a draft file instead of creating a real issue, which is exactly the kind of degraded-mode behavior an agent must know. It still omits permission/auth requirements beyond that, whether duplicates are created, and what a successful call returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with no filler; the purpose and required content come first and the credentials fallback follows. Small overlap with the title ("File a bug"), but nothing is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter write tool with no annotations and no output schema, the description covers purpose and one important behavioral trait but leaves parameter meaning and result expectations unaddressed. An agent could call it, but would be guessing about priority and labels.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% on four parameters, so the description must compensate and largely does not. It describes what belongs inside the free-text description field but never mentions the summary, the priority enum values, or the labels array, leaving half the surface undocumented anywhere.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
"File a bug for a confirmed product defect" gives a specific verb (file/create) and resource (Jira bug), plus a qualifying condition that separates it from the read-only siblings like get_failure_details or get_test_results. It does not explicitly name a sibling, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Confirmed product defect" establishes a clear precondition for invoking this tool versus simply reading test output, and the enumeration of required content (reproduction steps, expected vs actual, failing test name) frames intent. No explicit exclusion or named alternative is offered, keeping it at 4.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Return the error message, stack trace and captured output for failing tests whose name contains the given text.
| Name | Required | Description | Default |
|---|---|---|---|
| suite | Yes | ||
| testName | Yes | Full or partial test name |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=true; the description usefully adds the return payload (error message, stack trace, captured output) in the absence of an output schema. It does not state no-match behavior or result size, which keeps it below a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence with zero filler, front-loading the returned content before the filter condition. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-param read-only tool with no output schema, the description covers what is returned and how the name filter works. Only the undefined 'suite' scope and no-match behavior keep it short of complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: testName is documented as 'Full or partial test name', and the description reinforces that this is a 'contains' match. The required 'suite' parameter is undocumented in both schema and description, which is a real gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb ('Return') plus precise resource ('error message, stack trace and captured output for failing tests'), which tells an agent exactly what comes back. It is distinguishable in spirit from get_test_results, but the description never names or contrasts with that sibling.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'failing tests whose name contains the given text' implies the usage context (drilling into a known failure) and the substring matching rule. However, it gives no explicit when-to-use versus get_test_results, and no guidance on what happens when nothing matches.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Summarize the most recent JUnit report for a suite without re-running it.
| Name | Required | Description | Default |
|---|---|---|---|
| suite | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, so the safety profile is covered. The description adds useful context that no execution occurs ('without re-running it') and that only the latest report is read, but it omits what happens when no report exists or what the summary contains.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with the action and resource, with no filler. Every clause (most recent, JUnit report, without re-running) carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple and has no output schema, so return-value detail is not required. Still, the definition leaves the agent guessing on the suite identifier format and on fallback behavior when no prior report exists.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% for the single 'suite' parameter, so the description must compensate and largely doesn't. 'For a suite' only vaguely indicates the parameter's role; it never says whether the value is a suite name, path, or ID.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb ('Summarize') plus resource ('most recent JUnit report') with scope ('for a suite'), and the phrase 'without re-running it' distinguishes it from run_tests. Sibling differentiation is implicit rather than named, so not a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The clause 'without re-running it' implies the use case (inspect existing results rather than trigger a new run), which implicitly routes against run_tests. However, no alternatives are named and there is no guidance on when to prefer get_failure_details or query_test_data for drill-down.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
List the test suites this server can run, with their commands and report locations.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe read, so the description only needs to add context; it does so by disclosing what comes back (commands and report locations), which is important because there is no output schema. It doesn't mention whether the list is static or paginated, keeping it off a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One front-loaded sentence with no filler. Every clause — what is listed, and that commands and report locations are included — carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema and no parameters, the description carries the return-value burden and does so adequately by naming the fields the agent will get. It stops short of describing entry structure or count, so it is not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate; the baseline for a param-free tool applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (List) and resource (test suites) and adds what each entry contains (commands and report locations). It is clearly distinguishable from run_tests or get_test_results by scope, though it doesn't explicitly name those siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The discovery framing is implied — an agent would reasonably infer this is the entry point before run_tests — but the description never says when to use this versus run_tests or query_test_data, and offers no exclusions or ordering advice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Look up test accounts, products or other fixtures, filtered by exact field values.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| entity | Yes | e.g. users, products | |
| filter | No | Field/value pairs, e.g. {"role": "admin"} |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already establishes the safety profile, so the bar is lower. The description contributes the important constraint that matching is exact field values only, but says nothing about pagination, ordering, or what happens when no rows match.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with zero filler; the resource comes first and the filtering constraint follows. Nothing is padded or repeated from the title.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with a nested-object filter parameter and no output schema, the description leaves real gaps: whether filters AND together, whether only string equality is supported (no ranges or operators), and whether results are bounded by limit. Adequate as a minimal definition but not complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 67% with entity and filter documented, and the description usefully reinforces that filter works on exact values (consistent with the string-only additionalProperties). The 'limit' parameter is undocumented in both schema and description, so the description doesn't fully compensate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('look up') and resource ('test accounts, products or other fixtures'), and adds the query model ('filtered by exact field values'). The resource is clearly distinct from the sibling tools (suites, runs, results, bugs), though no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied: exact-match filtering over fixtures, not running or inspecting tests. There is no explicit when-to-use/when-not guidance and no pointer to a sibling for related needs (e.g. get_test_results for actual outcomes).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Run a configured test suite (optionally filtered) and return pass/fail counts and the failing tests. Use get_failure_details next to dig into a specific failure.
| Name | Required | Description | Default |
|---|---|---|---|
| suite | Yes | Suite name from list_test_suites | |
| filter | No | Optional test name filter, passed with the suite grep flag |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose that it executes a pre-configured suite and returns counts plus failing test names. However, it says nothing about execution cost/latency, timeouts, or whether running a suite has side effects, which matters for a test-execution tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly written sentences, with the primary action and output front-loaded and the alternative-routing hint trailing. Nothing redundant.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, but the description compensates by naming the returned fields. Parameters are fully covered by the schema. Only the lack of execution-behavior detail (duration, failure semantics) keeps it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so both parameters are already documented (including the grep-flag behavior of filter). The description only restates that the suite is 'configured' and can be 'optionally filtered', adding little beyond the schema baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (run) and resource (a configured test suite), plus the output shape (pass/fail counts and failing tests). It also distinguishes itself from get_failure_details, so an agent can route without opening a schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly notes the optional filter and names the follow-up tool (get_failure_details) for drilling into a specific failure. The when-to-use-this-vs-alternative condition is spelled out clearly.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.0create_jira_bugget_failure_detailsget_test_resultslist_test_suitesquery_test_datarun_testsScored across 6 tools
Each tool targets a distinct step in the test workflow: discovering suites, running tests, reading recent results, drilling into failures, querying fixtures, and filing a bug. The run_tests vs. get_test_results distinction is explicit: one executes, the other summarizes an existing report. get_failure_details is also clearly scoped to digging into named failing tests.
All tools use consistent snake_case with a clear verb_noun pattern: list_test_suites, run_tests, get_test_results, get_failure_details, query_test_data, create_jira_bug. The convention is predictable throughout with no mixed styles.
Six tools is well-scoped for a test automation server, covering the core workflow without redundant or filler operations. Each tool has a clear role in the run-inspect-debug-file lifecycle.
The surface covers the main workflow from listing suites through running tests, inspecting failures, querying fixtures, and filing a Jira bug. Minor gaps remain, such as historical results beyond the most recent report and suite configuration/management, but agents can work around them.
Direct access to Cypress tests results and accessibility reports in your AI workflow.
The AI orchestration agent for modern software teams.
Write, run and check natural-language browser tests of your web app from your AI assistant.
Approved test intent, reviewed Playwright automation and run evidence, inside your editor.
