Model Context Protocol 서버로, AI 어시스턴트가 추측하도록 요구하는 대신, 구체적인 QA/테스트 인텔리전스 도구 세트를 제공합니다. Playwright 실행 요약, 여러 실행에서 플래키 테스트 찾기, 휴리스틱 기반 실패 원인 설명, Playwright 테스트 스텁 생성, GitHub 저장소의 최근 CI 상태 읽기가 그 도구들입니다. Claude Desktop, Claude Code 또는 기타 다른 MCP 호환 호스트가 이 서버에 연결되면, 이 기능들은 자연어 추측이 아닌 실제 도구 호출이 됩니다.
이것은 "AI에 실제 데이터를 주자, 감에 의존하지 말자"는 QA 자동화 포트폴리오의 한 축입니다. 이 포트폴리오의 다른 저장소(healthcare-qa-automation-framework, healthcare-data-quality-framework)에서 생성하는 Playwright JSON 리포트와 CI 실행이 그 원, 이 서버는 그것들을 어시스턴트가 실제로 질의할 수 있는 형태로 바꿔 줍니다.
프로토콜 연결은 일반 함수 위에 있는 얇은 계층일 뿐입니다. 각 도구의 실제 로직은 src/playwright/, src/github/, src/tools/ 에서, 작고 (거의) 순수하며 독립적으로 단위 테스트되는 모듈들로 구현됩니다. src/index.ts 는 이들 로직을 MCP SDK에 등록하는 일만 합니다. 실제 파싱·플래키 감지·휴리스틱과 같은 진짜 로직은 빠른 단위 테스트로 보호하고, 프로토콜 연결 부위는 별도의 종단 간 스모크 테스트(아래 참조)로 보호하는 방식입니다.
GitHub 접근은 포트폴리오의 다른 LLM 통합과 동일한 패턴으로 플러그인할 수 있습니다. GitHubClient는 인터페이스입니다. RealGitHubClient는 GitHub REST API 에 대한 fetch를 감싸고, FakeGitHubClient는 준비된 데이터를 반환합니다. 그래서 get_repo_ci_status 는 오프라인에서도 완전히 단위 테스트 가능하면서, 실제 클라이언트는 진짜로 실행 가능한 코드로 남습니다. 개발자 컴퓨터의 MCP 호스트는 정상적인 인터넷 접근이 가능하며, 이런 환경이 없는 곳(예: 이 저장소의 CI 샌드박스) 에도 존재할 수 있습니다.
테스트 스텁 생성은 의도적으로 결정적이고 LLM 기반이 아닙니다. 이름이 지정된 data-testid 상호작용 시퀀스가 주어지면 generate_playwright_test_stub 은 늘 동일한 출력을 만듭니다. 즉·무료·재현 가능한 결과입니다. 다른 작업을 위한 다른 도구(스펙이나 스토리에서 테스트 케이스를 생성하는 것처럼 LLM이 실제 가치를 더하는 작업)는 이 포트폴리오의 또 다른 별도 프로젝트입니다.
단위 테스트만이 아니라 프로토콜 수준의 스모크 테스트. smoke-test.mjs 는 빌드된 서버를 실제 자식 프로세스로 실행하고, 공식적인 MCP Client로 stdin/stdout을 통해 구동합니다. tools/list 뒤에 각 도구별 tools/call — 실제로는 그 뒤의 함수뿐 아니라 와이어 프로토콜이 잘 동작한다는 것을 CI에서 검증해 줍니다.
Related MCP server: Playwright Debug MCP Server
도구 | 기능 |
| Playwright JSON 리포터 출력을 파싱하여 통과/실패/제한시간 초과/스킵/플래키 개수, 기간, 전체 실패 상세 정보로 정리합니다. |
| 2개 이상의 Playwright JSON 리포트에서 같은 테스트들을 비교한 뒤, 결과가 일관되지 않은 테스트를 보고합니다. |
| 원시 오류 메시지를 휴리스틱하고 패턴 매칭 방식으로 패턴적으로 근본 원인을 추정합니다. 완전 오프라인으로 작동합니다. |
| 이름이 지정된 |
| 공개 GitHub 저장소의 최근 Actions 실행을 가져와 통과/실패/진행 중 개수를 요약합니다. |
src/
index.ts MCP server: registers all 5 tools, connects over stdio
playwright/reportParser.ts Pure Playwright-JSON parsing: summarizeReport, findFlakyTests
github/githubClient.ts GitHubClient interface + RealGitHubClient (fetch) + FakeGitHubClient
tools/
explainFailure.ts Heuristic pattern-matching (ported from the triage_failure.py
script in healthcare-qa-automation-framework)
generateTestStub.ts Deterministic Playwright test-file template generator
getRepoCiStatus.ts Wraps GitHubClient into pass/fail/in-progress counts
test/ Vitest unit tests for every module above
fixtures/ Sample Playwright JSON reports (3 runs, used for flaky-test tests)
smoke-test.mjs End-to-end MCP protocol smoke test (real child process, real client)
.github/workflows/ci.yml Type-check, unit tests, build, protocol smoke testTypeScript · Node.js · @modelcontextprotocol/sdk · Zod · Vitest · GitHub REST API · GitHub Actions
npm install
npm run build # compiles to dist/
npm test # unit tests (27 tests, fully offline)
npm run smoke # builds, then drives the real server over stdio via the MCP client모든 MCP 호환 클라이언트가 아래처럼 빌드된 서버를 가리키게 하세요. 예를 들어 Claude Desktop의 claude_desktop_config.json:
{
"mcpServers": {
"qa-copilot": {
"command": "node",
"args": ["/absolute/path/to/qa-copilot-mcp-server/dist/index.js"]
}
}
}get_repo_ci_status의 인증되지 않은 GitHub 요청 한계(토큰 없으면 시간당 60회, 있으면 5,000회)를 늘리려면 환경변수에 GITHUB_TOKEN을 설정하세요. 간헐적 사용에는 토큰이 없어도 정상 동작합니다.
main 으로의 모든 푸시와 풀 리퀘스트는 하나의 잡으로 수행됩니다: 타입 검사(tsc --noEmit), 단위 테스트 스위트(vitest run), 프로덕션 빌드, 그리고 빌드된 결과를 대상한 프로토콜 수준 스모크 테스트(node smoke-test.mjs)를 실행합니다. 어떤 CI 환경에서도 네트워크 접근을 가정하거나 요구하지 않습니다. get_repo_ci_status 는 단위 테스트에서 FakeGitHubClient 대상으로만 검증되고, 스모오크 테스트는네트워크 없는 환경에서도 서버가 MCP의 isError 채널로 오류를 보고만 하면(크래시 없이) 통과로 처리합니다.
범용 GitHub 또는 CI 대시보드가 아닙니다. 이 저장소의 다른 리포지토리가 실제 만들어내는 Playwright 리포트 / GitHub Actions 형태를 중심으로 설계된, 작고 집중된 도구 모음입니다. AI 어시스턴트가 테스트 결과를 실재 동작하고 검증 가능한 데이터로 얻을 수 있게 하는 방법을 보여주기 위한 것이며, 문단을 요약한 자연어 정보를 가지고 테스트 결과를 추론하게 하는 것과는 대적입니다.
Given a Playwright (or similar) error message, returns a heuristic, pattern-matched guess at the root cause and a suggested next step. Fully offline - no LLM call.
| Name | Required | Description | Default |
|---|---|---|---|
| errorMessage | Yes | The raw error/assertion message from a failed test. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since there are no annotations, the description carries the full burden of behavioral disclosure. It does well by stating that the output is a 'guess', that the analysis is 'heuristic, pattern-matched', and that it is 'Fully offline - no LLM call'. This conveys uncertainty and operational constraints clearly, though a bit more detail about output format or confidence would have made it even richer.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences accomplish all necessary tasks: defining input, action, output, and an important operational constraint. The key behavioral details are front-loaded, with no wasted or repetitive wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a single parameter, absent annotations, and no output schema, the description is nearly complete: it defines the input, the output type (root cause and next step), and the heuristic/offline behavior. The only minor gap is a precise description of the return value's shape, but for this simple tool the current wording is sufficient for an agent to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema's description of errorMessage already covers 'raw error/assertion message from a failed test'. The tool description adds value by scoping the input to a 'Playwright (or similar) error message', which clarifies the expected domain. With exceptionally high schema coverage at the file level, the description's incremental input is small but relevant.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies a specific verb ('returns') and a distinct resource ('a heuristic, pattern-matched guess at the root cause and a suggested next step'). The input is scoped to 'Playwright (or similar) error messages', making it easy to differentiate from sibling tools like summarize_test_run or list_flaky_tests. An agent can understand this tool's purpose without inferring anything.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The opening 'Given a Playwright (or similar) error message' clearly indicates when this tool should be used. However, the description does not explicitly name sibling tools as exclusions or state when not to use it, so the comparison with alternatives is left to the agent's own judgment.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Deterministically scaffolds a runnable Playwright test file from a named sequence of data-testid interactions (fill, click, expectText, expectVisible). Template-based, not LLM-based - instant and 100% reproducible.
| Name | Required | Description | Default |
|---|---|---|---|
| steps | Yes | ||
| testName | Yes | The test's title, e.g. "logs in with valid credentials". | |
| describeName | No | The test.describe() block title. Defaults to "Generated suite". |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description does a solid job of setting expectations: it is template-based, deterministic, instant, reproducible, and produces a runnable file. It does not disclose whether the file is written to disk or returned as text, but the key behavioral traits for agent decision-making are clearly surfaced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is only two sentences, concise and front-loaded. It states the action/resource first, then adds the relevant constraints (deterministic, template-based, reproducible). No sentence is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a code generation tool with three parameters and no output schema, the description communicates enough for most invocation decisions: inputs are structured steps, the output is a runnable Playwright test stub, and generation is deterministic. The main missing piece is whether 'scaffolds' means writing a file, returning code, or both, which could affect how the agent uses or verifies the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents testName and describeName, covering about 67% of parameters. The description adds minor value by characterizing steps as a named sequence of data-testid interactions and enumerating the action types, but it does not provide additional meaning about testName or describeName beyond the schema. The nested schema definition of steps already covers most of the needed detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('scaffolds') with a clear resource ('a runnable Playwright test file') and defines the input as a sequence of data-testid interactions with named action types. It also differentiates itself from analysis-focused siblings like summarize_test_run and explain_failure by emphasizing instant, deterministic generation rather than interpretation.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the tool should be used to generate a Playwright test stub when you have a deterministic sequence of data-testid interactions, and emphasizes that it is not LLM-based, which rules out ad-hoc/creative generation use cases. It does not explicitly name alternatives or exclusion conditions, but no sibling tool competes for the same role.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetches the last N GitHub Actions workflow runs for a public repo (owner/repo) and summarizes pass/fail/in-progress counts alongside the raw run list. Requires network access from wherever this server is running; set GITHUB_TOKEN in the environment to raise the unauthenticated rate limit.
| Name | Required | Description | Default |
|---|---|---|---|
| repo | Yes | Repo name, e.g. "healthcare-qa-automation-framework". | |
| limit | No | How many recent runs to fetch (default 10). | |
| owner | Yes | Repo owner, e.g. "sharath9271-design". |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since no annotations are provided, the description bears the full burden, and it adds real value: it reveals that the tool needs network access, only works on public repos, and can raise its rate limit by setting GITHUB_TOKEN. 'Fetches' implies a read-only operation, and the description does not conceal any dangerous or surprising side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loaded with the core fetching and summarizing behavior, followed by concise operational caveats. Nothing feels like filler; both sentences earn their place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only, three-parameter tool with no output schema, the description covers everything essential: what is fetched, what the response contains, public-repo scope, network requirements, and the optional auth token. An agent has enough context to decide whether to reply and how to handle it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already documents all three parameters (owner, repo, limit) with types, explanations, and examples, covering 100% of its parameters. The description merely restates the limit in prose as 'last N', so it does not add meaningful parameter semantics beyond what the schema conveys. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb-resource pair: 'Fetches the last N GitHub Actions workflow runs' and 'summarizes pass/fail/in-progress counts alongside the raw run list'. This clearly distinguishes it from the sibling test-analysis tools, which talk about tests, flakes, and failures rather than repo-level CI status.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended use is implied and fairly obvious: an agent would call it when asked for recent CI status of a public repo. However, it never names a sibling alternative or states when this tool is not appropriate, so routing is left to inference rather than explicit guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Compares the same tests across two or more Playwright JSON reports (e.g. the last N CI runs of the same suite) and returns every test whose outcome wasn't consistent - it passed in some runs and failed/timed out in others.
| Name | Required | Description | Default |
|---|---|---|---|
| reportJsons | Yes | Two or more Playwright JSON reporter outputs, oldest first, as raw strings. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations available, the description correctly conveys that this is a read-only comparison and what task qualifies as flaky. However, it does not disclose how missing tests across reports are treated, whether the result is influenced by report ordering beyond the schema note, or what fields are returned per test — leaving some behavioral details implied rather than explicit.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One tight sentence that states the operation, the input shape, the example use, and the result. There is no filler, and the purpose appears immediately. It is concise despite the long sentence because every clause adds new information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter query tool with no output schema, the description explains the input to the comparison, what counts as 'flaky', and what the output represents. It omits edge-case details, such as how test records without entries in all reports are handled or the exact shape of each returned test, but the overall information is sufficient for an agent to invoke it correctly for a typical flaky-test scenario.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already documents reportJsons as raw Playwright JSON reporter outputs, oldest first, with minItems=2 (100% schema coverage). The description adds the meaningful context that these should be 'the last N CI runs of the same suite', which clarifies the intended input beyond the bare schema definition.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair ('Compares the same tests across two or more Playwright JSON reports') and explicitly names the result ('returns every test whose outcome wasn't consistent'). It clearly differentiates this from siblings like get_repo_ci_status or explain_failure by focusing on flaky test detection across multiple runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides contextual use ('two or more... e.g. the last N CI runs of the same suite') and makes the minimum-input condition explicit. It does not spell out 'when not to use' or name alternatives, but the multi-report scope plus sibling tool context makes the boundary reasonably clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Parses Playwright's JSON reporter output and returns pass/fail/timeout/skipped/flaky counts, total duration, and full details on every failure.
| Name | Required | Description | Default |
|---|---|---|---|
| reportJson | Yes | The raw contents of a Playwright JSON reporter output file (results.json). |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations present, the description carries the full burden; it handles this well by describing what the tool does: parse JSON and return counts, duration, and failure details. It doesn't discuss edge cases, output formatting, or potential errors, but for a straightforward parsing and summarization tool this is a substantially clear behavioral explanation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that immediately states the verb, the input, and the exact outputs. It is front-loaded with 'Parses...' and contains no filler or redundant phrasing.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter, conceptually simple tool, the description fully covers what the input is (Playwright JSON reporter output), what the tool produces, and how the result will be structured. No output schema is present, but the return values are described in enough detail for an agent to consume the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the parameter is already fully documented—'The raw contents of a Playwright JSON reporter output file (results.json)'. The tool description adds no additional parameter-level meaning beyond re-stating that it parses Playwright JSON, so the baseline of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a clear action ('parses Playwright's JSON reporter output') and reports a specific set of results: pass/fail/timeout/skipped/flaky counts, total duration, and failure details. This distinguishes it from sibling tools like list_flaky_tests or explain_failure, which have narrower or different purposes.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The intended context is clear: this tool is used when a Playwright JSON reporter output is available and a summary of counts, duration, and failures is needed. It does not explicitly mention when not to use it or compare itself with sibling tools, but the usage context is strongly implied and unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v1.0.0explain_failuregenerate_playwright_test_stubget_repo_ci_statuslist_flaky_testssummarize_test_runScored across 5 tools
Each tool targets a distinct responsibility—summarizing test output, identifying flaky tests, explaining failures, generating test stubs, and fetching CI status. There is no meaningful overlap between any pair of tools.
All tool names follow a consistent verb_noun pattern with clear action-prefixed names: summarize_, list_, explain_, generate_, get_. Minor pluralization differences do not create confusion.
Five tools cover a tightly scoped QA workflow—analyze results, detect flakiness, explain failures, generate tests, and check CI—without redundancy or bloat.
The core QA reporting and triage loop is well supported, from parsing reports to explaining failures and detecting flaky tests. Minor gaps like actionable test-edit or test-deletion operations are absent, but they fall outside the visible purpose.
Direct access to Cypress tests results and accessibility reports in your AI workflow.
Approved test intent, reviewed Playwright automation and run evidence, inside your editor.
Run, debug, and triage tests from your IDE using natural language, no dashboard switching, no manual data transfers. The TestMu AI (formerly LambdaTest) MCP Server is a single remote server exposing four tool suites: HyperExecute — analyze your project, generate YAML configs and test runner commands, then monitor jobs and sessions. Automation — pull a TestID's details plus command, network, and console logs into one chat for instant root-cause analysis. Includes mobile app upload. SmartUI — explain pixel, layout, DOM, and perceptual changes in a visual regression run, with context-aware React/HTML/CSS fixes. Accessibility — audit any public URL or a local React app against WCAG and get ready-to-apply remediation steps. Connects over https://mcp.lambdatest.com/mcp using OAuth 2.1 — no API keys in your config. One-click install in Cursor; works with Claude, GitHub Copilot, Cline, and any MCP client. Tests execute on the TestMu AI cloud: 3,000+ browsers and 10,000+ real devices.
Create and manage test cases, test runs and results in Testiny from your AI assistant.
