Which integrations are available for this server?

Provides a GitHub connector for integrating GitHub repositories and issues into QA project workflows. Provides a Jira connector for linking requirements and QA artifacts to Jira issues. Provides Slack integration behind service/client abstractions for QA workflow collaboration.

How do I use QA MCP Server?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@QA MCP Server Analyze requirement for user login and generate test cases" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

de en es ja ko ru zh

QA MCP Server

by sanumenon

Overview Schema Source Related Servers Discussions Admin

Python

Remote

QA MCP

QA MCP is a Model Context Protocol (MCP) server for structured software-quality workflows.

The project is being developed incrementally toward a full-fledged AI-powered QA platform using:

Layered architecture
Test-first development
Pydantic-based contracts
Persistent SQLite storage
Immutable QA versioning
Project import/export
Safe external connectors
MCP tool boundaries
LLM-assisted QA analysis and automation generation
Automation candidate selection
Playwright automation generation
Controlled automation execution
Eventual QA-agent orchestration
Eventual CI/CD and hosted product capabilities

CONTINUITY RULE: This README is the authoritative development, deployment, roadmap, and continuity checkpoint for future QA MCP development sessions. Read it before starting new development. Do not recreate completed work.

1. CURRENT DEVELOPMENT CHECKPOINT

Current implementation checkpoint: P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation

P2-S9.17 status: COMPLETE — committed as 5759211 and f94f05c, pushed to origin/main.

Current Git base: f94f05c — Fix local development operator access (corrective commit for P2-S9.17). This commit is on main and origin/main.

Approved product direction: Model B — Internal QA-Team Platform

Latest committed checkpoint: P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation, committed as 5759211 (implementation) and f94f05c (local development operator correction), pushed to origin/main.

Repository: https://github.com/sanumenon/qa-mcp

Branch: main

Historical P2-S9.15 verification baseline:

Focused artifact/workspace/history/API tests: 30 passed
Focused artifact-review/execution browser test: 1 passed
Full regression: 329 passed
Warnings: 8
Failures: 0
git diff --check: clean
Project Workspace two-artifact review and execution browser verification: passed
Existing artifact response shape, API/MCP fields, persistence schema, and execution behavior preserved
P2-S9.15 is committed as ba9e27c and pushed to origin/main

Latest verified implementation notes

Bedrock test-case generation hardening

Commit 9e5f8f7 normalizes a valid single test-case object into the required {"test_cases": [...]} response wrapper.
Commit 70d1f8f increases the Bedrock maxTokens configuration from 4096 to 12000.
The full regression suite remains green after both changes.
A single test-case response is accepted only when it matches the valid test-case schema; unrelated or incomplete payloads remain errors.
The generation prompt already requires comprehensive coverage of positive, negative, and edge scenarios.
Runtime verification has now confirmed that Bedrock returns a complete multi-case suite.

Bedrock runtime timeout resolution

The remaining runtime failure was not caused by guardrails or incomplete test-case parsing.

The actual production failure was a Bedrock ReadTimeoutError caused by the default 60-second read timeout. Real test-case generation could exceed that duration.

The production Bedrock configuration was hardened to:

Connect timeout: 60 seconds
Read timeout: 180 seconds
Retry mode: standard
Maximum attempts: 1

The configuration supports environment-variable overrides for these values.

A real diagnostic run using the hardened configuration completed successfully and generated a multi-case suite. Scenario-batched generation was therefore not required to resolve this timeout blocker.

QA suite completeness verification

The live Customer Portal QA-suite endpoint was verified using the actual QASuiteWorkspaceRequest contract.

The successful runtime verification returned:

HTTP status: 200
test_cases response: wrapped object
Generated test cases: 50
First ID: TC001
Last ID: TC050

An earlier live verification also produced 60 test cases, confirming that the generated collection is not limited to a single test case. The exact count may vary with the LLM response.

The established backend contract remains:

test_cases
└── test_cases[]

No backend API contract was changed.

Related MCP server: test-case-generator-ai-mcp

Completed capabilities

The following capabilities are implemented and verified:

Persistent project QA workspace.
Requirement analysis and test-case generation.
Test-case selection using Select All and Clear All.
Selective QA-suite persistence.
QA-suite version feedback after successful persistence.
Automation test-case generation.
Executable Playwright code generation.
Automation execution service.
Execution history persistence.
Execution result review.
Environment-independent Playwright URL handling.
BASE_URL injection into generated automation execution.
Configurable execution timeout.
Configurable automation workspace root.
Configurable workspace retention.
Controlled environment-variable propagation.
Hardened automation execution configuration.
Focused and full regression coverage for the completed implementation.

Current implementation boundary

P2-S9.14-B adds project-scoped execution metrics and recent failure review to the existing Project QA Workspace. P2-S9.14-C adds read-only review of generated automation artifacts in that workspace, reusing artifact data already returned by the workspace API. P2-S9.14-D adds read-only review of persisted test case details from the existing project workspace response. Review remains separate from candidate selection, artifact review, and execution. The global Execution and Reports pages and all backend contracts remain unchanged.

P2-S9.16 — Project-Centric QA Workspace UX

Status: COMPLETE AND APPROVED — committed as 6b8c532 and pushed to origin/main

The approved deployment direction remains Model B — a trusted internal QA team. P2-S9.16 organizes the existing Project QA Workspace into these areas:

Overview
Requirements
Test Cases
Automation
Executions
Reports

The workspace carries project context through the existing project_id. Overview metrics and recent failure information derive from existing workspace, reporting, and failure-analysis data. The Requirements area retains the existing QA-suite generation, generated-case review and selection, and selective-save workflow. Client-side testcase search and filtering use persisted fields already returned by the workspace API.

P2-S9.16 preserves the explicit separation between Review and Execute and safe, inert rendering. It includes responsive layout and accessibility improvements. Existing API and MCP contracts, persistence behavior, database schema, and execution behavior are unchanged.

Validation

Focused web tests:                 30 passed, 1 warning
Full regression suite:             329 passed, 8 warnings, 0 failures
JavaScript syntax checks:           passed
git diff --check:                    clean
Project Workspace browser flow:    passed

Browser verification covered project navigation and overview metrics, generated and persisted test-case review, search/filter and selection/save, inert rendering, automation candidate selection, artifact review, explicit execution, execution history and insights, and empty states. JavaScript syntax checks completed. API/MCP contracts, persistence/database schema, and execution behavior remain unchanged. The implementation is complete and approved, committed as 6b8c532, and pushed to origin/main.

P2-S9.16 does not claim internal network deployment readiness. Model B readiness still requires decisions about identity/access and SQLite backup/recovery. Execution remains a local subprocess and is not a sandbox. Model C network exposure and Model D untrusted execution remain out of scope.

P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation

Status: COMPLETE

Implementation commit: 5759211 — Complete P2-S9.17 internal identity and authorization

Corrective commit: f94f05c — Fix local development operator access (ensures configured development subject maps to trusted LOCAL_OPERATOR identity)

The selected identity provider is Google Workspace using Google OAuth 2.0 / OpenID Connect. Google client credentials, redirect URI, and Workspace domain are deployment settings; the Workspace domain is not hard-coded. Production startup requires Google OIDC configuration, a 32-character session secret, an HTTPS redirect URI, and Secure session cookies. The local development actor is configuration-controlled and is rejected in production.

Identity and project access

Authlib validates Google OIDC discovery issuer, ID-token signature against Google JWKS, audience, expiry, and nonce. QA-MCP accepts only a verified email and the configured Google hd Workspace domain. The stable Google sub is the actor key; access/ID tokens are not stored in the application session or exposed to browser JavaScript.
Signed, HttpOnly, SameSite=Lax sessions carry the verified subject/email and expiry. Browser state-changing requests use a session-bound CSRF token.
The minimal roles are global_admin (configured Google subjects) and project-scoped project_member. A new project's creator receives membership. Global admins provision additional memberships through the protected project-membership API.
The versioned SQLite migration adds membership and security-audit tables transactionally without rebuilding existing tables. QA_LEGACY_PROJECT_OWNERS explicitly maps legacy project IDs to Google subject IDs. Unmapped legacy projects are denied to members and counted in readiness; there is no default-wide grant.
Project authorization is enforced in shared project, versioning, workspace, and import/export services. Suite creation rejects a requirement version from a different project. Non-admin users receive project-filtered global execution/history/report/failure results.

Execution, MCP, and operations

Generated automation receives only the explicit runtime allowlist (PATH, HOME, temporary-directory/locale variables, PLAYWRIGHT_BROWSERS_PATH, CI, SYSTEMROOT) plus configured BASE_URL. Application/provider credentials and arbitrary inherited variables are excluded.
MCP remains trusted-local stdio only. Local stdio calls use the explicit local-operator trust boundary; Google browser authentication does not confer MCP access. The analyze_automation_failures registration now occurs before the stdio server starts, and the launched process inventory is verified.
/health/live reports process liveness. /api/ready checks the database and reports when legacy project ownership mapping remains outstanding. Existing /api/health compatibility is preserved.
Sensitive project/API operations record actor, project, action, outcome, timestamp, and request ID without request bodies, credentials, generated source, or execution output. Responses include a validated or generated X-Request-ID.
SQLite online backup/restore is available through python -m qa_mcp.operations backup and python -m qa_mcp.operations restore <backup-path>. Backups are integrity-checked, atomically written with mode 0600 in a mode 0700 directory, and pruned to configured retention. Stop QA-MCP before restore and verify the restored service before resuming.
The existing project-centric UI is unchanged in structure. It displays the signed-in identity and minimal sign-in, session-expiry, and access-denied states.

Configuration

Production must set QA_ENVIRONMENT=production and QA_AUTH_MODE=google, then configure the Google OAuth client ID/secret, exact callback URI, Google Workspace domain, session secret, and global-admin Google subject IDs. Explicit legacy project mappings are required before mapped members can access old projects. See Sections 16 and 17 for the environment-variable contract. Never commit the values or a populated .env file.

Validation

Focused security, workflow, execution, and MCP tests: 39 passed, 2 warnings
Full regression:                                  348 passed, 8 warnings, 0 failures
Browser verification:                             passed (mocked OIDC identity/access states and existing P2-S9.16 workflows)
JavaScript syntax checks:                         passed
MCP launched stdio tool inventory:                 passed (all 35 registered tools)
SQLite online backup/restore/integrity/retention:  passed
git diff --check:                                  clean

Google OIDC callback behavior was verified through a mocked identity boundary; no live Google credentials or organization deployment were used. P2-S9.17 does not claim deployment readiness until the organization's Google client/domain settings, HTTPS callback, production secret provisioning, legacy ownership mappings, and operational backup schedule are configured and verified. SQLite remains a single-host store, and automation remains a local subprocess rather than a sandbox. Model C network exposure and Model D untrusted execution remain out of scope.

Model B Readiness Assessment

A post-P2-S9.17 production-readiness audit was completed on 2026-10-09. The audit covered 12 security and operational areas including continuity, authentication, authorization, web security, MCP boundary, execution security, database/recovery, legacy data, health/operations, UI/workflow, test confidence, and Model B deployment readiness.

Readiness Classification: READY FOR MODEL B WITH DEPLOYMENT/OPERATIONAL PREREQUISITES

The technical implementation is complete for a trusted internal QA-team platform (Model B). Deployment requires configuration of Google OAuth credentials, Workspace domain, production session secret, global administrator subjects, legacy project ownership mappings, HTTPS deployment, and operational backup scheduling. Model C network exposure hardening and Model D untrusted execution sandboxing remain explicitly out of scope.

2. PRODUCT VISION

The long-term goal is to evolve QA MCP from a collection of QA utilities into an intelligent QA agent/platform.

Requirement
    |
    v
Requirement Understanding
    |
    v
Scenario Analysis
    |
    v
Test Case Generation
    |
    v
Test Case Review
    |
    v
QA Suite / Version
    |
    v
Automation Candidate Selection
    |
    v
Automation Case
    |
    v
Automation Validation
    |
    v
Playwright Code Generation
    |
    v
Generated Automation Artifact
    |
    v
Controlled Execution
    |
    v
Execution Results
    |
    v
Reporting / Analysis
    |
    v
QA Agent / Orchestration

Eventually the platform should support:

Jira
GitHub
Slack
CI/CD
Test repositories
Automation environments
Cloud execution
Interactive UI
Hosted/cloud product

The UI/hosted product layer must be introduced only after the core QA-agent capabilities are sufficiently stable.

3. DEVELOPMENT RULES — MUST FOLLOW

These rules apply to every future change.

Implement one phase/sub-step at a time.
Test first wherever practical.
Focused tests must pass before moving to the next increment.
The relevant feature test group must pass.
The full regression suite must pass before closing a milestone.
Never weaken or delete tests merely to obtain green output.
Inspect existing code before modifying it.
Preserve the layered architecture.
Core business logic must remain independent of MCP transport.
Persistence must remain behind repository interfaces.
External integrations must remain mockable.
LLM providers must remain replaceable.
AI output must be validated before downstream use.
Never commit secrets or a real .env file.
Never delete persistent databases merely to make tests pass.
Keep unrelated refactoring separate from feature work.
A major capability is not complete until its MCP/runtime path is verified.
Update this README at every verified milestone.
Commit only after feature, tests, README, and checkpoint have been reviewed.
Do not recreate completed work from earlier milestones.
Do not introduce production-grade container/cloud complexity before the local execution contract is stable.
Keep generated automation execution behind explicit framework validation and controlled command construction.
Preserve traceability: Requirement → Test Case → Automation Case → Artifact → Execution Result.
Do not silently change established contracts.
Prefer deterministic behavior over clever behavior.
Keep execution safety ahead of execution convenience.
Deployment/configuration details must remain documented here.
A new chat/session must begin from this README and the current GitHub main branch.

Mandatory development sequence

Read README / current checkpoint
        |
        v
Inspect GitHub main + repository state
        |
        v
Inspect existing implementation
        |
        v
Define ONE next sub-step
        |
        v
Write/update focused tests
        |
        v
Implement smallest production change
        |
        v
Focused tests green
        |
        v
Feature tests green
        |
        v
Full regression green
        |
        v
Runtime/MCP verification
        |
        v
Update README
        |
        v
git diff --check
        |
        v
Commit + push
        |
        v
Verify clean working tree

4. ARCHITECTURE

                         MCP CLIENT / AI ASSISTANT
                                      |
                                      v
                                QA MCP Server
                                      |
                                      v
                                 MCP Tool Layer
                                      |
              +-----------------------+------------------------+
              |                       |                        |
              v                       v                        v
         QA Workflows          Core Services             Connectors
              |                       |               +--------+--------+
              |                       |               |        |        |
              v                       v               Jira    GitHub   Slack
     Requirement Analyzer       Automation             |        |        |
     Test Case Generator        Execution              v        v        v
     Test Case Reviewer         Versioning           Service  Service  Service
     QA Suite Workflow          Project Context        |        |        |
     Candidate Selection                             v        v        v
     Automation Generation                         Client   Client   Client
     Automation Execution                            / \      / \      / \
                                                     Mock     Mock    Mock
                                                     Cloud    Cloud   Cloud

Layer responsibilities:

models/
    Domain and data contracts

core/
    Business/application services
    Factories
    Orchestration boundaries
    Automation execution mechanics

infrastructure/
    Persistence
    External clients
    Concrete implementations

tools/
    QA-oriented application workflows

server.py
    MCP transport and tool registration

Core business logic must not become coupled to MCP transport.

5. REPOSITORY STRUCTURE

Important current structure:

qa-mcp/
|
+-- config/
|   +-- settings.yaml
|
+-- src/qa_mcp/
|   +-- core/
|   |   +-- automation/
|   |   |   +-- candidate_generation_service.py
|   |   |   +-- candidate_selector.py
|   |   |   +-- candidate_service.py
|   |   |   +-- code_generation_service.py
|   |   |   +-- execution_config.py
|   |   |   +-- execution_runner.py
|   |   |   +-- execution_service.py
|   |   |   +-- workspace.py
|   |   |   +-- service.py
|   |   |   +-- validator.py
|   |   |
|   |   +-- github/
|   |   +-- jira/
|   |   +-- slack/
|   |   +-- import_export/
|   |   +-- project/
|   |   +-- versioning/
|   |   +-- config.py
|   |   +-- llm.py
|   |
|   +-- infrastructure/
|   |   +-- github/
|   |   +-- jira/
|   |   +-- slack/
|   |   +-- versioning/
|   |   +-- project repositories
|   |
|   +-- models/
|   |   +-- schemas.py
|   |
|   +-- tools/
|   |   +-- automation/
|   |   +-- requirement/
|   |   +-- testcase/
|   |   +-- workflow/
|   |
|   +-- server.py
|
+-- tests/
+-- data/
+-- README.md
+-- requirements.txt

6. COMPLETED PRODUCT CAPABILITIES

Phase 1 — Foundation & QA Intelligence

STATUS: COMPLETE

Completed:

MCP server foundation
Configuration loading
LLM abstraction
Mock LLM support
Requirement analysis
Test-case generation
Test-case review
End-to-end QA suite workflow

Core MCP capabilities:

health
test_llm
analyze_requirement
generate_test_cases
review_test_cases
generate_qa_suite

Phase 2 — QA Platform Foundation

Milestone	Capability	Status
P2-S1	QA Project Context	COMPLETE
P2-S2	SQLite Persistence	COMPLETE
P2-S3	Requirement & Suite Versioning	COMPLETE
P2-S4	Project Import / Export	COMPLETE
P2-S5	Jira Connector	COMPLETE
P2-S6	GitHub Connector	COMPLETE
P2-S8	Automation Pipeline	COMPLETE through current checkpoints
P2-S8.6	Automation Candidate Selection	COMPLETE
P2-S8.7	Candidate → Automation Generation	COMPLETE
P2-S8.8	Automation Case Validation	COMPLETE
P2-S8.8+	Automation Code Generation	COMPLETE
P2-S8.9	Controlled Automation Execution	COMPLETE

Slack integration exists behind service/client abstractions.

7. PROJECT CONTEXT AND PERSISTENCE

Conceptually:

QAProject
    |
    +-- project_id
    +-- name
    +-- description
    +-- application
    +-- environment
    +-- metadata
    +-- requirements
    +-- test suites

Persistence:

ProjectContext
      |
      v
ProjectRepository
      |
      v
SQLiteProjectRepository
      |
      v
SQLite

Database:

data/qa_mcp.db

Important: Never delete the persistent database merely to make tests pass.

Persistence-focused tests should use isolated database state.

8. EXTERNAL CONNECTORS

Jira

Abstraction:

MCP
 |
 v
JiraService
 |
 v
JiraClient
 +-- MockJiraClient
 +-- JiraCloudClient

Current real operations are read-only:

get_jira_issue(issue_key)
search_jira_issues(jql, max_results=50)

No Jira write operations are part of the completed connector milestone.

GitHub

Abstraction:

MCP
 |
 v
GitHubService
 |
 v
GitHubClient
 +-- MockGitHubClient
 +-- GitHubCloudClient

Current read-only tools:

get_github_repository(owner, repository)
get_github_issue(owner, repository, issue_number)
get_github_pull_request(owner, repository, pull_number)
search_github_issues(query, max_results=50)

No GitHub write operations are part of the completed connector milestone.

Slack

Abstraction:

SlackService
    |
    v
SlackClient
    +-- MockSlackClient
    +-- SlackCloudClient

Current tools include:

get_slack_channel
get_slack_messages
search_slack_messages
get_slack_thread

9. AUTOMATION PIPELINE

Test Cases
    |
    v
Automation Candidate Selection
    |
    v
Automation Case
    |
    v
Automation Validation
    |
    v
Framework-specific Code Generation
    |
    v
GeneratedAutomationArtifact
    |
    v
Controlled Execution
    |
    v
AutomationExecutionResult

Candidate selection deliberately distinguishes:

Recommended for automation
        |
        +---- Automated
        |
        +---- Manual-only

Manual-only test cases must not be sent to the automation generator.

10. AUTOMATION CHECKPOINTS ALREADY COMPLETE

P2-S8.6 — Candidate Selection

AutomationCandidateSelector / AutomationCandidateService

Result:

AutomationCandidateResult
    +-- candidate_ids
    +-- manual_ids
    +-- total

MCP tool:

select_automation_candidates

P2-S8.7 — Candidate → Automation Generation

Service:

AutomationCandidateGenerationService

Flow:

TestCase[]
    |
    v
Candidate Selection
    |
    v
candidate_ids
    |
    v
Generate automation ONLY for candidates
    |
    v
AutomationCase[]

Zero-candidate behavior:

No automation candidates
        |
        v
[]
        |
        v
Automation generator is NOT called

MCP tool:

generate_automation_for_candidates

P2-S8.8 — Automation Case Validation

Validator:

AutomationValidator

Result:

AutomationValidationResult
    +-- automation_case_id
    +-- test_case_id
    +-- valid
    +-- errors
    +-- warnings

Minimum integrity:

At least one automation step.
Validation failures are structured errors.
Non-blocking concerns can be warnings.
Validation remains separate from generation.

P2-S8.8+ — Automation Code Generation

Artifact:

GeneratedAutomationArtifact
    +-- id
    +-- automation_case_id
    +-- framework
    +-- language
    +-- file_name
    +-- code

Current execution target:

Framework: Playwright
Language: Python

Generated automation must be validated before downstream execution.

11. CONTROLLED AUTOMATION EXECUTION — P2-S8.9 COMPLETE

The committed local execution pipeline is:

GeneratedAutomationArtifact
        |
        v
AutomationExecutionConfig
        |
        v
AutomationWorkspace
        |
        v
AutomationExecutionRunner
        |
        v
AutomationExecutionService
        |
        v
AutomationExecutionResult
        |
        v
MCP execute_automation_code

Execution configuration

AutomationExecutionConfig
    +-- timeout_seconds = 60
    +-- workspace_root = optional

The configuration is immutable.

Automation workspace

AutomationWorkspace creates an isolated temporary directory for the generated artifact.

The workspace is cleaned up after execution unless explicit retention is requested.

The project working tree must not be used as the normal generated-artifact execution directory.

Controlled subprocess runner

AutomationExecutionRunner:

accepts an explicit command list
runs from a supplied working directory
captures stdout
captures stderr
captures exit code
measures execution duration
enforces a timeout
reports timeout separately
reports operating-system execution errors separately

The runner is injectable so tests do not need to execute real automation processes.

Execution service

Current validation:

Empty code
    -> ValueError

Missing framework
    -> ValueError

Unsupported framework
    -> ValueError

Current supported framework:

Playwright

Current Python execution command:

python -m pytest <generated_file_name>

Status mapping:

exit_code == 0
    -> PASSED

exit_code != 0
    -> FAILED

timed_out
    -> TIMEOUT

runner error
    -> ERROR

Separation:

Runner
    = process mechanics

ExecutionService
    = QA execution semantics

AutomationExecutionResult
    = stable domain contract

Execution IDs are UUID-based identifiers generated by the execution service.

12. EXECUTION SAFETY REQUIREMENTS

The current subprocess runner is a controlled local execution boundary, not the final production-grade sandbox.

Intended progression:

Current
Local controlled subprocess
        |
        v
Hardened execution boundary
        |
        v
Container / isolated execution
        |
        v
Cloud or CI execution

Mandatory safety direction:

Do not introduce arbitrary shell execution.
Do not construct unrestricted commands from user input.
Keep framework support explicit.
Keep generated filenames and execution paths controlled.
Keep execution bounded by timeouts.
Preserve workspace isolation.
Keep the runner injectable and testable.
Introduce containerization before exposing execution to untrusted production workloads.

Do not add container/cloud complexity before the local execution contract and orchestration behavior are stable.

13. MCP AUTOMATION SURFACE

Current automation-related MCP tools:

generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code

execute_automation_code(artifact):

Validates the incoming artifact through GeneratedAutomationArtifact.
Delegates to AutomationExecutionService.
Returns AutomationExecutionResult.model_dump().
Converts invalid execution-artifact input into a controlled MCP-facing error.

The MCP layer must not contain subprocess implementation details.

14. TEST STRATEGY AND CURRENT BASELINE

Test-first development remains mandatory.

Expected sequence:

Write failing test
        |
        v
Implement smallest production change
        |
        v
Focused test
        |
        v
Related tests
        |
        v
Full regression
        |
        v
Runtime/MCP verification
        |
        v
README update
        |
        v
git diff --check
        |
        v
Commit + push

Current verified baseline before P2-S9.1.a:

pytest -q
190 passed
7 warnings
0 failures

P2-S9.1.a verified regression:

pytest -q
204 passed
7 warnings
0 failures

P2-S9.1.a focused execution suite:

16 passed
1 warning

Execution service suite:

7 passed

No test was removed or weakened to obtain the current green baseline.

15. KNOWN WARNINGS / TECHNICAL DEBT

Pytest collection warnings

Pydantic models named:

TestCase
TestCaseReview

can be interpreted by pytest as possible test classes, producing PytestCollectionWarning.

These are non-functional warnings.

Future cleanup may use test-only import aliases. Keep this separate from feature work.

Pydantic settings warning

Existing:

IncompleteFieldDefinitionWarning

related to the lifespan forward reference in pydantic_settings.

It does not currently cause test failures.

Keep this as separate technical debt unless it blocks development.

16. ENVIRONMENT / .ENV DOCUMENTATION

Critical rule

The following is the documented .env template currently used by the development setup.

These are placeholders, not real credentials.

Never commit a real .env file, API token, password, or secret to Git.

The actual local .env remains developer-machine configuration.

Current `.env` template

JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token
GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org
# ---------------------------------------------------------
# Slack
# ---------------------------------------------------------
SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=
# Google OIDC production configuration (required when QA_AUTH_MODE=google)
GOOGLE_OIDC_CLIENT_ID=
GOOGLE_OIDC_CLIENT_SECRET=
GOOGLE_OIDC_REDIRECT_URI=https://qa.example.test/auth/callback
GOOGLE_WORKSPACE_DOMAIN=your-workspace-domain.example
QA_SESSION_SECRET=
# Optional identity/project administration settings
QA_GLOBAL_ADMIN_SUBJECTS=
QA_LEGACY_PROJECT_OWNERS={}

The Google entries above are placeholders. Do not use a domain or credentials that have not been supplied for the deployment.

Variable purpose

Variable	Purpose	Secret?
`JIRA_URL`	Jira Cloud base URL	No
`JIRA_EMAIL`	Jira API account email	No, but treat as configuration
`JIRA_API_TOKEN`	Jira API authentication	YES
`GITHUB_URL`	GitHub API base URL	No
`GITHUB_TOKEN`	GitHub API authentication	YES
`GITHUB_OWNER`	GitHub username/org used by configuration	No
`SLACK_URL`	Slack API base URL	No
`SLACK_TOKEN`	Slack API authentication	YES
`SLACK_DEFAULT_CHANNEL`	Default Slack channel configuration	No
`GOOGLE_OIDC_CLIENT_ID`	Google OAuth web client ID	No
`GOOGLE_OIDC_CLIENT_SECRET`	Google OAuth web client secret	YES
`GOOGLE_OIDC_REDIRECT_URI`	Exact registered OIDC callback URI	No
`GOOGLE_WORKSPACE_DOMAIN`	Allowed Google Workspace hosted domain	No
`QA_SESSION_SECRET`	Signs QA-MCP browser sessions; use at least 32 random characters	YES
`QA_GLOBAL_ADMIN_SUBJECTS`	Comma-separated stable Google `sub` identifiers for global administrators	No
`QA_LEGACY_PROJECT_OWNERS`	JSON mapping of legacy project IDs to Google `sub` identifiers	No

Deployment/configuration rule

When configuring a new environment:

Copy the documented template into a local .env.
Replace only the placeholder values required for that environment.
Never paste real secrets into this README.
Never commit the populated .env.
Verify .gitignore protects .env.
Keep configuration changes documented here when they materially affect deployment.
If new environment variables are introduced, update this section in the same development checkpoint.

17. CONFIGURATION AND ENVIRONMENT CONTRACT

The primary application configuration file is:

config/settings.yaml

The application also supports environment-specific configuration through environment variables.

Environment variables take precedence over default configuration values where supported by the implementation.

Application configuration

The configuration file contains settings for:

Application metadata
LLM provider and model configuration
Feature flags
Automation execution
Identity, authentication, and project authorization
SQLite database and backup/recovery
Jira
GitHub
Slack

Automation environment variables

Variable	Purpose
`DEFAULT_TEST_ENV`	Default execution environment when no environment is explicitly selected
`QA_BASE_URL`	Base URL for the QA environment
`STAGE_BASE_URL`	Base URL for the staging environment
`PROD_BASE_URL`	Base URL for the production environment
`QA_AUTOMATION_TIMEOUT_SECONDS`	Automation execution timeout in seconds
`QA_AUTOMATION_WORKSPACE_ROOT`	Root directory used for automation execution workspaces
`QA_AUTOMATION_KEEP_WORKSPACE`	Controls whether automation workspaces are retained after execution
`QA_AUTH_MODE`	`google` for Google OIDC; explicit `development` mode is only valid outside production
`QA_ENVIRONMENT`	Application environment; production requires Google OIDC configuration
`QA_SESSION_MAX_AGE_SECONDS`	Maximum signed browser session age
`QA_SESSION_COOKIE_SECURE`	Enables Secure session cookies; required in production
`QA_DATABASE_PATH`	SQLite database file path
`QA_BACKUP_DIRECTORY`	Protected SQLite backup destination directory
`QA_BACKUP_RETENTION`	Number of verified backups to retain

Integration environment variables

Variable	Purpose	Secret?
`JIRA_URL`	Jira Cloud base URL	No
`JIRA_EMAIL`	Jira API account email	No, but treat as configuration
`JIRA_API_TOKEN`	Jira API authentication	Yes
`GITHUB_URL`	GitHub API base URL	No
`GITHUB_TOKEN`	GitHub API authentication	Yes
`GITHUB_OWNER`	GitHub username or organization	No
`SLACK_URL`	Slack API base URL	No
`SLACK_TOKEN`	Slack API authentication	Yes
`SLACK_DEFAULT_CHANNEL`	Default Slack channel	No

Model B authentication and database operations

For production, set QA_ENVIRONMENT=production, QA_AUTH_MODE=google, QA_SESSION_COOKIE_SECURE=true, all five Google/session settings shown above, and an HTTPS callback URL registered in the Google OAuth client. Configure the Workspace domain rather than hard-coding it in application code. Set QA_LEGACY_PROJECT_OWNERS explicitly for every existing project that should be available to a project member; keys are project IDs and values are Google stable subject IDs (sub). An unmapped project remains inaccessible to members and causes readiness to report mapping_required.

Project membership uses global_admin and project_member. Configure at least one trusted global administrator with QA_GLOBAL_ADMIN_SUBJECTS; administrators can provision additional project members through the protected membership API. The development actor is a fixed server-side identity, cannot be selected by request headers or query values, and cannot start in production.

Create a consistent active-database backup with:

python -m qa_mcp.operations backup

The operation uses SQLite's online backup API, verifies PRAGMA integrity_check, sets backup directory/file permissions to 0700/0600, and retains the newest QA_BACKUP_RETENTION files. To restore, stop QA-MCP, run:

python -m qa_mcp.operations restore /protected/path/to/qa_mcp-backup.sqlite3

The restore path verifies both the source backup and restored database. Restart QA-MCP only after the restore succeeds and readiness is healthy. Keep backup storage on protected single-host local storage, schedule backups through the deployment's existing operations tooling, and periodically perform a restore drill. Do not use a shared network filesystem as the live SQLite database.

Example local configuration

The following values are examples only:

application:
  name: qa-mcp
  environment: local

llm:
  provider: mock
  region: ""
  model_id: ""

features:
  requirement_analyzer: true
  testcase_generator: true
  testcase_reviewer: true
  jira_connector: false
  github_connector: false
  slack_connector: false
  automation_generator: false

automation_execution:
  base_url: ""
  timeout_seconds: 60
  workspace_root: ""
  keep_workspace: false

jira:
  url: ""
  email: ""
  api_token: ""

github:
  url: "https://api.github.com"
  token: ""
  owner: ""

slack:
  url: "https://slack.com/api"
  token: ""
  default_channel: ""

Example `.env` template

These are placeholders only. Never commit real credentials or secrets.

JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token

GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org

SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=

DEFAULT_TEST_ENV=qa
QA_BASE_URL=
STAGE_BASE_URL=
PROD_BASE_URL=
QA_AUTOMATION_TIMEOUT_SECONDS=60
QA_AUTOMATION_WORKSPACE_ROOT=
QA_AUTOMATION_KEEP_WORKSPACE=false

Secret-handling rules

Never commit real credentials, access tokens, API keys, passwords, or production secrets.
Use environment variables or a local ignored .env file for secrets.
Never paste populated secret values into this README.
Verify that .env is protected by .gitignore.
Review git diff before committing configuration changes.
Run git diff --check.
Run focused tests.
Run the full regression suite.
Update this README with verified results before committing.

18. DEVELOPMENT ENVIRONMENT

Python requirement:

Python >= 3.11

Current development environment used during the latest verification:

Python 3.12 virtual environment
.venv/

Activate:

source .venv/bin/activate

Install project dependencies according to the repository's requirements.txt.

Run all tests:

pytest -q

Run a specific test:

pytest -q tests/<test_file>.py

Check MCP tools:

python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools()])"

Check automation tools:

python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools() if 'automation' in t.name])"

Check formatting issues:

git diff --check

Check repository state:

git status

Review recent commits:

git log -5 --oneline

19. GIT CHECKPOINT HISTORY

Important checkpoints:

a288569 Initial commit with configured gitignore
1144ddd Resolve README.md merge conflict
882e149 Jira Connector Added
06dbe61 Complete GitHub connector
71c893e Initial commit with Slack Configured
169c1a1 Complete automation case generator
1d2360b Add automation candidate pipeline
a226e9e Add automation case validation
5ced4e3 Add automation code generation
715ad52 Complete P2-S8.8 automation code generation
703cdb1 Complete automation execution foundation
3bdf761 Implement controlled automation execution
add5ba2 Update project continuity roadmap

Every completed checkpoint must contain:

Implementation
Tests
README
Verification evidence
Commit
Push
Clean working tree

20. WHAT HAS ALREADY BEEN COMPLETED — DO NOT REBUILD

These capabilities are already implemented/tested and must not be redesigned or recreated as if they were new:

MCP server foundation
Configuration
LLM abstraction
Mock LLM
Requirement analysis
Test case generation
Test case review
QA suite workflow
Project context
SQLite persistence
Requirement/suite versioning
Import/export
Jira connector
GitHub connector
Slack connector
Automation case generation
Automation candidate selection
Candidate → automation orchestration
Automation case validation
Playwright/Python automation code generation
GeneratedAutomationArtifact contract
AutomationExecutionResult contract
AutomationWorkspace
AutomationExecutionRunner
AutomationExecutionService
execute_automation_code MCP boundary

Future work must build on these components.

21. HISTORICAL EXECUTION-HARDENING ROADMAP

This section is retained for historical continuity.

It is not the current development checkpoint.

The current development checkpoint is defined at the beginning of this README. This historical roadmap has been superseded by the completed P2-S9.12 and P2-S9.14 checkpoints and the current P2-S9.15 implementation.

The earlier P2-S9.1 execution-hardening roadmap has been superseded by the completed implementation and later P2-S9.12 work.

Historical direction included:

Safe workspace and file handling
Stronger command validation
Execution identity
Configurable limits
Failure classification
Artifact and result retention
Execution evidence
Execution history
Reporting and analysis
Failure analysis
Web dashboard integration
Agent orchestration

The completed historical checkpoints below must be preserved as historical records only.

Future work must follow the current checkpoint at the top of this README and must not return to already-completed functionality without a new, explicitly documented requirement.

21A. COMPLETED CHECKPOINT — P2-S9.1.a

Execution Hardening — Safe Workspace/File Handling

STATUS: COMPLETE

P2-S9.1.a hardens the generated automation workspace boundary without changing the established execution contracts or MCP execution flow.

Implemented:

Strict generated artifact filename validation.
Rejection of empty and whitespace-only filenames.
Rejection of . and ...
Rejection of POSIX absolute paths and traversal paths.
Rejection of Windows-style traversal and drive-style paths.
Explicit resolved-path containment verification before writing.
Filename validation before workspace creation.
Generated artifacts remain constrained to the controlled workspace.

Tests:

14 new workspace-hardening tests
Focused execution suite: 28 passed
Full regression: 204 passed, 0 failures, 7 warnings
git diff --check: clean

The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15 and are intentionally outside this checkpoint.

No existing Pydantic execution contracts were changed.

Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening

21B. COMPLETED CHECKPOINT — P2-S9.1.b.1

Execution Hardening — Controlled Automation Command Boundary

STATUS: COMPLETE

Implemented:

Explicit controlled command construction in AutomationExecutionService.
Playwright automation is restricted to python -m pytest <artifact_file_name>.
Unsupported automation frameworks are rejected before command construction.
Unsafe artifact filenames are rejected before command construction.
AutomationExecutionRunner remains a generic subprocess execution wrapper.
Existing Pydantic execution contracts and the MCP execution boundary remain unchanged.

Tests:

Focused execution service suite: 10 passed
Full regression: 207 passed, 0 failures, 7 warnings
git diff --check: clean

The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15.

Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening

22. FUTURE EXECUTION ARCHITECTURE

Target:

GeneratedAutomationArtifact
        |
        v
Execution Policy / Safety Validation
        |
        v
Isolated Execution Environment
        |
        v
Framework Runner
        |
        v
Execution Evidence
        |
        v
Execution Result
        |
        v
Persistent Execution History
        |
        v
Reporting / AI Analysis

Potential isolation progression:

Local hardened process
        |
        v
Docker/container
        |
        v
CI worker
        |
        v
Cloud execution

Do not implement all layers at once.

23. EVENTUAL AGENT-DRIVEN QA WORKFLOW

The eventual product experience:

Understanding requirement...
        |
        v
Analyzing scenarios...
        |
        v
Generating test cases...
        |
        v
Reviewing coverage...
        |
        v
Identifying automation candidates...
        |
        v
Generating automation...
        |
        v
Validating automation...
        |
        v
Executing automation...
        |
        v
Analyzing results...
        |
        v
Preparing QA report...

MCP is intended to become the capability layer underneath an agent-driven QA product.

24. EVENTUAL PRODUCT / UI DIRECTION

The eventual UI should make the agent's progress, generated artifacts, execution state, and results visible and understandable.

User
  |
  v
QA MCP UI
  |
  v
Agent / MCP Orchestration
  |
  +-- Requirement analysis
  +-- Test generation
  +-- Test review
  +-- Candidate selection
  +-- Automation generation
  +-- Automation validation
  +-- Automation execution
  +-- Results / reporting
  +-- Jira
  +-- GitHub
  +-- Slack

The UI and hosted product layer should be introduced only after the core QA-agent capabilities are sufficiently stable.

25. LONG-TERM PRODUCT DIRECTION

The final product should evolve toward:

Understand
    |
    v
Plan
    |
    v
Generate
    |
    v
Validate
    |
    v
Execute
    |
    v
Observe
    |
    v
Analyze
    |
    v
Report
    |
    v
Learn / Improve

Long-term capabilities:

Requirements intelligence
Test design
Test review
Automation selection
Automation generation
Automation validation
Safe execution
Execution evidence
Failure analysis
Coverage analysis
Regression intelligence
External engineering-system context
CI/CD integration
Agent orchestration
Interactive UI
Hosted/cloud execution

These are future goals, not permission to prematurely implement everything.

26. DEVELOPMENT PRINCIPLES

The following principles must remain unchanged:

Build incrementally.
Write tests before implementation where practical.
Keep services small and composable.
Keep MCP tools thin.
Keep external integrations behind infrastructure abstractions.
Avoid destabilizing existing workflows.
Preserve structured Pydantic contracts.
Keep secrets outside source control.
Run full regression before every feature checkpoint.
Update this README whenever a meaningful feature checkpoint is committed.
Commit code, tests and README together for each completed checkpoint.
Prefer explicit contracts over implicit behavior.
Prefer deterministic behavior over clever behavior.
Keep execution safety ahead of execution convenience.
Keep production concerns separated from prototype convenience.
Do not duplicate completed capabilities.
Do not silently change established contracts.
Maintain requirement → test case → automation case → artifact → execution result traceability.
Treat deployment/configuration documentation as part of the implementation.
Treat this README as the continuity record, not optional documentation.

27. CURRENT RESUME POINT

Resume from

P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation

P2-S9.14-A — QA-MCP Production UI Architecture Shell, P2-S9.14-B — Project Execution Insights, and P2-S9.14-C — Generated Automation Artifact Review are complete and must not be recreated. P2-S9.14-D — Persisted Test Case Review is complete, committed as 207e31e, and pushed to origin/main. P2-S9.15 corrects persisted automation artifact identity while preserving legacy records and existing execution behavior. P2-S9.15 is complete, committed as ba9e27c, and pushed to origin/main.

Previous completed checkpoints:

P2-S8.6   Automation Candidate Selection          COMPLETE
P2-S8.7   Candidate → Automation Generation       COMPLETE
P2-S8.8   Automation Case Validation              COMPLETE
P2-S8.8+  Automation Code Generation              COMPLETE
P2-S8.9   Controlled Automation Execution         COMPLETE

P2-S9.14-B verified baseline:

Focused web tests: 30 passed
Full regression: 327 passed
8 warnings
0 failures

P2-S9.14-B checkpoint commit: 7439119. P2-S9.14-C checkpoint commit: f5dbc8d — Complete P2-S9.14-C generated artifact review. P2-S9.14-D checkpoint commit: 207e31e — Complete P2-S9.14-D persisted test case review. It was pushed to origin/main with a clean working tree. Its validation baseline was 327 passed, 8 warnings, and 0 failures.

P2-S9.15 validation baseline: 329 passed, 8 warnings, and 0 failures. P2-S9.16 — Project-Centric QA Workspace UX is complete, committed as 6b8c532, and pushed to origin/main. P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation is complete, committed as 5759211 (implementation) and f94f05c (corrective commit for local development operator access), and pushed to origin/main. P2-S9.17 validation baseline: 349 passed (full regression), 8 warnings, and 0 failures. Model B remains the approved product direction; Model C and Model D remain out of scope.

Current automation MCP surface:

generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code

28. CRITICAL CONTINUITY INSTRUCTION FOR A NEW CHAT

A future development session must:

Read this README first.
Inspect the current GitHub main branch: https://github.com/sanumenon/qa-mcp/tree/main
Confirm the latest commit and test baseline.
Inspect the existing implementation before proposing changes.
Treat P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation as the latest implementation checkpoint. P2-S9.17 is committed as 5759211 (implementation) and f94f05c (corrective commit for local development operator access) and pushed to origin/main. P2-S9.16 remains committed as 6b8c532; P2-S9.15 remains committed as ba9e27c. Model B is the approved deployment direction; Model C and Model D remain out of scope.
Treat P2-S9.1.a — Safe Workspace/File Handling as complete.
Treat P2-S9.1.b.1 — Controlled Automation Command Boundary as complete.
Do not recreate candidate selection.
Do not recreate automation generation.
Do not recreate automation validation.
Do not recreate controlled local execution.
Do not silently replace established architecture/contracts.
Add tests first wherever practical.
Keep the architecture layered.
Verify focused tests.
Verify the full regression suite.
Verify the MCP/runtime path for major capabilities.
Update this README at the end of every verified checkpoint.
Include deployment/configuration changes in this README.
Never commit real secrets or a populated .env.
Commit and push code + tests + README together.
Verify the working tree is clean after the checkpoint.
Never make the user repeat already-completed development work when the repository and README contain it.
Never use a new chat as a reason to restart the project from an earlier phase.

This README is part of the implementation and must be treated as the project's authoritative continuity record.

P2-S9.2 — Executable Playwright Code Generation

Status: COMPLETE

Implementation commit:

176ab59 Implement executable Playwright code generation

Implemented in:

src/qa_mcp/core/automation/code_generation_service.py

Generated Playwright/Python artifacts now translate the controlled automation DSL into executable Playwright code.

Supported automation steps:

goto: <url>
fill: <selector> = <value>
click: <selector>
press: <selector> = <key>

Supported assertions:

visible: <selector>
text: <selector> = <expected text>
url: <expected url>

The generator now:

Produces executable Playwright/Python code instead of comments.
Generates Page and expect based Playwright code.
Rejects unsupported automation steps.
Rejects unsupported automation assertions.
Validates malformed step/assertion expressions.
Preserves the existing GeneratedAutomationArtifact contract.
Keeps the existing execute_automation_code MCP boundary unchanged.
Supports generated-artifact → execution-service integration.

Verification:

pytest -q tests/test_automation_code_generation_service.py tests/test_automation_code_generation_empty.py tests/test_automation_code_generation_result.py

8 passed
0 failures

pytest -q tests/test_automation_execution_service.py

17 passed
0 failures

pytest -q

216 passed
7 warnings
0 failures

git diff --check
clean

The 7 pytest/Pydantic warnings are existing non-blocking technical debt and are not part of P2-S9.2.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result

Next implementation checkpoint

P2-S9.3 — Real Playwright Execution Validation

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, or controlled execution functionality.

P2-S9.3 — Execution History and Persistence

Status: COMPLETE

Implementation commit:

7b8a67b Add automation execution history

Verification:

Focused execution/history suite: 26 passed
Full regression suite: 227 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

Unique execution IDs are generated for every automation execution.
Automation execution results are persisted in SQLite.
Execution history can be retrieved by execution ID.
Execution history can be listed with optional automation-case filtering and result limits.
execute_automation_code now persists execution results.
Added MCP tool: get_automation_execution.
Added MCP tool: list_automation_executions.
Added repository, application-service, execution-service, and MCP-tool test coverage.
Existing execution behavior and contracts remain intact.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Reporting / Analysis

Next implementation checkpoint

P2-S9.4 — Reporting / Analysis

P2-S9.4 — Reporting / Analysis

Status: COMPLETE

Implementation commit: 55ddd09 Add automation execution reporting

Verification:

Focused P2-S9.4 tests: 12 passed
Full regression suite: 234 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

Added aggregated automation execution reporting.
Added execution totals by result status.
Added pass-rate calculation.
Added total and average execution duration metrics.
Added latest execution identification and status.
Added optional automation-case filtering.
Added persistent SQLite-backed reporting.
Added reporting application-service support.
Added MCP reporting capability.
Added repository, service, and MCP-tool test coverage.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis

Next implementation checkpoint

P2-S9.5 — Next functional capability

P2-S9.5 — Failure Analysis

Status: COMPLETE

Implementation commit: 77a8d9b Add automation execution failure analysis

Verification:

Focused P2-S9.5 tests: 15 passed
Full regression suite: 242 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

Added structured automation execution failure analysis.
Added failure-analysis models and application-service support.
Added persisted failure-analysis retrieval from execution history.
Added analysis of failed and errored executions.
Added execution identifiers, automation artifact identifiers, and automation case traceability.
Added MCP failure-analysis capability.
Added repository, service, and MCP-tool test coverage.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis

Next implementation checkpoint

P2-S9.6 — Next functional capability

P2-S9.6 — Web Dashboard

Status: COMPLETE

Implementation commit: 79e3c3f Add web dashboard

Verification:

Focused Web Dashboard tests: 5 passed
Full regression suite: 247 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

Added FastAPI-based web application.
Added browser-accessible QA automation dashboard.
Added execution reporting view.
Added execution history view.
Added failure-analysis view.
Added REST endpoints for execution reporting, execution history, and failure analysis.
Added dedicated web entrypoint through run_web.py.
Added web dashboard automated test coverage.
Added FastAPI to requirements.txt.

Start the Web Dashboard

From the project root with the virtual environment activated:

cd ~/pythonPrograms/qa-mcp
source .venv/bin/activate
python run_web.py

The dashboard is then available at:

http://127.0.0.1:8000

Web API endpoints

GET /
GET /api/executions/report
GET /api/executions?limit=20
GET /api/executions/failures?limit=20

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard

Next implementation checkpoint

P2-S9.7 — Next functional capability

P2-S9.7 — AI QA Workspace

Status: COMPLETE

Implementation scope:

Added a browser-based AI QA Workspace to the existing web dashboard.
Added persistent QA project creation and retrieval.
Added project-aware requirement analysis.
Added project-aware QA test-suite generation.
Added AI-generated test-case review.
Added requirement version persistence.
Added QA-suite version persistence.
Added workspace REST APIs.
Added deterministic MockLLM support for workspace development and tests.
Added automated coverage for project, workspace service, and MockLLM behavior.
Preserved all existing execution, reporting, failure-analysis, and dashboard functionality.

AI QA Workspace flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

Workspace API endpoints

POST /api/projects
GET  /api/projects/{project_id}
POST /api/projects/{project_id}/qa-suite

Workspace behavior

A QA project must exist before a QA suite can be generated for that project.

Existing projects can be retrieved using:

GET /api/projects/{project_id}

A requirement can then be submitted using:

POST /api/projects/{project_id}/qa-suite

The generated response contains:

project
requirement_version
suite_version
requirement
analysis
test_cases
review

Requirement versions and QA-suite versions are persisted independently so that generated QA work remains traceable to the project and requirement history.

Verification

Focused S9.7 workspace tests: 19 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite

P2-S9.8 — Automation Candidate Selection

Status: COMPLETE

Implementation scope:

Connected the AI QA Workspace to the existing Automation Candidate Service.
Added automation candidate selection to QA-suite generation.
Added automation candidate results to the generated QA-suite response.
Preserved manual-test identification through the existing candidate-selection workflow.
Added focused regression coverage for automation candidate selection.
Preserved all existing QA Workspace, execution, reporting, failure-analysis, and dashboard functionality.

Automation Candidate flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

QA-suite response

The generated QA-suite response now includes:

automation_candidates
    candidate_ids
    manual_ids
    total

Candidate selection is performed through the existing:

AutomationCandidateService
        ↓
AutomationCandidateSelector

Verification

Focused S9.8 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection

P2-S9.9 — AI QA Workspace Automation Generation

Status: COMPLETE

Implementation scope:

Connected the AI QA Workspace to the existing automation candidate generation service.
Added dependency injection for automation candidate generation.
Added automation case generation during QA-suite creation.
Added generated automation cases to the QA-suite response.
Preserved the existing automation candidate selection flow.
Added focused regression coverage for automation generation.
Preserved all existing QA Workspace, candidate selection, execution, reporting, failure-analysis, and dashboard functionality.

AI QA Workspace automation flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Case Generation
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

QA-suite automation response

The generated QA-suite response now includes:

automation_candidates
    candidate_ids
    manual_ids
    total

automation_cases
    test_case_id
    automation_type

Automation generation is performed through the existing service boundary:

AutomationCandidateGenerationService
        ↓
Generated Automation Cases

Verification

Focused S9.9 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection
        ↓
Automation Case Generation

P2-S9.10 — AI QA Workspace Automation Wiring

Status: COMPLETE

Implementation scope:

Connected the production web application to the existing automation-generation pipeline.
Added the workspace automation case generator using the workspace LLM.
Added the workspace automation service.
Added the existing automation candidate service and selector to the production workspace.
Added the automation candidate generation service to the AI QA Workspace.
Connected QAWorkspaceService to the production automation candidate generation service.
Added focused regression coverage confirming the production dependency is injected correctly.
Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.

Automation generation flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Candidate Generation
    ↓
Automation Case Generation
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

Production wiring

Workspace LLM
      ↓
AutomationCaseGenerator
      ↓
AutomationService
      ↓
AutomationCandidateGenerationService
      ↑
AutomationCandidateService
      ↑
AutomationCandidateSelector
      ↓
QAWorkspaceService
      ↓
AI QA Workspace API

The web application now constructs the real automation-generation dependencies instead of relying on test-only or implicit service construction.

Verification

Focused S9.10 workspace-service tests: 3 passed
Full regression suite: 262 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection
        ↓
Automation Candidate Generation
        ↓
Automation Case Generation

P2-S9.12 — Project Workspace Execution History and Result Review

Implemented:

Added project-scoped execution history listing.
Added project-scoped execution lookup.
Added project-scoped execution reporting.
Added project-scoped failure analysis.
Added artifact ownership filtering through the project artifact repository.
Added API coverage for project execution history and result review.

Verification:

Focused tests: 4 passed.
Full regression: 310 passed, 8 warnings.
git diff --check: passed.

Next implementation:

Audit remaining backend-to-UI integration.
Define and implement the QA Agent skills.md capability.

P2-S9.12 — Test Case Persistence and Automation Candidate Workflow

Status: IN PROGRESS

Current implementation state:

Persistent Project QA Workspace is implemented.
Persisted project requirements, requirement versions, saved QA suite versions, test cases, automation candidates, and generated automation artifacts are exposed through the workspace.
Automation candidates are now actionable from the Project QA Workspace.
Added QAProjectAutomationGenerationRequest for project-level automation generation requests.
Added QAWorkspaceService.generate_automation_from_project() to continue from persisted test cases.
Persisted test-case dictionaries are reconstructed through the existing TestCase model before candidate processing.
Requested test-case IDs are validated against the persisted project workspace.
Selected test cases are rechecked through the existing AutomationCandidateService.
Non-candidate selections are rejected instead of bypassing the existing candidate policy.
Existing AutomationCandidateGenerationService is reused to generate automation cases.
Existing AutomationCodeGenerationService is reused to generate automation artifacts.
Added POST /api/projects/{project_id}/automation.
Added Project QA Workspace UI controls for selecting automation candidates.
Added a Generate Automation action in the Project QA Workspace.
Browser-side UI wiring invokes the new project automation endpoint and refreshes the persisted workspace state.
Existing Generate QA Suite functionality remains separate and unchanged.
Added dashboard/API regression coverage for the new workflow.
Focused workspace/dashboard tests: 34 passed.
Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.
git diff --check is clean.

P2-S9.12 product flow:

Project -> Requirement -> Requirement Analysis -> Test Case Generation -> AI Review -> Preview -> User selects test cases -> Save -> QASuiteVersioningService -> SQLite qa_suite_versions -> Project QA Workspace -> Existing Requirements -> Requirement Versions -> Saved Suite Versions -> Existing Test Cases -> Automation Candidates -> Select Automation Candidates -> Generate Automation -> Automation Case Generation -> Automation Artifact Generation -> Controlled Automation Execution -> Execution History / Reporting -> Failure Analysis

Implemented user capability:

View requirements already provided or prepared for the project.
View requirement analysis and version information already prepared.
View test cases already generated and persisted for the project.
View automation candidates derived from persisted test cases.
Select one or more persisted automation candidates.
Generate automation from the selected candidates.
Generate automation artifacts through the existing automation code-generation pipeline.
Keep generated automation associated with the project workspace.
Preserve the existing active Generate QA Suite workflow independently.

Remaining P2-S9.12 scope:

Project QA Workspace -> View persisted test cases -> View automation candidates -> Select automation candidates -> Generate/use automation cases -> Validate automation cases -> Generate automation artifacts -> Controlled automation execution -> Execution result/history/reporting -> Failure analysis

Completed automation candidate selection, candidate generation, automation case generation, validation, Playwright code generation, artifact generation, command-boundary enforcement, execution configuration, controlled execution, execution history, reporting, and failure-analysis services must continue to be reused rather than rebuilt.

Checkpoint status at P2-S9.12:

P2-S9.12 — Controlled Automation Execution from Project QA Workspace

The Project QA Workspace now supports controlled execution of persisted automation artifacts. Execution is scoped to the selected project, reuses the existing controlled execution pipeline, and persists results through the existing execution-history service.

Implemented in this continuation:

Added project-scoped persisted automation-artifact lookup.
Added controlled execution through the existing AutomationExecutionService.
Added execution-result persistence through AutomationExecutionHistoryService.
Added POST /api/projects/{project_id}/automation/{artifact_id}/execute.
Added Execute actions for generated automation artifacts in the Project QA Workspace.
Added workspace feedback for execution status, stdout, stderr, and errors.
Preserved existing automation generation, validation, artifact generation, command-boundary enforcement, execution configuration, reporting, and failure-analysis functionality.

Validation completed:

Python compilation:                 passed
Full regression suite:              306 passed
Warnings:                            8 known non-blocking warnings
Failures:                            0
git diff --check:                    clean

The Project QA Workspace QA-suite generation flow has been hardened to require a complete test-case suite covering the supplied positive scenarios, negative scenarios, and edge cases. The generator now rejects malformed or incomplete LLM payloads instead of silently normalizing them into a single test case.

The LLM generation boundary now distinguishes unusable provider output from application validation failures. In particular, a provider/guardrail refusal is surfaced as LLMGenerationError and returned by the QA-suite API as HTTP 502 rather than being incorrectly classified as HTTP 404.

During validation, the configured Bedrock provider returned the following non-JSON response:

The response was blocked by dev-guardrails policy. If this looks like a false positive, ping #ai-guardrails.

This confirmed that the observed single-test-case UI symptom was not caused by application-side test-case truncation. The provider response was blocked before a test-case suite could be produced.

Validation completed:

Focused generator/dashboard tests:  30 passed
Full regression suite:              304 passed
Warnings:                            8 known non-blocking warnings
Failures:                            0
git diff --check:                    clean

The provider-failure diagnostic path now preserves the raw LLM provider response on LLMGenerationError.provider_response and logs that response at error level for IT troubleshooting. The API continues to return only the safe generic HTTP 502 message and does not expose the provider response to the end user.

The diagnostic regression test verifies that the exact provider response is retained. Focused diagnostic validation passed with 39 tests, and the full regression suite remains at 304 passed with 8 known non-blocking warnings and 0 failures.

The implementation is intentionally limited to prompt hardening, strict test-case response validation, LLM-generation error classification, API error mapping, and regression coverage. Do not bypass or weaken provider/dev-guardrail policy as part of this fix.

Next implementation:

P2-S9.12 continuation — Project Workspace Execution History and Result Review

The next implementation should expose persisted execution history and result review from the Project QA Workspace by reusing the existing execution-history, reporting, and failure-analysis services.

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, automation artifact generation, or the QA-suite generation/error-boundary hardening completed in this checkpoint.

P2-S9.11 — AI QA Workspace Artifact Generation

Status: COMPLETE

Implementation scope:

Connected the AI QA Workspace to the existing automation code-generation service.
Added production construction of AutomationCodeGenerationService.
Connected the service to QAWorkspaceService through dependency injection.
Added automation artifact generation for generated automation cases.
Added automation_artifacts to the QA-suite workspace response.
Added persistent SQLite repository support for QA Workspace automation artifacts.
Added focused regression coverage for artifact persistence, artifact generation, and dependency injection.
Added browser-level Playwright regression coverage for the AI QA Workspace dashboard.
Verified the rendered dashboard preserves the AI QA Workspace and existing Automation Execution Overview.
Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.

Automation artifact flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Candidate Generation
    ↓
Automation Case Generation
    ↓
Automation Code Generation
    ↓
Automation Artifact
    ↓
Persistent Artifact Repository
    ↓
QA Workspace Response

Workspace response

The generated QA-suite response now includes:

automation_candidates
automation_cases
automation_artifacts
requirement_version
suite_version

Production wiring

Workspace LLM
      ↓
AutomationCaseGenerator
      ↓
AutomationService
      ↓
AutomationCandidateGenerationService
      ↓
AutomationCase
      ↓
AutomationCodeGenerationService
      ↓
AutomationArtifact
      ↓
QAWorkspaceService
      ↓
AI QA Workspace API

Dashboard regression protection

The AI QA Workspace is protected by both structural and browser-level regression tests.

The dashboard regression coverage verifies:

AI QA Workspace
      ↓
Create QA Project controls
      ↓
Generate QA Suite controls
      ↓
Project / requirement inputs
      ↓
JavaScript action wiring
      ↓
Workspace API wiring
      ↓
Rendered dashboard
      ↓
Existing Automation Execution Overview preserved

The browser regression test uses Playwright with Chromium against a live Uvicorn instance and verifies the rendered dashboard and the important workspace controls.

The browser regression test is now part of the permanent dashboard regression suite and must remain green during future UI changes.

Verification

Focused dashboard/workspace tests: 12 passed
Browser regression test:           1 passed, 11 deselected
Full regression suite:             269 passed
Warnings:                           8 known non-blocking warnings
Failures:                           0
git diff --check:                   clean

P2-S9.11 is complete. Do not recreate or redesign this capability.

Next implementation checkpoint

P2-S9.12 — Test Case Persistence and Automation Candidate Workflow

P2-S9.12 has progressed from the persistent Project QA Workspace to actionable automation-candidate selection and generation. The workspace can now select persisted automation candidates and invoke the existing automation generation and artifact-generation pipeline. This implementation is committed as e186391. The remaining work is to continue from generated automation artifacts into the completed controlled execution, execution-history, reporting, and failure-analysis pipeline.

P2-S9.14-A — QA-MCP Production UI Architecture Shell

Status: COMPLETE

Implementation scope:

Introduced explicit workflow navigation across the existing QA-MCP web UI.
Added dedicated /execution workflow page.
Added dedicated /reports workflow page.
Added shared workflow navigation styling.
Connected the Execution page to the existing execution reporting and execution-history APIs.
Connected the Reports page to the existing execution reporting and failure-analysis APIs.
Added dedicated JavaScript assets for the new workflow pages.
Added regression coverage for the new page routes, navigation, and static assets.
Preserved the existing Dashboard implementation.
Preserved the existing Project QA Workspace implementation.
Preserved all existing backend API contracts.
No core business logic, persistence, execution, reporting, or failure-analysis services were redesigned.

Production UI workflow structure

QA MCP
  |
  +-- Dashboard
  |
  +-- Projects / QA Workspace
  |
  +-- Execution
  |      +-- Execution Metrics
  |      +-- Recent Executions
  |      +-- Execution History
  |
  +-- Reports & Analysis
         +-- Execution Reporting
         +-- Pass Rate
         +-- Average Duration
         +-- Failure Analysis

Backend capability reuse

Execution UI
    |
    +-- GET /api/executions/report
    +-- GET /api/executions?limit=20

Reports UI
    |
    +-- GET /api/executions/report
    +-- GET /api/executions/failures?limit=20

No duplicate execution or reporting business logic was introduced in the UI.

Dashboard
Projects / QA Workspace
Execution
Reports & Analysis

The active workflow is visually identified on each page.

Validation

Focused web tests:                 29 passed
Full regression suite:             326 passed
Warnings:                            8 known non-blocking warnings
Failures:                            0
git diff --check:                    clean
Browser/API verification:            passed

P2-S9.14-A is complete.

Do not redesign or rebuild the existing Dashboard, Project QA Workspace, execution services, reporting services, failure-analysis services, or established backend API contracts as part of the next increment.

Next implementation

P2-S9.14-B — Production UI workflow expansion

Before implementation:

Inspect the existing backend capabilities.
Identify stable APIs/services that already exist.
Define ONE next UI workflow boundary.
Do not invent backend capabilities merely to populate UI pages.
Do not redesign existing working workflows.
Obtain approval for the next sub-step before coding.

P2-S9.14-B — Project Execution Insights

Status: COMPLETE — committed as 7439119

Implementation scope

Added a compact Project Execution Insights section to the existing Project QA Workspace.
Displays project-scoped total, passed, failed, error, and pass-rate metrics.
Displays recent project-scoped failures with status, automation case, failure message, and a Review action.
Reuses the existing project execution detail flow for failure review.
Handles loading, no executions, no failures, unavailable failure details, and API errors explicitly.
Refreshes project insights after the existing project automation execution action completes.
Preserved existing workspace controls, Dashboard behavior, global Execution and Reports pages, backend services, and API contracts.

Reused project-scoped capabilities

GET /api/projects/{project_id}/executions/report
GET /api/projects/{project_id}/executions/failures?limit=10
GET /api/projects/{project_id}/executions/{execution_id}

No backend execution, reporting, failure-analysis, or persistence changes were required.

Validation

Focused web tests:                 30 passed
Full regression suite:             327 passed
Warnings:                            8
Failures:                            0
git diff --check:                    clean
Browser verification:               passed

The browser verification covered project selection, project-scoped metrics and failures, execution Review/detail, empty execution state, healthy no-failure state, API error state, and continued availability of existing workspace controls.

The P2-S9.14-B implementation is committed as 7439119 and was pushed to origin/main. At the P2-S9.14-C baseline, main was synchronized with origin/main and the working tree was clean.

Follow-on checkpoint (completed)

P2-S9.14-C — Generated Automation Artifact Review

Adds read-only inspection of generated automation artifact source in the existing Project QA Workspace, reusing artifact data already returned by the workspace API. Review and Execute remain separate actions. No backend/API, generation, persistence, or execution changes are in scope.

P2-S9.14-C — Generated Automation Artifact Review

Status: COMPLETE — committed as f5dbc8d

Implementation scope

Added a Review action for each generated automation artifact in the existing Project QA Workspace.
Added an inline read-only panel with filename, framework, language, creation time when available, project, associated test case, automation case, and generated source.
Rendered generated source through DOM text nodes so markup-like source stays inert; the source area is scrollable for long files.
Added close and artifact-switch behavior while leaving Execute as a separate action.
Reused artifact data from the existing project workspace response. No new endpoint or backend/API contract was added.
Preserved project scoping, artifact generation/persistence, execution semantics, Project Execution Insights, and other existing workflows.

Validation

Focused web tests:                 30 passed
Full regression suite:             327 passed
Warnings:                            8
Failures:                            0
JavaScript syntax check:             passed
git diff --check:                    clean
Browser verification:               passed

The browser verification covered artifact metadata and source display, inert markup-like source, switching artifacts, closing review, no execution request from Review, and the existing project-scoped Execute action working independently. The P2-S9.14-B execution insights flow remained covered by the same browser regression.

The P2-S9.14-C implementation is committed as f5dbc8d and is present on origin/main.

P2-S9.14-D — Persisted Test Case Review

Status: COMPLETE — committed as 207e31e

Implementation scope

Added a Review action to each persisted test case row in the existing Project QA Workspace.
Displays preconditions, ordered steps, expected result, priority, test type, and the exact suite ID/version and requirement-version metadata supplied by that workspace row.
Binds review to the selected row, so identical test case IDs in separate suite versions retain their row-specific metadata.
Uses DOM text APIs for testcase values; markup-like content remains inert.
Keeps Review separate from candidate selection, artifact review, and execution. No backend/API, persistence/schema, integration, or execution changes were made.

Validation

Focused web tests:                 30 passed
Full regression suite:             327 passed
Warnings:                            8
Failures:                            0
JavaScript syntax check:             passed
git diff --check:                    clean
Browser verification:               passed

Browser coverage verified case-specific details and version metadata, precondition and step ordering, inert markup-like values, switching and closing review, no mutation/generation/execution requests, candidate selection, artifact review, Execute, and empty testcase state. Existing P2-S9.14-B Project Execution Insights behavior remained covered.

P2-S9.14-D was committed as 207e31e, pushed to origin/main, and verified with a clean working tree. Its verified baseline was 327 passed, 8 warnings, and 0 failures.

P2-S9.15 — Persisted Automation Artifact Identity and Traceability

Status: COMPLETE — committed as ba9e27c and pushed to origin/main

Root cause

AutomationCodeGenerationService.generate() assigned every generated automation artifact the ID GA001. The SQLite artifact repository uses artifact_id as its primary key and saves with replace semantics, so distinct artifacts could overwrite one another. This broke artifact listing and could misassociate execution history and project reporting.

Implementation scope

Generate a UUID string for each new artifact inside AutomationCodeGenerationService.
Keep the artifact model, API response shape, MCP fields, repository schema, replace semantics, UI behavior, and execution semantics unchanged.
Preserve existing stored artifact IDs, including legacy GA001 records.
No migration or general-purpose ID framework was added.

Validation

Focused artifact/workspace/history/API tests: 30 passed, 1 warning
Focused Project Workspace browser test:        1 passed, 1 warning
Full regression suite:                         329 passed
Warnings:                                        8
Failures:                                        0
git diff --check:                                clean

Regression coverage verifies distinct IDs across repeated generation, multiple artifacts persisting within and across projects, legacy GA001 readability, exact artifact source selection during execution, execution history and project reporting, and browser Review/Execute selection for two artifacts.

P2-S9.15 is complete, committed as ba9e27c, and pushed to origin/main. P2-S9.16 — Project-Centric QA Workspace UX is complete, committed as 6b8c532, and pushed to origin/main.

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityActive

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

Qase Test ManagementOAuth
io.qase
Official MCP server for Qase — manage test cases, runs, suites, defects via AI tools.
Testing & QA Tools Developer Tools
237
WithNettleOAuth
com.withnettle
Project management shared by people and AI agents, with persistent project state through MCP.
151
SmartBear MCPOAuth
com.smartbear
MCP server for AI access to SmartBear tools, including BugSnag, Reflect, Swagger, PactFlow, QTM4J.
Monitoring Testing & QA Tools API Testing
144
Katalon True Platform MCPOAuth
io.github.katalon-studio
Katalon True Platform MCP: design tests, run with AI, upload reports, assess release readiness.
Testing & QA Tools Monitoring CI/CD & DevOps
22

Related MCP Servers

TestOps MCP Server
Testing & QA Tools Project Management
syn7xx
A
license
B
quality
C
maintenance
Enables AI assistants to interact with TestOps 5.25 for managing projects, test runs, test plans, and test cases via MCP tools.
Updated 4 months ago (2026-06-27 12:29 UTC)
31
28 npm
MIT
test-case-generator-ai-mcp
Testing & QA Tools Developer Tools
CSOAI-ORG
A
license
Not graded
quality
B
maintenance
Enables generation of test cases, edge cases, and test matrices for software testing, integrated with MCP protocol and EU AI Act compliance.
Updated 5 days ago (2026-10-08 04:26 UTC)
1 npm
25 PyPI
MIT
QA Copilot AI
Testing & QA Tools AI & Machine Learning
Sham-coder16
F
license
B
quality
B
maintenance
MCP server for AI-powered QA analysis. It enables analyzing test failures, identifying root causes, suggesting fixes, classifying defects, detecting flaky tests, and generating test cases and bug reports.
Updated 2 months ago (2026-07-31 05:26 UTC)
10
-
loopcheck
Testing & QA Tools Browser Automation
WeiR-h
A
license
Not graded
quality
A
maintenance
Enables AI builders to approve requirements, replay browser checks, and return regression evidence through MCP, supporting acceptance testing without code modification.
Updated a month ago (2026-09-12 10:45 UTC)
MIT

QA MCP

1. CURRENT DEVELOPMENT CHECKPOINT

Latest verified implementation notes

Bedrock test-case generation hardening

Bedrock runtime timeout resolution

QA suite completeness verification

Completed capabilities

Current implementation boundary

P2-S9.16 — Project-Centric QA Workspace UX

Validation

P2-S9.17 — Internal Identity, Project Authorization, and Execution Credential Isolation

Identity and project access

Execution, MCP, and operations

Configuration

Validation

Model B Readiness Assessment

2. PRODUCT VISION

3. DEVELOPMENT RULES — MUST FOLLOW

Mandatory development sequence

4. ARCHITECTURE

5. REPOSITORY STRUCTURE

6. COMPLETED PRODUCT CAPABILITIES

Phase 1 — Foundation & QA Intelligence

Phase 2 — QA Platform Foundation

7. PROJECT CONTEXT AND PERSISTENCE

8. EXTERNAL CONNECTORS

Jira

GitHub

Slack

9. AUTOMATION PIPELINE

10. AUTOMATION CHECKPOINTS ALREADY COMPLETE

P2-S8.6 — Candidate Selection

P2-S8.7 — Candidate → Automation Generation

P2-S8.8 — Automation Case Validation

P2-S8.8+ — Automation Code Generation

11. CONTROLLED AUTOMATION EXECUTION — P2-S8.9 COMPLETE

Execution configuration

Automation workspace

Controlled subprocess runner

Execution service

12. EXECUTION SAFETY REQUIREMENTS

13. MCP AUTOMATION SURFACE

14. TEST STRATEGY AND CURRENT BASELINE

15. KNOWN WARNINGS / TECHNICAL DEBT

Pytest collection warnings

Pydantic settings warning

16. ENVIRONMENT / .ENV DOCUMENTATION

Critical rule

Current .env template

Variable purpose

Deployment/configuration rule

17. CONFIGURATION AND ENVIRONMENT CONTRACT

Application configuration

Automation environment variables

Integration environment variables

Model B authentication and database operations

Example local configuration

Example .env template

Secret-handling rules

18. DEVELOPMENT ENVIRONMENT

19. GIT CHECKPOINT HISTORY

20. WHAT HAS ALREADY BEEN COMPLETED — DO NOT REBUILD

21. HISTORICAL EXECUTION-HARDENING ROADMAP

21A. COMPLETED CHECKPOINT — P2-S9.1.a

Execution Hardening — Safe Workspace/File Handling

21B. COMPLETED CHECKPOINT — P2-S9.1.b.1

Execution Hardening — Controlled Automation Command Boundary

22. FUTURE EXECUTION ARCHITECTURE

23. EVENTUAL AGENT-DRIVEN QA WORKFLOW

24. EVENTUAL PRODUCT / UI DIRECTION

25. LONG-TERM PRODUCT DIRECTION

26. DEVELOPMENT PRINCIPLES

27. CURRENT RESUME POINT

Resume from

28. CRITICAL CONTINUITY INSTRUCTION FOR A NEW CHAT

P2-S9.2 — Executable Playwright Code Generation

Current implementation state

Next implementation checkpoint

P2-S9.3 — Execution History and Persistence

Implementation delivered

Current `.env` template

Example `.env` template