QA MCP

QA MCP is a Model Context Protocol (MCP) server for structured software-quality workflows.

The project is being developed incrementally toward a full-fledged AI-powered QA platform using:

• Layered architecture

• Test-first development

• Pydantic-based contracts

• Persistent SQLite storage

• Immutable QA versioning

• Project import/export

• Safe external connectors

• MCP tool boundaries

• LLM-assisted QA analysis and automation generation

• Automation candidate selection

• Playwright automation generation

• Controlled automation execution

• Eventual QA-agent orchestration

• Eventual CI/CD and hosted product capabilities

CONTINUITY RULE: This README is the authoritative development, deployment, roadmap, and continuity checkpoint for future QA MCP development sessions. Read it before starting new development. Do not recreate completed work.

1. CURRENT DEVELOPMENT CHECKPOINT

Repository checkpoint

Repository:          https://github.com/sanumenon/qa-mcp/tree/main
Branch:              main
Latest commit:       a3e6f76 Update continuity for project automation workflow
Previous commit:     e186391 Implement project automation candidate workflow
Previous implementation checkpoint: P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
Remote:              origin/main
Working tree before checkpoint: clean
Current checkpoint:  P2-S9.12 — Automation Candidate Selection and Execution from Project QA Workspace
Next implementation: P2-S9.12 continuation — Controlled Automation Execution from Project QA Workspace
Checkpoint commit:   a3e6f76

Latest validation:
- 302 pytest tests passing.
- Project QA Workspace and dashboard focused tests: 34 passed.
- Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.
- `git diff --check` is clean.
- Project QA Workspace now provides actionable automation-candidate selection.
- Users can select persisted automation candidates and invoke automation generation from the Project QA Workspace.
- The browser UI is wired to `POST /api/projects/{project_id}/automation`.
- Persisted test cases are revalidated through the existing automation candidate-selection service before generation.
- The existing automation candidate generation, automation case generation, validation, Playwright code generation, artifact generation, controlled execution, execution history, reporting, and failure-analysis pipeline remains reused rather than duplicated.
- Existing Generate QA Suite functionality remains preserved.
- Today's implementation changes are committed in checkpoint `e186391`.

Next action:
- Complete P2-S9.12 by continuing from generated automation artifacts into controlled automation execution from the Project QA Workspace.
- Reuse the completed execution configuration, command boundary, execution runner, execution history, reporting, and failure-analysis services.
- Do not recreate completed capabilities or redesign the existing QA Suite generation workflow.

## Latest verified baseline

```text
pytest -q
298 passed
8 warnings
0 failures

P2-S9.11 Web Dashboard focused suite:
16 passed
0 failures

P2-S9.11 Browser regression test:
1 passed
0 failures

git diff --check
clean

git status
working tree clean

The warnings are known non-blocking technical debt and are documented below.

P2-S9.x Bedrock integration completion:

• Added Bedrock Converse API support to BedrockLLM.

• Preserved the existing replaceable LLMProvider abstraction.

• Added Converse-based response handling for supported Bedrock models.

• Verified organization-provided Bedrock bearer-key authentication through the application path.

• Verified us-east-1 with us.anthropic.claude-sonnet-4-5-20250929-v1:0.

• Verified real BedrockLLM.generate() execution successfully.

• Expanded LLM regression coverage.

• Full regression suite: 276 passed, 8 known warnings, 0 failures.

• .env remains untracked and secrets are not committed.

• moonshotai.kimi-k2.5 was tested separately but is not the validated application baseline; do not make it the default until its organization-approved application path is confirmed.

The full regression suite and the browser-level dashboard regression test have both been verified after the P2-S9.11 implementation.

P2-S9.11 UI correction:

• Corrected the QA Workspace button wiring so Create QA Project invokes project creation and Generate QA Suite invokes QA suite generation.

• Preserved the stable create-qa-project-button and generate-qa-suite-button identifiers.

• Verified the focused dashboard suite: 16 passed, 1 known warning.

• Verified the browser-level dashboard flow: 1 passed.

• Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.

• This correction is committed as 9dce673.

P2-S9.11 UI execution hardening:

• Hardened the Generate QA Suite button state so a generation request disables the button and displays Generating QA Suite....

• The button is restored to Generate QA Suite and re-enabled in a finally path after both successful and failed generation attempts.

• Generation errors are surfaced through the existing qa-workspace-error result area so the user receives a visible completion/error state instead of repeatedly submitting the request.

• Added stable create-qa-project-button and generate-qa-suite-button identifiers for browser-level regression coverage.

• Updated dashboard tests to verify the new UI wiring and generation-state behavior.

• Verified the focused web dashboard suite: 16 passed, 1 known warning.

• Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.

P2-S9.11 artifact generation completion:

• Added deterministic browser-level verification of the AI QA Workspace.

• Added UI-driven QA suite generation through the selected project.

• Added generated test-case selection with Select All / Clear All controls.

• Added selected test-case count tracking in the dashboard.

• Added selective QA suite persistence through POST /api/projects/{project_id}/qa-suite/save.

• Added validation that only generated test-case IDs can be persisted.

• Added suite version feedback after successful save.

• Added browser regression coverage for partial and full test-case selection/save flows.

• Verified the focused dashboard suite: 16 passed, 1 known warning, 0 failures.

• Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.

• git diff --check is clean.

• Commit: 24a341b Implement AI QA workspace artifact generation.

P2-S9.11 project workspace completion:

• Added persistent project listing through ProjectRepository and SQLiteProjectRepository.

• Added ProjectContext and QAWorkspaceService project-listing support.

• Added GET /api/projects for dashboard project discovery.

• Replaced the QA Suite Project ID text input with a project dropdown.

• Project selection now drives QA Suite generation using the selected project_id.

• Newly created projects are automatically refreshed into the dropdown and selected.

• Verified end-to-end project selection and successful AI QA test-case generation.

Related MCP server: TestOps MCP Server

Latest completed automation checkpoint

P2-S8.6   Automation Candidate Selection          COMPLETE
P2-S8.7   Candidate → Automation Generation       COMPLETE
P2-S8.8   Automation Case Validation              COMPLETE
P2-S8.8+  Automation Code Generation              COMPLETE
P2-S8.9   Controlled Automation Execution         COMPLETE

Do not rebuild or redesign these completed checkpoints.

2. PRODUCT VISION

The long-term goal is to evolve QA MCP from a collection of QA utilities into an intelligent QA agent/platform.

Requirement
    |
    v
Requirement Understanding
    |
    v
Scenario Analysis
    |
    v
Test Case Generation
    |
    v
Test Case Review
    |
    v
QA Suite / Version
    |
    v
Automation Candidate Selection
    |
    v
Automation Case
    |
    v
Automation Validation
    |
    v
Playwright Code Generation
    |
    v
Generated Automation Artifact
    |
    v
Controlled Execution
    |
    v
Execution Results
    |
    v
Reporting / Analysis
    |
    v
QA Agent / Orchestration

Eventually the platform should support:

Jira
GitHub
Slack
CI/CD
Test repositories
Automation environments
Cloud execution
Interactive UI
Hosted/cloud product

The UI/hosted product layer must be introduced only after the core QA-agent capabilities are sufficiently stable.

3. DEVELOPMENT RULES — MUST FOLLOW

These rules apply to every future change.

1. Implement one phase/sub-step at a time.

2. Test first wherever practical.

3. Focused tests must pass before moving to the next increment.

4. The relevant feature test group must pass.

5. The full regression suite must pass before closing a milestone.

6. Never weaken or delete tests merely to obtain green output.

7. Inspect existing code before modifying it.

8. Preserve the layered architecture.

9. Core business logic must remain independent of MCP transport.

10. Persistence must remain behind repository interfaces.

11. External integrations must remain mockable.

12. LLM providers must remain replaceable.

13. AI output must be validated before downstream use.

14. Never commit secrets or a real .env file.

15. Never delete persistent databases merely to make tests pass.

16. Keep unrelated refactoring separate from feature work.

17. A major capability is not complete until its MCP/runtime path is verified.

18. Update this README at every verified milestone.

19. Commit only after feature, tests, README, and checkpoint have been reviewed.

20. Do not recreate completed work from earlier milestones.

21. Do not introduce production-grade container/cloud complexity before the local execution contract is stable.

22. Keep generated automation execution behind explicit framework validation and controlled command construction.

23. Preserve traceability: Requirement → Test Case → Automation Case → Artifact → Execution Result.

24. Do not silently change established contracts.

25. Prefer deterministic behavior over clever behavior.

26. Keep execution safety ahead of execution convenience.

27. Deployment/configuration details must remain documented here.

28. A new chat/session must begin from this README and the current GitHub main branch.

Mandatory development sequence

Read README / current checkpoint
        |
        v
Inspect GitHub main + repository state
        |
        v
Inspect existing implementation
        |
        v
Define ONE next sub-step
        |
        v
Write/update focused tests
        |
        v
Implement smallest production change
        |
        v
Focused tests green
        |
        v
Feature tests green
        |
        v
Full regression green
        |
        v
Runtime/MCP verification
        |
        v
Update README
        |
        v
git diff --check
        |
        v
Commit + push
        |
        v
Verify clean working tree

4. ARCHITECTURE

                         MCP CLIENT / AI ASSISTANT
                                      |
                                      v
                                QA MCP Server
                                      |
                                      v
                                 MCP Tool Layer
                                      |
              +-----------------------+------------------------+
              |                       |                        |
              v                       v                        v
         QA Workflows          Core Services             Connectors
              |                       |               +--------+--------+
              |                       |               |        |        |
              v                       v               Jira    GitHub   Slack
     Requirement Analyzer       Automation             |        |        |
     Test Case Generator        Execution              v        v        v
     Test Case Reviewer         Versioning           Service  Service  Service
     QA Suite Workflow          Project Context        |        |        |
     Candidate Selection                             v        v        v
     Automation Generation                         Client   Client   Client
     Automation Execution                            / \      / \      / \
                                                     Mock     Mock    Mock
                                                     Cloud    Cloud   Cloud

Layer responsibilities:

models/
    Domain and data contracts

core/
    Business/application services
    Factories
    Orchestration boundaries
    Automation execution mechanics

infrastructure/
    Persistence
    External clients
    Concrete implementations

tools/
    QA-oriented application workflows

server.py
    MCP transport and tool registration

Core business logic must not become coupled to MCP transport.

5. REPOSITORY STRUCTURE

Important current structure:

qa-mcp/
|
+-- config/
|   +-- settings.yaml
|
+-- src/qa_mcp/
|   +-- core/
|   |   +-- automation/
|   |   |   +-- candidate_generation_service.py
|   |   |   +-- candidate_selector.py
|   |   |   +-- candidate_service.py
|   |   |   +-- code_generation_service.py
|   |   |   +-- execution_config.py
|   |   |   +-- execution_runner.py
|   |   |   +-- execution_service.py
|   |   |   +-- workspace.py
|   |   |   +-- service.py
|   |   |   +-- validator.py
|   |   |
|   |   +-- github/
|   |   +-- jira/
|   |   +-- slack/
|   |   +-- import_export/
|   |   +-- project/
|   |   +-- versioning/
|   |   +-- config.py
|   |   +-- llm.py
|   |
|   +-- infrastructure/
|   |   +-- github/
|   |   +-- jira/
|   |   +-- slack/
|   |   +-- versioning/
|   |   +-- project repositories
|   |
|   +-- models/
|   |   +-- schemas.py
|   |
|   +-- tools/
|   |   +-- automation/
|   |   +-- requirement/
|   |   +-- testcase/
|   |   +-- workflow/
|   |
|   +-- server.py
|
+-- tests/
+-- data/
+-- README.md
+-- requirements.txt

6. COMPLETED PRODUCT CAPABILITIES

Phase 1 — Foundation & QA Intelligence

STATUS: COMPLETE

Completed:

• MCP server foundation

• Configuration loading

• LLM abstraction

• Mock LLM support

• Requirement analysis

• Test-case generation

• Test-case review

• End-to-end QA suite workflow

Core MCP capabilities:

health
test_llm
analyze_requirement
generate_test_cases
review_test_cases
generate_qa_suite

Phase 2 — QA Platform Foundation

Slack integration exists behind service/client abstractions.

7. PROJECT CONTEXT AND PERSISTENCE

Conceptually:

QAProject
    |
    +-- project_id
    +-- name
    +-- description
    +-- application
    +-- environment
    +-- metadata
    +-- requirements
    +-- test suites

Persistence:

ProjectContext
      |
      v
ProjectRepository
      |
      v
SQLiteProjectRepository
      |
      v
SQLite

Database:

data/qa_mcp.db

Important: Never delete the persistent database merely to make tests pass.

Persistence-focused tests should use isolated database state.

8. EXTERNAL CONNECTORS

Jira

Abstraction:

MCP
 |
 v
JiraService
 |
 v
JiraClient
 +-- MockJiraClient
 +-- JiraCloudClient

Current real operations are read-only:

get_jira_issue(issue_key)
search_jira_issues(jql, max_results=50)

No Jira write operations are part of the completed connector milestone.

GitHub

Abstraction:

MCP
 |
 v
GitHubService
 |
 v
GitHubClient
 +-- MockGitHubClient
 +-- GitHubCloudClient

Current read-only tools:

get_github_repository(owner, repository)
get_github_issue(owner, repository, issue_number)
get_github_pull_request(owner, repository, pull_number)
search_github_issues(query, max_results=50)

No GitHub write operations are part of the completed connector milestone.

Slack

Abstraction:

SlackService
    |
    v
SlackClient
    +-- MockSlackClient
    +-- SlackCloudClient

Current tools include:

get_slack_channel
get_slack_messages
search_slack_messages
get_slack_thread

9. AUTOMATION PIPELINE

Test Cases
    |
    v
Automation Candidate Selection
    |
    v
Automation Case
    |
    v
Automation Validation
    |
    v
Framework-specific Code Generation
    |
    v
GeneratedAutomationArtifact
    |
    v
Controlled Execution
    |
    v
AutomationExecutionResult

Candidate selection deliberately distinguishes:

Recommended for automation
        |
        +---- Automated
        |
        +---- Manual-only

Manual-only test cases must not be sent to the automation generator.

10. AUTOMATION CHECKPOINTS ALREADY COMPLETE

P2-S8.6 — Candidate Selection

AutomationCandidateSelector / AutomationCandidateService

Result:

AutomationCandidateResult
    +-- candidate_ids
    +-- manual_ids
    +-- total

MCP tool:

select_automation_candidates

P2-S8.7 — Candidate → Automation Generation

Service:

AutomationCandidateGenerationService

Flow:

TestCase[]
    |
    v
Candidate Selection
    |
    v
candidate_ids
    |
    v
Generate automation ONLY for candidates
    |
    v
AutomationCase[]

Zero-candidate behavior:

No automation candidates
        |
        v
[]
        |
        v
Automation generator is NOT called

MCP tool:

generate_automation_for_candidates

P2-S8.8 — Automation Case Validation

Validator:

AutomationValidator

Result:

AutomationValidationResult
    +-- automation_case_id
    +-- test_case_id
    +-- valid
    +-- errors
    +-- warnings

Minimum integrity:

• At least one automation step.

• Validation failures are structured errors.

• Non-blocking concerns can be warnings.

• Validation remains separate from generation.

P2-S8.8+ — Automation Code Generation

Artifact:

GeneratedAutomationArtifact
    +-- id
    +-- automation_case_id
    +-- framework
    +-- language
    +-- file_name
    +-- code

Current execution target:

Framework: Playwright
Language: Python

Generated automation must be validated before downstream execution.

11. CONTROLLED AUTOMATION EXECUTION — P2-S8.9 COMPLETE

The committed local execution pipeline is:

GeneratedAutomationArtifact
        |
        v
AutomationExecutionConfig
        |
        v
AutomationWorkspace
        |
        v
AutomationExecutionRunner
        |
        v
AutomationExecutionService
        |
        v
AutomationExecutionResult
        |
        v
MCP execute_automation_code

Execution configuration

AutomationExecutionConfig
    +-- timeout_seconds = 60
    +-- workspace_root = optional

The configuration is immutable.

Automation workspace

AutomationWorkspace creates an isolated temporary directory for the generated artifact.

The workspace is cleaned up after execution unless explicit retention is requested.

The project working tree must not be used as the normal generated-artifact execution directory.

Controlled subprocess runner

AutomationExecutionRunner:

• accepts an explicit command list

• runs from a supplied working directory

• captures stdout

• captures stderr

• captures exit code

• measures execution duration

• enforces a timeout

• reports timeout separately

• reports operating-system execution errors separately

The runner is injectable so tests do not need to execute real automation processes.

Execution service

Current validation:

Empty code
    -> ValueError

Missing framework
    -> ValueError

Unsupported framework
    -> ValueError

Current supported framework:

Playwright

Current Python execution command:

python -m pytest <generated_file_name>

Status mapping:

exit_code == 0
    -> PASSED

exit_code != 0
    -> FAILED

timed_out
    -> TIMEOUT

runner error
    -> ERROR

Separation:

Runner
    = process mechanics

ExecutionService
    = QA execution semantics

AutomationExecutionResult
    = stable domain contract

Execution IDs are currently deterministic in the service foundation (EX001). Durable unique execution IDs belong to the future execution-history/persistence layer.

12. EXECUTION SAFETY REQUIREMENTS

The current subprocess runner is a controlled local execution boundary, not the final production-grade sandbox.

Intended progression:

Current
Local controlled subprocess
        |
        v
Hardened execution boundary
        |
        v
Container / isolated execution
        |
        v
Cloud or CI execution

Mandatory safety direction:

• Do not introduce arbitrary shell execution.

• Do not construct unrestricted commands from user input.

• Keep framework support explicit.

• Keep generated filenames and execution paths controlled.

• Keep execution bounded by timeouts.

• Preserve workspace isolation.

• Keep the runner injectable and testable.

• Introduce containerization before exposing execution to untrusted production workloads.

Do not add container/cloud complexity before the local execution contract and orchestration behavior are stable.

13. MCP AUTOMATION SURFACE

Current automation-related MCP tools:

generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code

execute_automation_code(artifact):

1. Validates the incoming artifact through GeneratedAutomationArtifact.

2. Delegates to AutomationExecutionService.

3. Returns AutomationExecutionResult.model_dump().

4. Converts invalid execution-artifact input into a controlled MCP-facing error.

The MCP layer must not contain subprocess implementation details.

14. TEST STRATEGY AND CURRENT BASELINE

Test-first development remains mandatory.

Expected sequence:

Write failing test
        |
        v
Implement smallest production change
        |
        v
Focused test
        |
        v
Related tests
        |
        v
Full regression
        |
        v
Runtime/MCP verification
        |
        v
README update
        |
        v
git diff --check
        |
        v
Commit + push

Current verified baseline before P2-S9.1.a:

pytest -q
190 passed
7 warnings
0 failures

P2-S9.1.a verified regression:

pytest -q
204 passed
7 warnings
0 failures

P2-S9.1.a focused execution suite:

16 passed
1 warning

Execution service suite:

7 passed

No test was removed or weakened to obtain the current green baseline.

15. KNOWN WARNINGS / TECHNICAL DEBT

Pytest collection warnings

Pydantic models named:

TestCase
TestCaseReview

can be interpreted by pytest as possible test classes, producing PytestCollectionWarning.

These are non-functional warnings.

Future cleanup may use test-only import aliases. Keep this separate from feature work.

Pydantic settings warning

Existing:

IncompleteFieldDefinitionWarning

related to the lifespan forward reference in pydantic_settings.

It does not currently cause test failures.

Keep this as separate technical debt unless it blocks development.

16. ENVIRONMENT / .ENV DOCUMENTATION

Critical rule

The following is the documented .env template currently used by the development setup.

These are placeholders, not real credentials.

Never commit a real .env file, API token, password, or secret to Git.

The actual local .env remains developer-machine configuration.

Current .env template

JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token
GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org
# ---------------------------------------------------------
# Slack
# ---------------------------------------------------------
SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=

Variable purpose

Deployment/configuration rule

When configuring a new environment:

1. Copy the documented template into a local .env.

2. Replace only the placeholder values required for that environment.

3. Never paste real secrets into this README.

4. Never commit the populated .env.

5. Verify .gitignore protects .env.

6. Keep configuration changes documented here when they materially affect deployment.

7. If new environment variables are introduced, update this section in the same development checkpoint.

17. CONFIGURATION

Primary configuration:

config/settings.yaml

Environment-specific secrets are supplied through environment variables.

Known integration variables:

JIRA_URL
JIRA_EMAIL
JIRA_API_TOKEN

GITHUB_URL
GITHUB_TOKEN
GITHUB_OWNER

SLACK_URL
SLACK_TOKEN
SLACK_DEFAULT_CHANNEL

Secrets must remain outside source control.

18. DEVELOPMENT ENVIRONMENT

Python requirement:

Python >= 3.11

Current development environment used during the latest verification:

Python 3.12 virtual environment
.venv/

Activate:

source .venv/bin/activate

Install project dependencies according to the repository's requirements.txt.

Run all tests:

pytest -q

Run a specific test:

pytest -q tests/<test_file>.py

Check MCP tools:

python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools()])"

Check automation tools:

python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools() if 'automation' in t.name])"

Check formatting issues:

git diff --check

Check repository state:

git status

Review recent commits:

git log -5 --oneline

19. GIT CHECKPOINT HISTORY

Important checkpoints:

a288569 Initial commit with configured gitignore
1144ddd Resolve README.md merge conflict
882e149 Jira Connector Added
06dbe61 Complete GitHub connector
71c893e Initial commit with Slack Configured
169c1a1 Complete automation case generator
1d2360b Add automation candidate pipeline
a226e9e Add automation case validation
5ced4e3 Add automation code generation
715ad52 Complete P2-S8.8 automation code generation
703cdb1 Complete automation execution foundation
3bdf761 Implement controlled automation execution
add5ba2 Update project continuity roadmap

Every completed checkpoint must contain:

Implementation
Tests
README
Verification evidence
Commit
Push
Clean working tree

20. WHAT HAS ALREADY BEEN COMPLETED — DO NOT REBUILD

These capabilities are already implemented/tested and must not be redesigned or recreated as if they were new:

MCP server foundation
Configuration
LLM abstraction
Mock LLM
Requirement analysis
Test case generation
Test case review
QA suite workflow
Project context
SQLite persistence
Requirement/suite versioning
Import/export
Jira connector
GitHub connector
Slack connector
Automation case generation
Automation candidate selection
Candidate → automation orchestration
Automation case validation
Playwright/Python automation code generation
GeneratedAutomationArtifact contract
AutomationExecutionResult contract
AutomationWorkspace
AutomationExecutionRunner
AutomationExecutionService
execute_automation_code MCP boundary

Future work must build on these components.

21. NEXT DEVELOPMENT CHECKPOINT

P2-S9.1 — Execution Hardening

STATUS: IN PROGRESS — P2-S9.1.b.1 COMPLETE

Completed sub-step:

P2-S9.1.b.1 — Controlled Automation Command Boundary

Next sub-step:

P2-S9.1.b.2 — Further Command/Execution Policy Hardening

Do not return to:

• candidate selection

• automation generation

• automation validation

• already-completed controlled local execution foundation

Immediate direction:

P2-S8.9 Controlled local execution
        |
        v
P2-S9.1 Execution hardening
        |
        +-- safe workspace/file handling
        +-- stronger command validation
        +-- execution identity
        +-- configurable limits
        +-- better failure classification
        +-- artifact/result retention policy
        |
        v
P2-S9.2 Execution evidence
        |
        +-- generated artifact metadata
        +-- stdout/stderr evidence
        +-- execution metadata
        +-- result persistence
        +-- traceability
        |
        v
P2-S9.3 Execution history
        |
        v
P2-S9.4 Reporting / analysis
        |
        v
P2-S10 Agent orchestration

P2-S9.1.a and P2-S9.1.b.1 are implemented and verified. The next sub-step must be defined and tested before implementation.

21A. COMPLETED CHECKPOINT — P2-S9.1.a

Execution Hardening — Safe Workspace/File Handling

STATUS: COMPLETE

P2-S9.1.a hardens the generated automation workspace boundary without changing the established execution contracts or MCP execution flow.

Implemented:

• Strict generated artifact filename validation.

• Rejection of empty and whitespace-only filenames.

• Rejection of . and ...

• Rejection of POSIX absolute paths and traversal paths.

• Rejection of Windows-style traversal and drive-style paths.

• Explicit resolved-path containment verification before writing.

• Filename validation before workspace creation.

• Generated artifacts remain constrained to the controlled workspace.

Tests:

14 new workspace-hardening tests
Focused execution suite: 28 passed
Full regression: 204 passed, 0 failures, 7 warnings
git diff --check: clean

The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15 and are intentionally outside this checkpoint.

No existing Pydantic execution contracts were changed.

Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening

21B. COMPLETED CHECKPOINT — P2-S9.1.b.1

Execution Hardening — Controlled Automation Command Boundary

STATUS: COMPLETE

Implemented:

• Explicit controlled command construction in AutomationExecutionService.

• Playwright automation is restricted to python -m pytest <artifact_file_name>.

• Unsupported automation frameworks are rejected before command construction.

• Unsafe artifact filenames are rejected before command construction.

• AutomationExecutionRunner remains a generic subprocess execution wrapper.

• Existing Pydantic execution contracts and the MCP execution boundary remain unchanged.

Tests:

Focused execution service suite: 10 passed
Full regression: 207 passed, 0 failures, 7 warnings
git diff --check: clean

The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15.

Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening

22. FUTURE EXECUTION ARCHITECTURE

Target:

GeneratedAutomationArtifact
        |
        v
Execution Policy / Safety Validation
        |
        v
Isolated Execution Environment
        |
        v
Framework Runner
        |
        v
Execution Evidence
        |
        v
Execution Result
        |
        v
Persistent Execution History
        |
        v
Reporting / AI Analysis

Potential isolation progression:

Local hardened process
        |
        v
Docker/container
        |
        v
CI worker
        |
        v
Cloud execution

Do not implement all layers at once.

23. EVENTUAL AGENT-DRIVEN QA WORKFLOW

The eventual product experience:

Understanding requirement...
        |
        v
Analyzing scenarios...
        |
        v
Generating test cases...
        |
        v
Reviewing coverage...
        |
        v
Identifying automation candidates...
        |
        v
Generating automation...
        |
        v
Validating automation...
        |
        v
Executing automation...
        |
        v
Analyzing results...
        |
        v
Preparing QA report...

MCP is intended to become the capability layer underneath an agent-driven QA product.

24. EVENTUAL PRODUCT / UI DIRECTION

The eventual UI should make the agent's progress, generated artifacts, execution state, and results visible and understandable.

User
  |
  v
QA MCP UI
  |
  v
Agent / MCP Orchestration
  |
  +-- Requirement analysis
  +-- Test generation
  +-- Test review
  +-- Candidate selection
  +-- Automation generation
  +-- Automation validation
  +-- Automation execution
  +-- Results / reporting
  +-- Jira
  +-- GitHub
  +-- Slack

The UI and hosted product layer should be introduced only after the core QA-agent capabilities are sufficiently stable.

25. LONG-TERM PRODUCT DIRECTION

The final product should evolve toward:

Understand
    |
    v
Plan
    |
    v
Generate
    |
    v
Validate
    |
    v
Execute
    |
    v
Observe
    |
    v
Analyze
    |
    v
Report
    |
    v
Learn / Improve

Long-term capabilities:

• Requirements intelligence

• Test design

• Test review

• Automation selection

• Automation generation

• Automation validation

• Safe execution

• Execution evidence

• Failure analysis

• Coverage analysis

• Regression intelligence

• External engineering-system context

• CI/CD integration

• Agent orchestration

• Interactive UI

• Hosted/cloud execution

These are future goals, not permission to prematurely implement everything.

26. DEVELOPMENT PRINCIPLES

The following principles must remain unchanged:

1. Build incrementally.

2. Write tests before implementation where practical.

3. Keep services small and composable.

4. Keep MCP tools thin.

5. Keep external integrations behind infrastructure abstractions.

6. Avoid destabilizing existing workflows.

7. Preserve structured Pydantic contracts.

8. Keep secrets outside source control.

9. Run full regression before every feature checkpoint.

10. Update this README whenever a meaningful feature checkpoint is committed.

11. Commit code, tests and README together for each completed checkpoint.

12. Prefer explicit contracts over implicit behavior.

13. Prefer deterministic behavior over clever behavior.

14. Keep execution safety ahead of execution convenience.

15. Keep production concerns separated from prototype convenience.

16. Do not duplicate completed capabilities.

17. Do not silently change established contracts.

18. Maintain requirement → test case → automation case → artifact → execution result traceability.

19. Treat deployment/configuration documentation as part of the implementation.

20. Treat this README as the continuity record, not optional documentation.

27. CURRENT RESUME POINT

Resume from

P2-S9.1.b.2 — Further Command/Execution Policy Hardening

P2-S9.1.a — Safe Workspace/File Handling and P2-S9.1.b.1 — Controlled Automation Command Boundary are complete and must not be recreated.

Previous completed checkpoints:

P2-S8.6   Automation Candidate Selection          COMPLETE
P2-S8.7   Candidate → Automation Generation       COMPLETE
P2-S8.8   Automation Case Validation              COMPLETE
P2-S8.8+  Automation Code Generation              COMPLETE
P2-S8.9   Controlled Automation Execution         COMPLETE

Verified baseline after P2-S9.1.b.1:

207 passed
7 warnings
0 failures

Latest repository implementation commit:

b6190d9 Harden automation workspace file handling

Current checkpoint commit:

Pending user commit

Previous implementation checkpoint:

3bdf761 Implement controlled automation execution

Current automation MCP surface:

generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code

28. CRITICAL CONTINUITY INSTRUCTION FOR A NEW CHAT

A future development session must:

1. Read this README first.

2. Inspect the current GitHub main branch: https://github.com/sanumenon/qa-mcp/tree/main

3. Confirm the latest commit and test baseline.

4. Inspect the existing implementation before proposing changes.

5. Start from P2-S9.1.b.2 — Further Command/Execution Policy Hardening.

6. Treat P2-S9.1.a — Safe Workspace/File Handling as complete.

7. Treat P2-S9.1.b.1 — Controlled Automation Command Boundary as complete.

8. Do not recreate candidate selection.

9. Do not recreate automation generation.

10. Do not recreate automation validation.

11. Do not recreate controlled local execution.

12. Do not silently replace established architecture/contracts.

13. Add tests first wherever practical.

14. Keep the architecture layered.

15. Verify focused tests.

16. Verify the full regression suite.

17. Verify the MCP/runtime path for major capabilities.

18. Update this README at the end of every verified checkpoint.

19. Include deployment/configuration changes in this README.

20. Never commit real secrets or a populated .env.

21. Commit and push code + tests + README together.

22. Verify the working tree is clean after the checkpoint.

23. Never make the user repeat already-completed development work when the repository and README contain it.

24. Never use a new chat as a reason to restart the project from an earlier phase.

This README is part of the implementation and must be treated as the project's authoritative continuity record.

P2-S9.2 — Executable Playwright Code Generation

Status: COMPLETE

Implementation commit:

176ab59 Implement executable Playwright code generation

Implemented in:

src/qa_mcp/core/automation/code_generation_service.py

Generated Playwright/Python artifacts now translate the controlled automation DSL into executable Playwright code.

Supported automation steps:

goto: <url>
fill: <selector> = <value>
click: <selector>
press: <selector> = <key>

Supported assertions:

visible: <selector>
text: <selector> = <expected text>
url: <expected url>

The generator now:

• Produces executable Playwright/Python code instead of comments.

• Generates Page and expect based Playwright code.

• Rejects unsupported automation steps.

• Rejects unsupported automation assertions.

• Validates malformed step/assertion expressions.

• Preserves the existing GeneratedAutomationArtifact contract.

• Keeps the existing execute_automation_code MCP boundary unchanged.

• Supports generated-artifact → execution-service integration.

Verification:

pytest -q tests/test_automation_code_generation_service.py tests/test_automation_code_generation_empty.py tests/test_automation_code_generation_result.py

8 passed
0 failures

pytest -q tests/test_automation_execution_service.py

17 passed
0 failures

pytest -q

216 passed
7 warnings
0 failures

git diff --check
clean

The 7 pytest/Pydantic warnings are existing non-blocking technical debt and are not part of P2-S9.2.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result

Next implementation checkpoint

P2-S9.3 — Real Playwright Execution Validation

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, or controlled execution functionality.

P2-S9.3 — Execution History and Persistence

Status: COMPLETE

Implementation commit:

7b8a67b Add automation execution history

Verification:

Focused execution/history suite: 26 passed
Full regression suite: 227 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

• Unique execution IDs are generated for every automation execution.

• Automation execution results are persisted in SQLite.

• Execution history can be retrieved by execution ID.

• Execution history can be listed with optional automation-case filtering and result limits.

• execute_automation_code now persists execution results.

• Added MCP tool: get_automation_execution.

• Added MCP tool: list_automation_executions.

• Added repository, application-service, execution-service, and MCP-tool test coverage.

• Existing execution behavior and contracts remain intact.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Reporting / Analysis

Next implementation checkpoint

P2-S9.4 — Reporting / Analysis

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, or execution-history functionality.

P2-S9.4 — Reporting / Analysis

Status: COMPLETE

Implementation commit: 55ddd09 Add automation execution reporting

Verification:

Focused P2-S9.4 tests: 12 passed
Full regression suite: 234 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

• Added aggregated automation execution reporting.

• Added execution totals by result status.

• Added pass-rate calculation.

• Added total and average execution duration metrics.

• Added latest execution identification and status.

• Added optional automation-case filtering.

• Added persistent SQLite-backed reporting.

• Added reporting application-service support.

• Added MCP reporting capability.

• Added repository, service, and MCP-tool test coverage.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis

Next implementation checkpoint

P2-S9.5 — Next functional capability

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, or reporting functionality.

P2-S9.5 — Failure Analysis

Status: COMPLETE

Implementation commit: 77a8d9b Add automation execution failure analysis

Verification:

Focused P2-S9.5 tests: 15 passed
Full regression suite: 242 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

• Added structured automation execution failure analysis.

• Added failure-analysis models and application-service support.

• Added persisted failure-analysis retrieval from execution history.

• Added analysis of failed and errored executions.

• Added execution identifiers, automation artifact identifiers, and automation case traceability.

• Added MCP failure-analysis capability.

• Added repository, service, and MCP-tool test coverage.

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis

Next implementation checkpoint

P2-S9.6 — Next functional capability

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, or failure-analysis functionality.

P2-S9.6 — Web Dashboard

Status: COMPLETE

Implementation commit: 79e3c3f Add web dashboard

Verification:

Focused Web Dashboard tests: 5 passed
Full regression suite: 247 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Implementation delivered

• Added FastAPI-based web application.

• Added browser-accessible QA automation dashboard.

• Added execution reporting view.

• Added execution history view.

• Added failure-analysis view.

• Added REST endpoints for execution reporting, execution history, and failure analysis.

• Added dedicated web entrypoint through run_web.py.

• Added web dashboard automated test coverage.

• Added FastAPI to requirements.txt.

Start the Web Dashboard

From the project root with the virtual environment activated:

cd ~/pythonPrograms/qa-mcp
source .venv/bin/activate
python run_web.py

The dashboard is then available at:

http://127.0.0.1:8000

Web API endpoints

GET /
GET /api/executions/report
GET /api/executions?limit=20
GET /api/executions/failures?limit=20

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard

Next implementation checkpoint

P2-S9.7 — Next functional capability

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, or web-dashboard functionality.

P2-S9.7 — AI QA Workspace

Status: COMPLETE

Implementation scope:

• Added a browser-based AI QA Workspace to the existing web dashboard.

• Added persistent QA project creation and retrieval.

• Added project-aware requirement analysis.

• Added project-aware QA test-suite generation.

• Added AI-generated test-case review.

• Added requirement version persistence.

• Added QA-suite version persistence.

• Added workspace REST APIs.

• Added deterministic MockLLM support for workspace development and tests.

• Added automated coverage for project, workspace service, and MockLLM behavior.

• Preserved all existing execution, reporting, failure-analysis, and dashboard functionality.

AI QA Workspace flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

Workspace API endpoints

POST /api/projects
GET  /api/projects/{project_id}
POST /api/projects/{project_id}/qa-suite

Workspace behavior

A QA project must exist before a QA suite can be generated for that project.

Existing projects can be retrieved using:

GET /api/projects/{project_id}

A requirement can then be submitted using:

POST /api/projects/{project_id}/qa-suite

The generated response contains:

project
requirement_version
suite_version
requirement
analysis
test_cases
review

Requirement versions and QA-suite versions are persisted independently so that generated QA work remains traceable to the project and requirement history.

Verification

Focused S9.7 workspace tests: 19 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite

P2-S9.8 — Automation Candidate Selection

Status: COMPLETE

Implementation scope:

• Connected the AI QA Workspace to the existing Automation Candidate Service.

• Added automation candidate selection to QA-suite generation.

• Added automation candidate results to the generated QA-suite response.

• Preserved manual-test identification through the existing candidate-selection workflow.

• Added focused regression coverage for automation candidate selection.

• Preserved all existing QA Workspace, execution, reporting, failure-analysis, and dashboard functionality.

Automation Candidate flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

QA-suite response

The generated QA-suite response now includes:

automation_candidates
    candidate_ids
    manual_ids
    total

Candidate selection is performed through the existing:

AutomationCandidateService
        ↓
AutomationCandidateSelector

Verification

Focused S9.8 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection

P2-S9.9 — AI QA Workspace Automation Generation

Status: COMPLETE

Implementation scope:

• Connected the AI QA Workspace to the existing automation candidate generation service.

• Added dependency injection for automation candidate generation.

• Added automation case generation during QA-suite creation.

• Added generated automation cases to the QA-suite response.

• Preserved the existing automation candidate selection flow.

• Added focused regression coverage for automation generation.

• Preserved all existing QA Workspace, candidate selection, execution, reporting, failure-analysis, and dashboard functionality.

AI QA Workspace automation flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Case Generation
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

QA-suite automation response

The generated QA-suite response now includes:

automation_candidates
    candidate_ids
    manual_ids
    total

automation_cases
    test_case_id
    automation_type

Automation generation is performed through the existing service boundary:

AutomationCandidateGenerationService
        ↓
Generated Automation Cases

Verification

Focused S9.9 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection
        ↓
Automation Case Generation

P2-S9.10 — AI QA Workspace Automation Wiring

Status: COMPLETE

Implementation scope:

• Connected the production web application to the existing automation-generation pipeline.

• Added the workspace automation case generator using the workspace LLM.

• Added the workspace automation service.

• Added the existing automation candidate service and selector to the production workspace.

• Added the automation candidate generation service to the AI QA Workspace.

• Connected QAWorkspaceService to the production automation candidate generation service.

• Added focused regression coverage confirming the production dependency is injected correctly.

• Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.

Automation generation flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Candidate Generation
    ↓
Automation Case Generation
    ↓
Requirement Version
    ↓
QA Suite Version
    ↓
Persisted QA Workspace Result

Production wiring

Workspace LLM
      ↓
AutomationCaseGenerator
      ↓
AutomationService
      ↓
AutomationCandidateGenerationService
      ↑
AutomationCandidateService
      ↑
AutomationCandidateSelector
      ↓
QAWorkspaceService
      ↓
AI QA Workspace API

The web application now constructs the real automation-generation dependencies instead of relying on test-only or implicit service construction.

Verification

Focused S9.10 workspace-service tests: 3 passed
Full regression suite: 262 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean

Current implementation state

Requirement/Test Case
        ↓
Automation Candidate
        ↓
Automation Case
        ↓
Generated Playwright/Python Artifact
        ↓
Controlled Automation Execution
        ↓
Execution Result
        ↓
Persistent Execution History
        ↓
Execution Reporting / Analysis
        ↓
Failure Analysis
        ↓
Web Dashboard
        ↓
AI QA Workspace
        ↓
Project-aware Requirement Analysis
        ↓
Versioned QA Suite
        ↓
Automation Candidate Selection
        ↓
Automation Candidate Generation
        ↓
Automation Case Generation

P2-S9.12 — Test Case Persistence and Automation Candidate Workflow

Status: IN PROGRESS

Current implementation state:

• Persistent Project QA Workspace is implemented.

• Persisted project requirements, requirement versions, saved QA suite versions, test cases, automation candidates, and generated automation artifacts are exposed through the workspace.

• Automation candidates are now actionable from the Project QA Workspace.

• Added QAProjectAutomationGenerationRequest for project-level automation generation requests.

• Added QAWorkspaceService.generate_automation_from_project() to continue from persisted test cases.

• Persisted test-case dictionaries are reconstructed through the existing TestCase model before candidate processing.

• Requested test-case IDs are validated against the persisted project workspace.

• Selected test cases are rechecked through the existing AutomationCandidateService.

• Non-candidate selections are rejected instead of bypassing the existing candidate policy.

• Existing AutomationCandidateGenerationService is reused to generate automation cases.

• Existing AutomationCodeGenerationService is reused to generate automation artifacts.

• Added POST /api/projects/{project_id}/automation.

• Added Project QA Workspace UI controls for selecting automation candidates.

• Added a Generate Automation action in the Project QA Workspace.

• Browser-side UI wiring invokes the new project automation endpoint and refreshes the persisted workspace state.

• Existing Generate QA Suite functionality remains separate and unchanged.

• Added dashboard/API regression coverage for the new workflow.

• Focused workspace/dashboard tests: 34 passed.

• Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.

• git diff --check is clean.

P2-S9.12 product flow:

Project -> Requirement -> Requirement Analysis -> Test Case Generation -> AI Review -> Preview -> User selects test cases -> Save -> QASuiteVersioningService -> SQLite qa_suite_versions -> Project QA Workspace -> Existing Requirements -> Requirement Versions -> Saved Suite Versions -> Existing Test Cases -> Automation Candidates -> Select Automation Candidates -> Generate Automation -> Automation Case Generation -> Automation Artifact Generation -> Controlled Automation Execution -> Execution History / Reporting -> Failure Analysis

Implemented user capability:

• View requirements already provided or prepared for the project.

• View requirement analysis and version information already prepared.

• View test cases already generated and persisted for the project.

• View automation candidates derived from persisted test cases.

• Select one or more persisted automation candidates.

• Generate automation from the selected candidates.

• Generate automation artifacts through the existing automation code-generation pipeline.

• Keep generated automation associated with the project workspace.

• Preserve the existing active Generate QA Suite workflow independently.

Remaining P2-S9.12 scope:

Project QA Workspace -> View persisted test cases -> View automation candidates -> Select automation candidates -> Generate/use automation cases -> Validate automation cases -> Generate automation artifacts -> Controlled automation execution -> Execution result/history/reporting -> Failure analysis

Completed automation candidate selection, candidate generation, automation case generation, validation, Playwright code generation, artifact generation, command-boundary enforcement, execution configuration, controlled execution, execution history, reporting, and failure-analysis services must continue to be reused rather than rebuilt.

Current checkpoint:

P2-S9.12 QA Suite Generation and LLM Error-Boundary Hardening

The Project QA Workspace QA-suite generation flow has been hardened to require a complete test-case suite covering the supplied positive scenarios, negative scenarios, and edge cases. The generator now rejects malformed or incomplete LLM payloads instead of silently normalizing them into a single test case.

The LLM generation boundary now distinguishes unusable provider output from application validation failures. In particular, a provider/guardrail refusal is surfaced as LLMGenerationError and returned by the QA-suite API as HTTP 502 rather than being incorrectly classified as HTTP 404.

During validation, the configured Bedrock provider returned the following non-JSON response:

The response was blocked by dev-guardrails policy. If this looks like a false positive, ping #ai-guardrails.

This confirmed that the observed single-test-case UI symptom was not caused by application-side test-case truncation. The provider response was blocked before a test-case suite could be produced.

Validation completed:

Focused generator/dashboard tests:  30 passed
Full regression suite:              304 passed
Warnings:                            8 known non-blocking warnings
Failures:                            0
git diff --check:                    clean

The provider-failure diagnostic path now preserves the raw LLM provider response on LLMGenerationError.provider_response and logs that response at error level for IT troubleshooting. The API continues to return only the safe generic HTTP 502 message and does not expose the provider response to the end user.

The diagnostic regression test verifies that the exact provider response is retained. Focused diagnostic validation passed with 39 tests, and the full regression suite remains at 304 passed with 8 known non-blocking warnings and 0 failures.

The implementation is intentionally limited to prompt hardening, strict test-case response validation, LLM-generation error classification, API error mapping, and regression coverage. Do not bypass or weaken provider/dev-guardrail policy as part of this fix.

Next implementation:

P2-S9.12 continuation — Controlled Automation Execution from Project QA Workspace

The next implementation should wire the already-generated project automation artifacts into the completed controlled execution pipeline, including execution configuration, command-boundary enforcement, execution history, reporting, and failure analysis.

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, automation artifact generation, or the QA-suite generation/error-boundary hardening completed in this checkpoint.

P2-S9.11 — AI QA Workspace Artifact Generation

Status: COMPLETE

Implementation scope:

• Connected the AI QA Workspace to the existing automation code-generation service.

• Added production construction of AutomationCodeGenerationService.

• Connected the service to QAWorkspaceService through dependency injection.

• Added automation artifact generation for generated automation cases.

• Added automation_artifacts to the QA-suite workspace response.

• Added persistent SQLite repository support for QA Workspace automation artifacts.

• Added focused regression coverage for artifact persistence, artifact generation, and dependency injection.

• Added browser-level Playwright regression coverage for the AI QA Workspace dashboard.

• Verified the rendered dashboard preserves the AI QA Workspace and existing Automation Execution Overview.

• Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.

Automation artifact flow

QA Project
    ↓
Requirement
    ↓
Requirement Analysis
    ↓
Test Case Generation
    ↓
AI Test Case Review
    ↓
Automation Candidate Selection
    ↓
Automation Candidate Generation
    ↓
Automation Case Generation
    ↓
Automation Code Generation
    ↓
Automation Artifact
    ↓
Persistent Artifact Repository
    ↓
QA Workspace Response

Workspace response

The generated QA-suite response now includes:

automation_candidates
automation_cases
automation_artifacts
requirement_version
suite_version

Production wiring

Workspace LLM
      ↓
AutomationCaseGenerator
      ↓
AutomationService
      ↓
AutomationCandidateGenerationService
      ↓
AutomationCase
      ↓
AutomationCodeGenerationService
      ↓
AutomationArtifact
      ↓
QAWorkspaceService
      ↓
AI QA Workspace API

Dashboard regression protection

The AI QA Workspace is protected by both structural and browser-level regression tests.

The dashboard regression coverage verifies:

AI QA Workspace
      ↓
Create QA Project controls
      ↓
Generate QA Suite controls
      ↓
Project / requirement inputs
      ↓
JavaScript action wiring
      ↓
Workspace API wiring
      ↓
Rendered dashboard
      ↓
Existing Automation Execution Overview preserved

The browser regression test uses Playwright with Chromium against a live Uvicorn instance and verifies the rendered dashboard and the important workspace controls.

The browser regression test is now part of the permanent dashboard regression suite and must remain green during future UI changes.

Verification

Focused dashboard/workspace tests: 12 passed
Browser regression test:           1 passed, 11 deselected
Full regression suite:             269 passed
Warnings:                           8 known non-blocking warnings
Failures:                           0
git diff --check:                   clean

P2-S9.11 is complete. Do not recreate or redesign this capability.

Next implementation checkpoint

P2-S9.12 — Test Case Persistence and Automation Candidate Workflow

P2-S9.12 has progressed from the persistent Project QA Workspace to actionable automation-candidate selection and generation. The workspace can now select persisted automation candidates and invoke the existing automation generation and artifact-generation pipeline. This implementation is committed as e186391. The remaining work is to continue from generated automation artifacts into the completed controlled execution, execution-history, reporting, and failure-analysis pipeline.

Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, or automation artifact generation.