QA MCP Server
by sanumenon
README.md
# QA MCP
QA MCP is a Model Context Protocol (MCP) server for structured software-quality workflows.
The project is being developed incrementally toward a full-fledged AI-powered QA platform using:
- Layered architecture
- Test-first development
- Pydantic-based contracts
- Persistent SQLite storage
- Immutable QA versioning
- Project import/export
- Safe external connectors
- MCP tool boundaries
- LLM-assisted QA analysis and automation generation
- Automation candidate selection
- Playwright automation generation
- Controlled automation execution
- Eventual QA-agent orchestration
- Eventual CI/CD and hosted product capabilities
> **CONTINUITY RULE:** This README is the authoritative development, deployment, roadmap, and continuity checkpoint for future QA MCP development sessions. Read it before starting new development. Do not recreate completed work.
---
# 1. CURRENT DEVELOPMENT CHECKPOINT
## Repository checkpoint
```text
Repository: https://github.com/sanumenon/qa-mcp/tree/main
Branch: main
Latest commit: a3e6f76 Update continuity for project automation workflow
Previous commit: e186391 Implement project automation candidate workflow
Previous implementation checkpoint: P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
Remote: origin/main
Working tree before checkpoint: clean
Current checkpoint: P2-S9.12 — Automation Candidate Selection and Execution from Project QA Workspace
Next implementation: P2-S9.12 continuation — Controlled Automation Execution from Project QA Workspace
Checkpoint commit: a3e6f76
Latest validation:
- 302 pytest tests passing.
- Project QA Workspace and dashboard focused tests: 34 passed.
- Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.
- `git diff --check` is clean.
- Project QA Workspace now provides actionable automation-candidate selection.
- Users can select persisted automation candidates and invoke automation generation from the Project QA Workspace.
- The browser UI is wired to `POST /api/projects/{project_id}/automation`.
- Persisted test cases are revalidated through the existing automation candidate-selection service before generation.
- The existing automation candidate generation, automation case generation, validation, Playwright code generation, artifact generation, controlled execution, execution history, reporting, and failure-analysis pipeline remains reused rather than duplicated.
- Existing Generate QA Suite functionality remains preserved.
- Today's implementation changes are committed in checkpoint `e186391`.
Next action:
- Complete P2-S9.12 by continuing from generated automation artifacts into controlled automation execution from the Project QA Workspace.
- Reuse the completed execution configuration, command boundary, execution runner, execution history, reporting, and failure-analysis services.
- Do not recreate completed capabilities or redesign the existing QA Suite generation workflow.
## Latest verified baseline
```text
pytest -q
298 passed
8 warnings
0 failures
P2-S9.11 Web Dashboard focused suite:
16 passed
0 failures
P2-S9.11 Browser regression test:
1 passed
0 failures
git diff --check
clean
git status
working tree clean
```
The warnings are known non-blocking technical debt and are documented below.
P2-S9.x Bedrock integration completion:
- Added Bedrock Converse API support to `BedrockLLM`.
- Preserved the existing replaceable `LLMProvider` abstraction.
- Added Converse-based response handling for supported Bedrock models.
- Verified organization-provided Bedrock bearer-key authentication through the application path.
- Verified `us-east-1` with `us.anthropic.claude-sonnet-4-5-20250929-v1:0`.
- Verified real `BedrockLLM.generate()` execution successfully.
- Expanded LLM regression coverage.
- Full regression suite: 276 passed, 8 known warnings, 0 failures.
- `.env` remains untracked and secrets are not committed.
- `moonshotai.kimi-k2.5` was tested separately but is not the validated application baseline; do not make it the default until its organization-approved application path is confirmed.
The full regression suite and the browser-level dashboard regression test have both been verified after the P2-S9.11 implementation.
P2-S9.11 UI correction:
- Corrected the QA Workspace button wiring so `Create QA Project` invokes project creation and `Generate QA Suite` invokes QA suite generation.
- Preserved the stable `create-qa-project-button` and `generate-qa-suite-button` identifiers.
- Verified the focused dashboard suite: 16 passed, 1 known warning.
- Verified the browser-level dashboard flow: 1 passed.
- Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.
- This correction is committed as `9dce673`.
P2-S9.11 UI execution hardening:
- Hardened the Generate QA Suite button state so a generation request disables the button and displays `Generating QA Suite...`.
- The button is restored to `Generate QA Suite` and re-enabled in a `finally` path after both successful and failed generation attempts.
- Generation errors are surfaced through the existing `qa-workspace-error` result area so the user receives a visible completion/error state instead of repeatedly submitting the request.
- Added stable `create-qa-project-button` and `generate-qa-suite-button` identifiers for browser-level regression coverage.
- Updated dashboard tests to verify the new UI wiring and generation-state behavior.
- Verified the focused web dashboard suite: 16 passed, 1 known warning.
- Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.
P2-S9.11 artifact generation completion:
- Added deterministic browser-level verification of the AI QA Workspace.
- Added UI-driven QA suite generation through the selected project.
- Added generated test-case selection with Select All / Clear All controls.
- Added selected test-case count tracking in the dashboard.
- Added selective QA suite persistence through POST /api/projects/{project_id}/qa-suite/save.
- Added validation that only generated test-case IDs can be persisted.
- Added suite version feedback after successful save.
- Added browser regression coverage for partial and full test-case selection/save flows.
- Verified the focused dashboard suite: 16 passed, 1 known warning, 0 failures.
- Verified the full regression suite: 281 passed, 8 known warnings, 0 failures.
- `git diff --check` is clean.
- Commit: `24a341b Implement AI QA workspace artifact generation`.
P2-S9.11 project workspace completion:
- Added persistent project listing through ProjectRepository and SQLiteProjectRepository.
- Added ProjectContext and QAWorkspaceService project-listing support.
- Added GET /api/projects for dashboard project discovery.
- Replaced the QA Suite Project ID text input with a project dropdown.
- Project selection now drives QA Suite generation using the selected project_id.
- Newly created projects are automatically refreshed into the dropdown and selected.
- Verified end-to-end project selection and successful AI QA test-case generation.
## Latest completed automation checkpoint
```text
P2-S8.6 Automation Candidate Selection COMPLETE
P2-S8.7 Candidate → Automation Generation COMPLETE
P2-S8.8 Automation Case Validation COMPLETE
P2-S8.8+ Automation Code Generation COMPLETE
P2-S8.9 Controlled Automation Execution COMPLETE
```
**Do not rebuild or redesign these completed checkpoints.**
---
# 2. PRODUCT VISION
The long-term goal is to evolve QA MCP from a collection of QA utilities into an intelligent QA agent/platform.
```text
Requirement
|
v
Requirement Understanding
|
v
Scenario Analysis
|
v
Test Case Generation
|
v
Test Case Review
|
v
QA Suite / Version
|
v
Automation Candidate Selection
|
v
Automation Case
|
v
Automation Validation
|
v
Playwright Code Generation
|
v
Generated Automation Artifact
|
v
Controlled Execution
|
v
Execution Results
|
v
Reporting / Analysis
|
v
QA Agent / Orchestration
```
Eventually the platform should support:
```text
Jira
GitHub
Slack
CI/CD
Test repositories
Automation environments
Cloud execution
Interactive UI
Hosted/cloud product
```
The UI/hosted product layer must be introduced only after the core QA-agent capabilities are sufficiently stable.
---
# 3. DEVELOPMENT RULES — MUST FOLLOW
These rules apply to every future change.
1. Implement one phase/sub-step at a time.
2. Test first wherever practical.
3. Focused tests must pass before moving to the next increment.
4. The relevant feature test group must pass.
5. The full regression suite must pass before closing a milestone.
6. Never weaken or delete tests merely to obtain green output.
7. Inspect existing code before modifying it.
8. Preserve the layered architecture.
9. Core business logic must remain independent of MCP transport.
10. Persistence must remain behind repository interfaces.
11. External integrations must remain mockable.
12. LLM providers must remain replaceable.
13. AI output must be validated before downstream use.
14. Never commit secrets or a real `.env` file.
15. Never delete persistent databases merely to make tests pass.
16. Keep unrelated refactoring separate from feature work.
17. A major capability is not complete until its MCP/runtime path is verified.
18. Update this README at every verified milestone.
19. Commit only after feature, tests, README, and checkpoint have been reviewed.
20. Do not recreate completed work from earlier milestones.
21. Do not introduce production-grade container/cloud complexity before the local execution contract is stable.
22. Keep generated automation execution behind explicit framework validation and controlled command construction.
23. Preserve traceability:
`Requirement → Test Case → Automation Case → Artifact → Execution Result`.
24. Do not silently change established contracts.
25. Prefer deterministic behavior over clever behavior.
26. Keep execution safety ahead of execution convenience.
27. Deployment/configuration details must remain documented here.
28. A new chat/session must begin from this README and the current GitHub `main` branch.
## Mandatory development sequence
```text
Read README / current checkpoint
|
v
Inspect GitHub main + repository state
|
v
Inspect existing implementation
|
v
Define ONE next sub-step
|
v
Write/update focused tests
|
v
Implement smallest production change
|
v
Focused tests green
|
v
Feature tests green
|
v
Full regression green
|
v
Runtime/MCP verification
|
v
Update README
|
v
git diff --check
|
v
Commit + push
|
v
Verify clean working tree
```
---
# 4. ARCHITECTURE
```text
MCP CLIENT / AI ASSISTANT
|
v
QA MCP Server
|
v
MCP Tool Layer
|
+-----------------------+------------------------+
| | |
v v v
QA Workflows Core Services Connectors
| | +--------+--------+
| | | | |
v v Jira GitHub Slack
Requirement Analyzer Automation | | |
Test Case Generator Execution v v v
Test Case Reviewer Versioning Service Service Service
QA Suite Workflow Project Context | | |
Candidate Selection v v v
Automation Generation Client Client Client
Automation Execution / \ / \ / \
Mock Mock Mock
Cloud Cloud Cloud
```
Layer responsibilities:
```text
models/
Domain and data contracts
core/
Business/application services
Factories
Orchestration boundaries
Automation execution mechanics
infrastructure/
Persistence
External clients
Concrete implementations
tools/
QA-oriented application workflows
server.py
MCP transport and tool registration
```
Core business logic must not become coupled to MCP transport.
---
# 5. REPOSITORY STRUCTURE
Important current structure:
```text
qa-mcp/
|
+-- config/
| +-- settings.yaml
|
+-- src/qa_mcp/
| +-- core/
| | +-- automation/
| | | +-- candidate_generation_service.py
| | | +-- candidate_selector.py
| | | +-- candidate_service.py
| | | +-- code_generation_service.py
| | | +-- execution_config.py
| | | +-- execution_runner.py
| | | +-- execution_service.py
| | | +-- workspace.py
| | | +-- service.py
| | | +-- validator.py
| | |
| | +-- github/
| | +-- jira/
| | +-- slack/
| | +-- import_export/
| | +-- project/
| | +-- versioning/
| | +-- config.py
| | +-- llm.py
| |
| +-- infrastructure/
| | +-- github/
| | +-- jira/
| | +-- slack/
| | +-- versioning/
| | +-- project repositories
| |
| +-- models/
| | +-- schemas.py
| |
| +-- tools/
| | +-- automation/
| | +-- requirement/
| | +-- testcase/
| | +-- workflow/
| |
| +-- server.py
|
+-- tests/
+-- data/
+-- README.md
+-- requirements.txt
```
---
# 6. COMPLETED PRODUCT CAPABILITIES
## Phase 1 — Foundation & QA Intelligence
**STATUS: COMPLETE**
Completed:
- MCP server foundation
- Configuration loading
- LLM abstraction
- Mock LLM support
- Requirement analysis
- Test-case generation
- Test-case review
- End-to-end QA suite workflow
Core MCP capabilities:
```text
health
test_llm
analyze_requirement
generate_test_cases
review_test_cases
generate_qa_suite
```
## Phase 2 — QA Platform Foundation
| Milestone | Capability | Status |
|---|---|---|
| P2-S1 | QA Project Context | COMPLETE |
| P2-S2 | SQLite Persistence | COMPLETE |
| P2-S3 | Requirement & Suite Versioning | COMPLETE |
| P2-S4 | Project Import / Export | COMPLETE |
| P2-S5 | Jira Connector | COMPLETE |
| P2-S6 | GitHub Connector | COMPLETE |
| P2-S8 | Automation Pipeline | COMPLETE through current checkpoints |
| P2-S8.6 | Automation Candidate Selection | COMPLETE |
| P2-S8.7 | Candidate → Automation Generation | COMPLETE |
| P2-S8.8 | Automation Case Validation | COMPLETE |
| P2-S8.8+ | Automation Code Generation | COMPLETE |
| P2-S8.9 | Controlled Automation Execution | COMPLETE |
Slack integration exists behind service/client abstractions.
---
# 7. PROJECT CONTEXT AND PERSISTENCE
Conceptually:
```text
QAProject
|
+-- project_id
+-- name
+-- description
+-- application
+-- environment
+-- metadata
+-- requirements
+-- test suites
```
Persistence:
```text
ProjectContext
|
v
ProjectRepository
|
v
SQLiteProjectRepository
|
v
SQLite
```
Database:
```text
data/qa_mcp.db
```
**Important:** Never delete the persistent database merely to make tests pass.
Persistence-focused tests should use isolated database state.
---
# 8. EXTERNAL CONNECTORS
## Jira
Abstraction:
```text
MCP
|
v
JiraService
|
v
JiraClient
+-- MockJiraClient
+-- JiraCloudClient
```
Current real operations are read-only:
```text
get_jira_issue(issue_key)
search_jira_issues(jql, max_results=50)
```
No Jira write operations are part of the completed connector milestone.
## GitHub
Abstraction:
```text
MCP
|
v
GitHubService
|
v
GitHubClient
+-- MockGitHubClient
+-- GitHubCloudClient
```
Current read-only tools:
```text
get_github_repository(owner, repository)
get_github_issue(owner, repository, issue_number)
get_github_pull_request(owner, repository, pull_number)
search_github_issues(query, max_results=50)
```
No GitHub write operations are part of the completed connector milestone.
## Slack
Abstraction:
```text
SlackService
|
v
SlackClient
+-- MockSlackClient
+-- SlackCloudClient
```
Current tools include:
```text
get_slack_channel
get_slack_messages
search_slack_messages
get_slack_thread
```
---
# 9. AUTOMATION PIPELINE
```text
Test Cases
|
v
Automation Candidate Selection
|
v
Automation Case
|
v
Automation Validation
|
v
Framework-specific Code Generation
|
v
GeneratedAutomationArtifact
|
v
Controlled Execution
|
v
AutomationExecutionResult
```
Candidate selection deliberately distinguishes:
```text
Recommended for automation
|
+---- Automated
|
+---- Manual-only
```
Manual-only test cases must not be sent to the automation generator.
---
# 10. AUTOMATION CHECKPOINTS ALREADY COMPLETE
## P2-S8.6 — Candidate Selection
`AutomationCandidateSelector` / `AutomationCandidateService`
Result:
```text
AutomationCandidateResult
+-- candidate_ids
+-- manual_ids
+-- total
```
MCP tool:
```text
select_automation_candidates
```
## P2-S8.7 — Candidate → Automation Generation
Service:
```text
AutomationCandidateGenerationService
```
Flow:
```text
TestCase[]
|
v
Candidate Selection
|
v
candidate_ids
|
v
Generate automation ONLY for candidates
|
v
AutomationCase[]
```
Zero-candidate behavior:
```text
No automation candidates
|
v
[]
|
v
Automation generator is NOT called
```
MCP tool:
```text
generate_automation_for_candidates
```
## P2-S8.8 — Automation Case Validation
Validator:
```text
AutomationValidator
```
Result:
```text
AutomationValidationResult
+-- automation_case_id
+-- test_case_id
+-- valid
+-- errors
+-- warnings
```
Minimum integrity:
- At least one automation step.
- Validation failures are structured errors.
- Non-blocking concerns can be warnings.
- Validation remains separate from generation.
## P2-S8.8+ — Automation Code Generation
Artifact:
```text
GeneratedAutomationArtifact
+-- id
+-- automation_case_id
+-- framework
+-- language
+-- file_name
+-- code
```
Current execution target:
```text
Framework: Playwright
Language: Python
```
Generated automation must be validated before downstream execution.
---
# 11. CONTROLLED AUTOMATION EXECUTION — P2-S8.9 COMPLETE
The committed local execution pipeline is:
```text
GeneratedAutomationArtifact
|
v
AutomationExecutionConfig
|
v
AutomationWorkspace
|
v
AutomationExecutionRunner
|
v
AutomationExecutionService
|
v
AutomationExecutionResult
|
v
MCP execute_automation_code
```
## Execution configuration
```text
AutomationExecutionConfig
+-- timeout_seconds = 60
+-- workspace_root = optional
```
The configuration is immutable.
## Automation workspace
`AutomationWorkspace` creates an isolated temporary directory for the generated artifact.
The workspace is cleaned up after execution unless explicit retention is requested.
The project working tree must not be used as the normal generated-artifact execution directory.
## Controlled subprocess runner
`AutomationExecutionRunner`:
- accepts an explicit command list
- runs from a supplied working directory
- captures stdout
- captures stderr
- captures exit code
- measures execution duration
- enforces a timeout
- reports timeout separately
- reports operating-system execution errors separately
The runner is injectable so tests do not need to execute real automation processes.
## Execution service
Current validation:
```text
Empty code
-> ValueError
Missing framework
-> ValueError
Unsupported framework
-> ValueError
```
Current supported framework:
```text
Playwright
```
Current Python execution command:
```text
python -m pytest <generated_file_name>
```
Status mapping:
```text
exit_code == 0
-> PASSED
exit_code != 0
-> FAILED
timed_out
-> TIMEOUT
runner error
-> ERROR
```
Separation:
```text
Runner
= process mechanics
ExecutionService
= QA execution semantics
AutomationExecutionResult
= stable domain contract
```
Execution IDs are currently deterministic in the service foundation (`EX001`). Durable unique execution IDs belong to the future execution-history/persistence layer.
---
# 12. EXECUTION SAFETY REQUIREMENTS
The current subprocess runner is a controlled local execution boundary, **not** the final production-grade sandbox.
Intended progression:
```text
Current
Local controlled subprocess
|
v
Hardened execution boundary
|
v
Container / isolated execution
|
v
Cloud or CI execution
```
Mandatory safety direction:
- Do not introduce arbitrary shell execution.
- Do not construct unrestricted commands from user input.
- Keep framework support explicit.
- Keep generated filenames and execution paths controlled.
- Keep execution bounded by timeouts.
- Preserve workspace isolation.
- Keep the runner injectable and testable.
- Introduce containerization before exposing execution to untrusted production workloads.
Do not add container/cloud complexity before the local execution contract and orchestration behavior are stable.
---
# 13. MCP AUTOMATION SURFACE
Current automation-related MCP tools:
```text
generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code
```
`execute_automation_code(artifact)`:
1. Validates the incoming artifact through `GeneratedAutomationArtifact`.
2. Delegates to `AutomationExecutionService`.
3. Returns `AutomationExecutionResult.model_dump()`.
4. Converts invalid execution-artifact input into a controlled MCP-facing error.
The MCP layer must not contain subprocess implementation details.
---
# 14. TEST STRATEGY AND CURRENT BASELINE
Test-first development remains mandatory.
Expected sequence:
```text
Write failing test
|
v
Implement smallest production change
|
v
Focused test
|
v
Related tests
|
v
Full regression
|
v
Runtime/MCP verification
|
v
README update
|
v
git diff --check
|
v
Commit + push
```
Current verified baseline before P2-S9.1.a:
```text
pytest -q
190 passed
7 warnings
0 failures
```
P2-S9.1.a verified regression:
```text
pytest -q
204 passed
7 warnings
0 failures
```
P2-S9.1.a focused execution suite:
```text
16 passed
1 warning
```
Execution service suite:
```text
7 passed
```
No test was removed or weakened to obtain the current green baseline.
---
# 15. KNOWN WARNINGS / TECHNICAL DEBT
## Pytest collection warnings
Pydantic models named:
```text
TestCase
TestCaseReview
```
can be interpreted by pytest as possible test classes, producing `PytestCollectionWarning`.
These are non-functional warnings.
Future cleanup may use test-only import aliases. Keep this separate from feature work.
## Pydantic settings warning
Existing:
```text
IncompleteFieldDefinitionWarning
```
related to the `lifespan` forward reference in `pydantic_settings`.
It does not currently cause test failures.
Keep this as separate technical debt unless it blocks development.
---
# 16. ENVIRONMENT / .ENV DOCUMENTATION
## Critical rule
The following is the **documented `.env` template currently used by the development setup**.
**These are placeholders, not real credentials.**
Never commit a real `.env` file, API token, password, or secret to Git.
The actual local `.env` remains developer-machine configuration.
## Current `.env` template
```dotenv
JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token
GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org
# ---------------------------------------------------------
# Slack
# ---------------------------------------------------------
SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=
```
## Variable purpose
| Variable | Purpose | Secret? |
|---|---|---|
| `JIRA_URL` | Jira Cloud base URL | No |
| `JIRA_EMAIL` | Jira API account email | No, but treat as configuration |
| `JIRA_API_TOKEN` | Jira API authentication | **YES** |
| `GITHUB_URL` | GitHub API base URL | No |
| `GITHUB_TOKEN` | GitHub API authentication | **YES** |
| `GITHUB_OWNER` | GitHub username/org used by configuration | No |
| `SLACK_URL` | Slack API base URL | No |
| `SLACK_TOKEN` | Slack API authentication | **YES** |
| `SLACK_DEFAULT_CHANNEL` | Default Slack channel configuration | No |
## Deployment/configuration rule
When configuring a new environment:
1. Copy the documented template into a local `.env`.
2. Replace only the placeholder values required for that environment.
3. Never paste real secrets into this README.
4. Never commit the populated `.env`.
5. Verify `.gitignore` protects `.env`.
6. Keep configuration changes documented here when they materially affect deployment.
7. If new environment variables are introduced, update this section in the same development checkpoint.
---
# 17. CONFIGURATION
Primary configuration:
```text
config/settings.yaml
```
Environment-specific secrets are supplied through environment variables.
Known integration variables:
```text
JIRA_URL
JIRA_EMAIL
JIRA_API_TOKEN
GITHUB_URL
GITHUB_TOKEN
GITHUB_OWNER
SLACK_URL
SLACK_TOKEN
SLACK_DEFAULT_CHANNEL
```
Secrets must remain outside source control.
---
# 18. DEVELOPMENT ENVIRONMENT
Python requirement:
```text
Python >= 3.11
```
Current development environment used during the latest verification:
```text
Python 3.12 virtual environment
.venv/
```
Activate:
```bash
source .venv/bin/activate
```
Install project dependencies according to the repository's `requirements.txt`.
Run all tests:
```bash
pytest -q
```
Run a specific test:
```bash
pytest -q tests/<test_file>.py
```
Check MCP tools:
```bash
python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools()])"
```
Check automation tools:
```bash
python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools() if 'automation' in t.name])"
```
Check formatting issues:
```bash
git diff --check
```
Check repository state:
```bash
git status
```
Review recent commits:
```bash
git log -5 --oneline
```
---
# 19. GIT CHECKPOINT HISTORY
Important checkpoints:
```text
a288569 Initial commit with configured gitignore
1144ddd Resolve README.md merge conflict
882e149 Jira Connector Added
06dbe61 Complete GitHub connector
71c893e Initial commit with Slack Configured
169c1a1 Complete automation case generator
1d2360b Add automation candidate pipeline
a226e9e Add automation case validation
5ced4e3 Add automation code generation
715ad52 Complete P2-S8.8 automation code generation
703cdb1 Complete automation execution foundation
3bdf761 Implement controlled automation execution
add5ba2 Update project continuity roadmap
```
Every completed checkpoint must contain:
```text
Implementation
Tests
README
Verification evidence
Commit
Push
Clean working tree
```
---
# 20. WHAT HAS ALREADY BEEN COMPLETED — DO NOT REBUILD
These capabilities are already implemented/tested and must not be redesigned or recreated as if they were new:
```text
MCP server foundation
Configuration
LLM abstraction
Mock LLM
Requirement analysis
Test case generation
Test case review
QA suite workflow
Project context
SQLite persistence
Requirement/suite versioning
Import/export
Jira connector
GitHub connector
Slack connector
Automation case generation
Automation candidate selection
Candidate → automation orchestration
Automation case validation
Playwright/Python automation code generation
GeneratedAutomationArtifact contract
AutomationExecutionResult contract
AutomationWorkspace
AutomationExecutionRunner
AutomationExecutionService
execute_automation_code MCP boundary
```
Future work must build on these components.
---
# 21. NEXT DEVELOPMENT CHECKPOINT
## P2-S9.1 — Execution Hardening
**STATUS: IN PROGRESS — P2-S9.1.b.1 COMPLETE**
Completed sub-step:
```text
P2-S9.1.b.1 — Controlled Automation Command Boundary
```
Next sub-step:
```text
P2-S9.1.b.2 — Further Command/Execution Policy Hardening
```
Do not return to:
- candidate selection
- automation generation
- automation validation
- already-completed controlled local execution foundation
Immediate direction:
```text
P2-S8.9 Controlled local execution
|
v
P2-S9.1 Execution hardening
|
+-- safe workspace/file handling
+-- stronger command validation
+-- execution identity
+-- configurable limits
+-- better failure classification
+-- artifact/result retention policy
|
v
P2-S9.2 Execution evidence
|
+-- generated artifact metadata
+-- stdout/stderr evidence
+-- execution metadata
+-- result persistence
+-- traceability
|
v
P2-S9.3 Execution history
|
v
P2-S9.4 Reporting / analysis
|
v
P2-S10 Agent orchestration
```
P2-S9.1.a and P2-S9.1.b.1 are implemented and verified. The next sub-step must be defined and tested before implementation.
---
# 21A. COMPLETED CHECKPOINT — P2-S9.1.a
## Execution Hardening — Safe Workspace/File Handling
**STATUS: COMPLETE**
P2-S9.1.a hardens the generated automation workspace boundary without changing the established execution contracts or MCP execution flow.
Implemented:
- Strict generated artifact filename validation.
- Rejection of empty and whitespace-only filenames.
- Rejection of `.` and `..`.
- Rejection of POSIX absolute paths and traversal paths.
- Rejection of Windows-style traversal and drive-style paths.
- Explicit resolved-path containment verification before writing.
- Filename validation before workspace creation.
- Generated artifacts remain constrained to the controlled workspace.
Tests:
```text
14 new workspace-hardening tests
Focused execution suite: 28 passed
Full regression: 204 passed, 0 failures, 7 warnings
git diff --check: clean
```
The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15 and are intentionally outside this checkpoint.
No existing Pydantic execution contracts were changed.
**Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening**
---
# 21B. COMPLETED CHECKPOINT — P2-S9.1.b.1
## Execution Hardening — Controlled Automation Command Boundary
**STATUS: COMPLETE**
Implemented:
- Explicit controlled command construction in `AutomationExecutionService`.
- Playwright automation is restricted to `python -m pytest <artifact_file_name>`.
- Unsupported automation frameworks are rejected before command construction.
- Unsafe artifact filenames are rejected before command construction.
- `AutomationExecutionRunner` remains a generic subprocess execution wrapper.
- Existing Pydantic execution contracts and the MCP execution boundary remain unchanged.
Tests:
```text
Focused execution service suite: 10 passed
Full regression: 207 passed, 0 failures, 7 warnings
git diff --check: clean
```
The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15.
**Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening**
---
# 22. FUTURE EXECUTION ARCHITECTURE
Target:
```text
GeneratedAutomationArtifact
|
v
Execution Policy / Safety Validation
|
v
Isolated Execution Environment
|
v
Framework Runner
|
v
Execution Evidence
|
v
Execution Result
|
v
Persistent Execution History
|
v
Reporting / AI Analysis
```
Potential isolation progression:
```text
Local hardened process
|
v
Docker/container
|
v
CI worker
|
v
Cloud execution
```
Do not implement all layers at once.
---
# 23. EVENTUAL AGENT-DRIVEN QA WORKFLOW
The eventual product experience:
```text
Understanding requirement...
|
v
Analyzing scenarios...
|
v
Generating test cases...
|
v
Reviewing coverage...
|
v
Identifying automation candidates...
|
v
Generating automation...
|
v
Validating automation...
|
v
Executing automation...
|
v
Analyzing results...
|
v
Preparing QA report...
```
MCP is intended to become the capability layer underneath an agent-driven QA product.
---
# 24. EVENTUAL PRODUCT / UI DIRECTION
The eventual UI should make the agent's progress, generated artifacts, execution state, and results visible and understandable.
```text
User
|
v
QA MCP UI
|
v
Agent / MCP Orchestration
|
+-- Requirement analysis
+-- Test generation
+-- Test review
+-- Candidate selection
+-- Automation generation
+-- Automation validation
+-- Automation execution
+-- Results / reporting
+-- Jira
+-- GitHub
+-- Slack
```
The UI and hosted product layer should be introduced only after the core QA-agent capabilities are sufficiently stable.
---
# 25. LONG-TERM PRODUCT DIRECTION
The final product should evolve toward:
```text
Understand
|
v
Plan
|
v
Generate
|
v
Validate
|
v
Execute
|
v
Observe
|
v
Analyze
|
v
Report
|
v
Learn / Improve
```
Long-term capabilities:
- Requirements intelligence
- Test design
- Test review
- Automation selection
- Automation generation
- Automation validation
- Safe execution
- Execution evidence
- Failure analysis
- Coverage analysis
- Regression intelligence
- External engineering-system context
- CI/CD integration
- Agent orchestration
- Interactive UI
- Hosted/cloud execution
These are future goals, not permission to prematurely implement everything.
---
# 26. DEVELOPMENT PRINCIPLES
The following principles must remain unchanged:
1. Build incrementally.
2. Write tests before implementation where practical.
3. Keep services small and composable.
4. Keep MCP tools thin.
5. Keep external integrations behind infrastructure abstractions.
6. Avoid destabilizing existing workflows.
7. Preserve structured Pydantic contracts.
8. Keep secrets outside source control.
9. Run full regression before every feature checkpoint.
10. Update this README whenever a meaningful feature checkpoint is committed.
11. Commit code, tests and README together for each completed checkpoint.
12. Prefer explicit contracts over implicit behavior.
13. Prefer deterministic behavior over clever behavior.
14. Keep execution safety ahead of execution convenience.
15. Keep production concerns separated from prototype convenience.
16. Do not duplicate completed capabilities.
17. Do not silently change established contracts.
18. Maintain requirement → test case → automation case → artifact → execution result traceability.
19. Treat deployment/configuration documentation as part of the implementation.
20. Treat this README as the continuity record, not optional documentation.
---
# 27. CURRENT RESUME POINT
## Resume from
**P2-S9.1.b.2 — Further Command/Execution Policy Hardening**
P2-S9.1.a — Safe Workspace/File Handling and P2-S9.1.b.1 — Controlled Automation Command Boundary are complete and must not be recreated.
Previous completed checkpoints:
```text
P2-S8.6 Automation Candidate Selection COMPLETE
P2-S8.7 Candidate → Automation Generation COMPLETE
P2-S8.8 Automation Case Validation COMPLETE
P2-S8.8+ Automation Code Generation COMPLETE
P2-S8.9 Controlled Automation Execution COMPLETE
```
Verified baseline after P2-S9.1.b.1:
```text
207 passed
7 warnings
0 failures
```
Latest repository implementation commit:
```text
b6190d9 Harden automation workspace file handling
```
Current checkpoint commit:
```text
Pending user commit
```
Previous implementation checkpoint:
```text
3bdf761 Implement controlled automation execution
```
Current automation MCP surface:
```text
generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code
```
---
# 28. CRITICAL CONTINUITY INSTRUCTION FOR A NEW CHAT
A future development session must:
1. Read this README first.
2. Inspect the current GitHub `main` branch:
`https://github.com/sanumenon/qa-mcp/tree/main`
3. Confirm the latest commit and test baseline.
4. Inspect the existing implementation before proposing changes.
5. Start from **P2-S9.1.b.2 — Further Command/Execution Policy Hardening**.
6. Treat **P2-S9.1.a — Safe Workspace/File Handling** as complete.
7. Treat **P2-S9.1.b.1 — Controlled Automation Command Boundary** as complete.
8. Do not recreate candidate selection.
9. Do not recreate automation generation.
10. Do not recreate automation validation.
11. Do not recreate controlled local execution.
12. Do not silently replace established architecture/contracts.
13. Add tests first wherever practical.
14. Keep the architecture layered.
15. Verify focused tests.
16. Verify the full regression suite.
17. Verify the MCP/runtime path for major capabilities.
18. Update this README at the end of every verified checkpoint.
19. Include deployment/configuration changes in this README.
20. Never commit real secrets or a populated `.env`.
21. Commit and push code + tests + README together.
22. Verify the working tree is clean after the checkpoint.
23. Never make the user repeat already-completed development work when the repository and README contain it.
24. Never use a new chat as a reason to restart the project from an earlier phase.
**This README is part of the implementation and must be treated as the project's authoritative continuity record.**
---
## P2-S9.2 — Executable Playwright Code Generation
Status: COMPLETE
Implementation commit:
```text
176ab59 Implement executable Playwright code generation
```
Implemented in:
```text
src/qa_mcp/core/automation/code_generation_service.py
```
Generated Playwright/Python artifacts now translate the controlled automation DSL into executable Playwright code.
Supported automation steps:
```text
goto: <url>
fill: <selector> = <value>
click: <selector>
press: <selector> = <key>
```
Supported assertions:
```text
visible: <selector>
text: <selector> = <expected text>
url: <expected url>
```
The generator now:
- Produces executable Playwright/Python code instead of comments.
- Generates `Page` and `expect` based Playwright code.
- Rejects unsupported automation steps.
- Rejects unsupported automation assertions.
- Validates malformed step/assertion expressions.
- Preserves the existing `GeneratedAutomationArtifact` contract.
- Keeps the existing `execute_automation_code` MCP boundary unchanged.
- Supports generated-artifact → execution-service integration.
Verification:
```text
pytest -q tests/test_automation_code_generation_service.py tests/test_automation_code_generation_empty.py tests/test_automation_code_generation_result.py
8 passed
0 failures
pytest -q tests/test_automation_execution_service.py
17 passed
0 failures
pytest -q
216 passed
7 warnings
0 failures
git diff --check
clean
```
The 7 pytest/Pydantic warnings are existing non-blocking technical debt and are not part of P2-S9.2.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
```
### Next implementation checkpoint
```text
P2-S9.3 — Real Playwright Execution Validation
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, or controlled execution functionality.
---
---
## P2-S9.3 — Execution History and Persistence
Status: COMPLETE
Implementation commit:
```text
7b8a67b Add automation execution history
```
Verification:
```text
Focused execution/history suite: 26 passed
Full regression suite: 227 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Unique execution IDs are generated for every automation execution.
- Automation execution results are persisted in SQLite.
- Execution history can be retrieved by execution ID.
- Execution history can be listed with optional automation-case filtering and result limits.
- `execute_automation_code` now persists execution results.
- Added MCP tool: `get_automation_execution`.
- Added MCP tool: `list_automation_executions`.
- Added repository, application-service, execution-service, and MCP-tool test coverage.
- Existing execution behavior and contracts remain intact.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Reporting / Analysis
```
### Next implementation checkpoint
```text
P2-S9.4 — Reporting / Analysis
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, or execution-history functionality.
---
## P2-S9.4 — Reporting / Analysis
Status: COMPLETE
Implementation commit: 55ddd09 Add automation execution reporting
Verification:
```text
Focused P2-S9.4 tests: 12 passed
Full regression suite: 234 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added aggregated automation execution reporting.
- Added execution totals by result status.
- Added pass-rate calculation.
- Added total and average execution duration metrics.
- Added latest execution identification and status.
- Added optional automation-case filtering.
- Added persistent SQLite-backed reporting.
- Added reporting application-service support.
- Added MCP reporting capability.
- Added repository, service, and MCP-tool test coverage.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
```
### Next implementation checkpoint
```text
P2-S9.5 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, or reporting functionality.
---
## P2-S9.5 — Failure Analysis
Status: COMPLETE
Implementation commit: 77a8d9b Add automation execution failure analysis
Verification:
```text
Focused P2-S9.5 tests: 15 passed
Full regression suite: 242 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added structured automation execution failure analysis.
- Added failure-analysis models and application-service support.
- Added persisted failure-analysis retrieval from execution history.
- Added analysis of failed and errored executions.
- Added execution identifiers, automation artifact identifiers, and automation case traceability.
- Added MCP failure-analysis capability.
- Added repository, service, and MCP-tool test coverage.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
```
### Next implementation checkpoint
```text
P2-S9.6 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, or failure-analysis functionality.
---
## P2-S9.6 — Web Dashboard
Status: COMPLETE
Implementation commit: 79e3c3f Add web dashboard
Verification:
```text
Focused Web Dashboard tests: 5 passed
Full regression suite: 247 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added FastAPI-based web application.
- Added browser-accessible QA automation dashboard.
- Added execution reporting view.
- Added execution history view.
- Added failure-analysis view.
- Added REST endpoints for execution reporting, execution history, and failure analysis.
- Added dedicated web entrypoint through `run_web.py`.
- Added web dashboard automated test coverage.
- Added FastAPI to `requirements.txt`.
### Start the Web Dashboard
From the project root with the virtual environment activated:
```bash
cd ~/pythonPrograms/qa-mcp
source .venv/bin/activate
python run_web.py
```
The dashboard is then available at:
```text
http://127.0.0.1:8000
```
### Web API endpoints
```text
GET /
GET /api/executions/report
GET /api/executions?limit=20
GET /api/executions/failures?limit=20
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
```
### Next implementation checkpoint
```text
P2-S9.7 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, or web-dashboard functionality.
---
## P2-S9.7 — AI QA Workspace
Status: COMPLETE
Implementation scope:
- Added a browser-based AI QA Workspace to the existing web dashboard.
- Added persistent QA project creation and retrieval.
- Added project-aware requirement analysis.
- Added project-aware QA test-suite generation.
- Added AI-generated test-case review.
- Added requirement version persistence.
- Added QA-suite version persistence.
- Added workspace REST APIs.
- Added deterministic MockLLM support for workspace development and tests.
- Added automated coverage for project, workspace service, and MockLLM behavior.
- Preserved all existing execution, reporting, failure-analysis, and dashboard functionality.
### AI QA Workspace flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### Workspace API endpoints
```text
POST /api/projects
GET /api/projects/{project_id}
POST /api/projects/{project_id}/qa-suite
```
### Workspace behavior
A QA project must exist before a QA suite can be generated for that project.
Existing projects can be retrieved using:
```text
GET /api/projects/{project_id}
```
A requirement can then be submitted using:
```text
POST /api/projects/{project_id}/qa-suite
```
The generated response contains:
```text
project
requirement_version
suite_version
requirement
analysis
test_cases
review
```
Requirement versions and QA-suite versions are persisted independently so that generated QA work remains traceable to the project and requirement history.
### Verification
```text
Focused S9.7 workspace tests: 19 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
```
### P2-S9.8 — Automation Candidate Selection
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing Automation Candidate Service.
- Added automation candidate selection to QA-suite generation.
- Added automation candidate results to the generated QA-suite response.
- Preserved manual-test identification through the existing candidate-selection workflow.
- Added focused regression coverage for automation candidate selection.
- Preserved all existing QA Workspace, execution, reporting, failure-analysis, and dashboard functionality.
### Automation Candidate flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### QA-suite response
The generated QA-suite response now includes:
```text
automation_candidates
candidate_ids
manual_ids
total
```
Candidate selection is performed through the existing:
```text
AutomationCandidateService
↓
AutomationCandidateSelector
```
### Verification
```text
Focused S9.8 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
```
### P2-S9.9 — AI QA Workspace Automation Generation
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing automation candidate generation service.
- Added dependency injection for automation candidate generation.
- Added automation case generation during QA-suite creation.
- Added generated automation cases to the QA-suite response.
- Preserved the existing automation candidate selection flow.
- Added focused regression coverage for automation generation.
- Preserved all existing QA Workspace, candidate selection, execution, reporting, failure-analysis, and dashboard functionality.
### AI QA Workspace automation flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Case Generation
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### QA-suite automation response
The generated QA-suite response now includes:
```text
automation_candidates
candidate_ids
manual_ids
total
automation_cases
test_case_id
automation_type
```
Automation generation is performed through the existing service boundary:
```text
AutomationCandidateGenerationService
↓
Generated Automation Cases
```
### Verification
```text
Focused S9.9 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
↓
Automation Case Generation
```
### P2-S9.10 — AI QA Workspace Automation Wiring
Status: COMPLETE
Implementation scope:
- Connected the production web application to the existing automation-generation pipeline.
- Added the workspace automation case generator using the workspace LLM.
- Added the workspace automation service.
- Added the existing automation candidate service and selector to the production workspace.
- Added the automation candidate generation service to the AI QA Workspace.
- Connected `QAWorkspaceService` to the production automation candidate generation service.
- Added focused regression coverage confirming the production dependency is injected correctly.
- Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.
### Automation generation flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### Production wiring
```text
Workspace LLM
↓
AutomationCaseGenerator
↓
AutomationService
↓
AutomationCandidateGenerationService
↑
AutomationCandidateService
↑
AutomationCandidateSelector
↓
QAWorkspaceService
↓
AI QA Workspace API
```
The web application now constructs the real automation-generation dependencies instead of relying on test-only or implicit service construction.
### Verification
```text
Focused S9.10 workspace-service tests: 3 passed
Full regression suite: 262 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
```
### P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
Status: IN PROGRESS
Current implementation state:
- Persistent Project QA Workspace is implemented.
- Persisted project requirements, requirement versions, saved QA suite versions, test cases, automation candidates, and generated automation artifacts are exposed through the workspace.
- Automation candidates are now actionable from the Project QA Workspace.
- Added `QAProjectAutomationGenerationRequest` for project-level automation generation requests.
- Added `QAWorkspaceService.generate_automation_from_project()` to continue from persisted test cases.
- Persisted test-case dictionaries are reconstructed through the existing `TestCase` model before candidate processing.
- Requested test-case IDs are validated against the persisted project workspace.
- Selected test cases are rechecked through the existing `AutomationCandidateService`.
- Non-candidate selections are rejected instead of bypassing the existing candidate policy.
- Existing `AutomationCandidateGenerationService` is reused to generate automation cases.
- Existing `AutomationCodeGenerationService` is reused to generate automation artifacts.
- Added `POST /api/projects/{project_id}/automation`.
- Added Project QA Workspace UI controls for selecting automation candidates.
- Added a Generate Automation action in the Project QA Workspace.
- Browser-side UI wiring invokes the new project automation endpoint and refreshes the persisted workspace state.
- Existing Generate QA Suite functionality remains separate and unchanged.
- Added dashboard/API regression coverage for the new workflow.
- Focused workspace/dashboard tests: 34 passed.
- Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.
- `git diff --check` is clean.
P2-S9.12 product flow:
Project
-> Requirement
-> Requirement Analysis
-> Test Case Generation
-> AI Review
-> Preview
-> User selects test cases
-> Save
-> QASuiteVersioningService
-> SQLite qa_suite_versions
-> Project QA Workspace
-> Existing Requirements
-> Requirement Versions
-> Saved Suite Versions
-> Existing Test Cases
-> Automation Candidates
-> Select Automation Candidates
-> Generate Automation
-> Automation Case Generation
-> Automation Artifact Generation
-> Controlled Automation Execution
-> Execution History / Reporting
-> Failure Analysis
Implemented user capability:
- View requirements already provided or prepared for the project.
- View requirement analysis and version information already prepared.
- View test cases already generated and persisted for the project.
- View automation candidates derived from persisted test cases.
- Select one or more persisted automation candidates.
- Generate automation from the selected candidates.
- Generate automation artifacts through the existing automation code-generation pipeline.
- Keep generated automation associated with the project workspace.
- Preserve the existing active Generate QA Suite workflow independently.
Remaining P2-S9.12 scope:
Project QA Workspace
-> View persisted test cases
-> View automation candidates
-> Select automation candidates
-> Generate/use automation cases
-> Validate automation cases
-> Generate automation artifacts
-> Controlled automation execution
-> Execution result/history/reporting
-> Failure analysis
Completed automation candidate selection, candidate generation, automation case generation, validation, Playwright code generation, artifact generation, command-boundary enforcement, execution configuration, controlled execution, execution history, reporting, and failure-analysis services must continue to be reused rather than rebuilt.
Current checkpoint:
P2-S9.12 QA Suite Generation and LLM Error-Boundary Hardening
The Project QA Workspace QA-suite generation flow has been hardened to require a complete test-case suite covering the supplied positive scenarios, negative scenarios, and edge cases. The generator now rejects malformed or incomplete LLM payloads instead of silently normalizing them into a single test case.
The LLM generation boundary now distinguishes unusable provider output from application validation failures. In particular, a provider/guardrail refusal is surfaced as `LLMGenerationError` and returned by the QA-suite API as HTTP 502 rather than being incorrectly classified as HTTP 404.
During validation, the configured Bedrock provider returned the following non-JSON response:
`The response was blocked by dev-guardrails policy. If this looks like a false positive, ping #ai-guardrails.`
This confirmed that the observed single-test-case UI symptom was not caused by application-side test-case truncation. The provider response was blocked before a test-case suite could be produced.
Validation completed:
```text
Focused generator/dashboard tests: 30 passed
Full regression suite: 304 passed
Warnings: 8 known non-blocking warnings
Failures: 0
git diff --check: clean
```
The provider-failure diagnostic path now preserves the raw LLM provider response on `LLMGenerationError.provider_response` and logs that response at error level for IT troubleshooting. The API continues to return only the safe generic HTTP 502 message and does not expose the provider response to the end user.
The diagnostic regression test verifies that the exact provider response is retained. Focused diagnostic validation passed with 39 tests, and the full regression suite remains at 304 passed with 8 known non-blocking warnings and 0 failures.
The implementation is intentionally limited to prompt hardening, strict test-case response validation, LLM-generation error classification, API error mapping, and regression coverage. Do not bypass or weaken provider/dev-guardrail policy as part of this fix.
Next implementation:
P2-S9.12 continuation — Controlled Automation Execution from Project QA Workspace
The next implementation should wire the already-generated project automation artifacts into the completed controlled execution pipeline, including execution configuration, command-boundary enforcement, execution history, reporting, and failure analysis.
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, automation artifact generation, or the QA-suite generation/error-boundary hardening completed in this checkpoint.
### P2-S9.11 — AI QA Workspace Artifact Generation
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing automation code-generation service.
- Added production construction of `AutomationCodeGenerationService`.
- Connected the service to `QAWorkspaceService` through dependency injection.
- Added automation artifact generation for generated automation cases.
- Added `automation_artifacts` to the QA-suite workspace response.
- Added persistent SQLite repository support for QA Workspace automation artifacts.
- Added focused regression coverage for artifact persistence, artifact generation, and dependency injection.
- Added browser-level Playwright regression coverage for the AI QA Workspace dashboard.
- Verified the rendered dashboard preserves the AI QA Workspace and existing Automation Execution Overview.
- Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.
### Automation artifact flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
↓
Automation Code Generation
↓
Automation Artifact
↓
Persistent Artifact Repository
↓
QA Workspace Response
```
### Workspace response
The generated QA-suite response now includes:
```text
automation_candidates
automation_cases
automation_artifacts
requirement_version
suite_version
```
### Production wiring
```text
Workspace LLM
↓
AutomationCaseGenerator
↓
AutomationService
↓
AutomationCandidateGenerationService
↓
AutomationCase
↓
AutomationCodeGenerationService
↓
AutomationArtifact
↓
QAWorkspaceService
↓
AI QA Workspace API
```
### Dashboard regression protection
The AI QA Workspace is protected by both structural and browser-level regression tests.
The dashboard regression coverage verifies:
```text
AI QA Workspace
↓
Create QA Project controls
↓
Generate QA Suite controls
↓
Project / requirement inputs
↓
JavaScript action wiring
↓
Workspace API wiring
↓
Rendered dashboard
↓
Existing Automation Execution Overview preserved
```
The browser regression test uses Playwright with Chromium against a live Uvicorn instance and verifies the rendered dashboard and the important workspace controls.
The browser regression test is now part of the permanent dashboard regression suite and must remain green during future UI changes.
### Verification
```text
Focused dashboard/workspace tests: 12 passed
Browser regression test: 1 passed, 11 deselected
Full regression suite: 269 passed
Warnings: 8 known non-blocking warnings
Failures: 0
git diff --check: clean
```
**P2-S9.11 is complete. Do not recreate or redesign this capability.**
### Next implementation checkpoint
```text
P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
```
P2-S9.12 has progressed from the persistent Project QA Workspace to actionable automation-candidate selection and generation. The workspace can now select persisted automation candidates and invoke the existing automation generation and artifact-generation pipeline. This implementation is committed as `e186391`. The remaining work is to continue from generated automation artifacts into the completed controlled execution, execution-history, reporting, and failure-analysis pipeline.
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, or automation artifact generation.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues