QA MCP Server
by sanumenon
README.md
# QA MCP
QA MCP is a Model Context Protocol (MCP) server for structured software-quality workflows.
The project is being developed incrementally toward a full-fledged AI-powered QA platform using:
- Layered architecture
- Test-first development
- Pydantic-based contracts
- Persistent SQLite storage
- Immutable QA versioning
- Project import/export
- Safe external connectors
- MCP tool boundaries
- LLM-assisted QA analysis and automation generation
- Automation candidate selection
- Playwright automation generation
- Controlled automation execution
- Eventual QA-agent orchestration
- Eventual CI/CD and hosted product capabilities
> **CONTINUITY RULE:** This README is the authoritative development, deployment, roadmap, and continuity checkpoint for future QA MCP development sessions. Read it before starting new development. Do not recreate completed work.
---
# 1. CURRENT DEVELOPMENT CHECKPOINT
**Current checkpoint:** P2-S9.12 — Bedrock Test-Case Generation Output and QA Suite Completeness
**Checkpoint commit:** `70d1f8f`
**Repository:** `https://github.com/sanumenon/qa-mcp`
**Branch:** `main`
**Latest verified baseline:**
- Full regression: **312 passed**
- Warnings: **8**
- Failures: **0**
- `git diff --check`: clean
- Working tree: clean at the last verified checkpoint
## Latest verified implementation notes
### Bedrock test-case generation hardening
- Commit `9e5f8f7` normalizes a valid single test-case object into the required
`{"test_cases": [...]}` response wrapper.
- Commit `70d1f8f` increases the Bedrock `maxTokens` configuration from `4096`
to `12000`.
- The full regression suite remains green after both changes.
- A single test-case response is accepted only when it matches the valid
test-case schema; unrelated or incomplete payloads remain errors.
- The generation prompt already requires comprehensive coverage of positive,
negative, and edge scenarios.
- Runtime verification is still required to confirm that the model returns the
complete suite rather than only one case.
### Current known investigation
The Customer Portal requirement analysis contains multiple scenario categories,
but the runtime previously displayed only one generated test case. The larger
Bedrock output limit is the first verification step.
If regeneration still produces only one or an incomplete subset, the next
implementation must be scenario-batched generation with:
1. Explicit scenario-category inputs.
2. Bounded batches.
3. Stable sequential test-case IDs.
4. Response validation for every batch.
5. Deduplication and deterministic merge behavior.
6. Minimum coverage/count validation.
7. Controlled retry behavior.
8. Focused tests and full regression coverage.
Do not solve this by weakening validation, deleting tests, redesigning the UI,
or blindly increasing token limits indefinitely.
## Completed capabilities
The following capabilities are implemented and verified:
1. Persistent project QA workspace.
2. Requirement analysis and test-case generation.
3. Test-case selection using Select All and Clear All.
4. Selective QA-suite persistence.
5. QA-suite version feedback after successful persistence.
6. Automation test-case generation.
7. Executable Playwright code generation.
8. Automation execution service.
9. Execution history persistence.
10. Execution result review.
11. Environment-independent Playwright URL handling.
12. `BASE_URL` injection into generated automation execution.
13. Configurable execution timeout.
14. Configurable automation workspace root.
15. Configurable workspace retention.
16. Controlled environment-variable propagation.
17. Hardened automation execution configuration.
18. Focused and full regression coverage for the completed implementation.
## Current implementation boundary
The current checkpoint covers Bedrock test-case generation output handling and the ongoing QA-suite completeness verification.
Do not redesign the existing UI.
Do not rebuild completed functionality.
Do not introduce unrelated features.
All future implementation must preserve the existing architecture, tests, configuration approach, and README continuity rules.
## Next implementation
**P2-S9.12 continuation — Verify QA Suite Completeness and Define QA Agent `skills.md` Contract**
The next step is to:
1. Restart the QA-MCP runtime and regenerate the Customer Portal QA suite.
2. Verify whether Bedrock now returns multiple test cases after increasing `maxTokens` to `12000`.
3. Verify that positive, negative, and edge scenarios are represented in the generated suite.
4. Confirm that the UI displays the complete generated collection without redesigning the existing UI.
5. If the runtime still returns only one or an incomplete subset, implement scenario-batched generation.
6. Add minimum-count and scenario-coverage validation without introducing brittle exact-count assumptions.
7. Define and implement the QA Agent `skills.md` contract after the generation flow is reliable.
8. Add focused tests before changing the implementation.
9. Run the complete regression suite and verify the MCP/runtime path.
10. Update this README before committing and pushing.
# 2. PRODUCT VISION
The long-term goal is to evolve QA MCP from a collection of QA utilities into an intelligent QA agent/platform.
```text
Requirement
|
v
Requirement Understanding
|
v
Scenario Analysis
|
v
Test Case Generation
|
v
Test Case Review
|
v
QA Suite / Version
|
v
Automation Candidate Selection
|
v
Automation Case
|
v
Automation Validation
|
v
Playwright Code Generation
|
v
Generated Automation Artifact
|
v
Controlled Execution
|
v
Execution Results
|
v
Reporting / Analysis
|
v
QA Agent / Orchestration
```
Eventually the platform should support:
```text
Jira
GitHub
Slack
CI/CD
Test repositories
Automation environments
Cloud execution
Interactive UI
Hosted/cloud product
```
The UI/hosted product layer must be introduced only after the core QA-agent capabilities are sufficiently stable.
---
# 3. DEVELOPMENT RULES — MUST FOLLOW
These rules apply to every future change.
1. Implement one phase/sub-step at a time.
2. Test first wherever practical.
3. Focused tests must pass before moving to the next increment.
4. The relevant feature test group must pass.
5. The full regression suite must pass before closing a milestone.
6. Never weaken or delete tests merely to obtain green output.
7. Inspect existing code before modifying it.
8. Preserve the layered architecture.
9. Core business logic must remain independent of MCP transport.
10. Persistence must remain behind repository interfaces.
11. External integrations must remain mockable.
12. LLM providers must remain replaceable.
13. AI output must be validated before downstream use.
14. Never commit secrets or a real `.env` file.
15. Never delete persistent databases merely to make tests pass.
16. Keep unrelated refactoring separate from feature work.
17. A major capability is not complete until its MCP/runtime path is verified.
18. Update this README at every verified milestone.
19. Commit only after feature, tests, README, and checkpoint have been reviewed.
20. Do not recreate completed work from earlier milestones.
21. Do not introduce production-grade container/cloud complexity before the local execution contract is stable.
22. Keep generated automation execution behind explicit framework validation and controlled command construction.
23. Preserve traceability:
`Requirement → Test Case → Automation Case → Artifact → Execution Result`.
24. Do not silently change established contracts.
25. Prefer deterministic behavior over clever behavior.
26. Keep execution safety ahead of execution convenience.
27. Deployment/configuration details must remain documented here.
28. A new chat/session must begin from this README and the current GitHub `main` branch.
## Mandatory development sequence
```text
Read README / current checkpoint
|
v
Inspect GitHub main + repository state
|
v
Inspect existing implementation
|
v
Define ONE next sub-step
|
v
Write/update focused tests
|
v
Implement smallest production change
|
v
Focused tests green
|
v
Feature tests green
|
v
Full regression green
|
v
Runtime/MCP verification
|
v
Update README
|
v
git diff --check
|
v
Commit + push
|
v
Verify clean working tree
```
---
# 4. ARCHITECTURE
```text
MCP CLIENT / AI ASSISTANT
|
v
QA MCP Server
|
v
MCP Tool Layer
|
+-----------------------+------------------------+
| | |
v v v
QA Workflows Core Services Connectors
| | +--------+--------+
| | | | |
v v Jira GitHub Slack
Requirement Analyzer Automation | | |
Test Case Generator Execution v v v
Test Case Reviewer Versioning Service Service Service
QA Suite Workflow Project Context | | |
Candidate Selection v v v
Automation Generation Client Client Client
Automation Execution / \ / \ / \
Mock Mock Mock
Cloud Cloud Cloud
```
Layer responsibilities:
```text
models/
Domain and data contracts
core/
Business/application services
Factories
Orchestration boundaries
Automation execution mechanics
infrastructure/
Persistence
External clients
Concrete implementations
tools/
QA-oriented application workflows
server.py
MCP transport and tool registration
```
Core business logic must not become coupled to MCP transport.
---
# 5. REPOSITORY STRUCTURE
Important current structure:
```text
qa-mcp/
|
+-- config/
| +-- settings.yaml
|
+-- src/qa_mcp/
| +-- core/
| | +-- automation/
| | | +-- candidate_generation_service.py
| | | +-- candidate_selector.py
| | | +-- candidate_service.py
| | | +-- code_generation_service.py
| | | +-- execution_config.py
| | | +-- execution_runner.py
| | | +-- execution_service.py
| | | +-- workspace.py
| | | +-- service.py
| | | +-- validator.py
| | |
| | +-- github/
| | +-- jira/
| | +-- slack/
| | +-- import_export/
| | +-- project/
| | +-- versioning/
| | +-- config.py
| | +-- llm.py
| |
| +-- infrastructure/
| | +-- github/
| | +-- jira/
| | +-- slack/
| | +-- versioning/
| | +-- project repositories
| |
| +-- models/
| | +-- schemas.py
| |
| +-- tools/
| | +-- automation/
| | +-- requirement/
| | +-- testcase/
| | +-- workflow/
| |
| +-- server.py
|
+-- tests/
+-- data/
+-- README.md
+-- requirements.txt
```
---
# 6. COMPLETED PRODUCT CAPABILITIES
## Phase 1 — Foundation & QA Intelligence
**STATUS: COMPLETE**
Completed:
- MCP server foundation
- Configuration loading
- LLM abstraction
- Mock LLM support
- Requirement analysis
- Test-case generation
- Test-case review
- End-to-end QA suite workflow
Core MCP capabilities:
```text
health
test_llm
analyze_requirement
generate_test_cases
review_test_cases
generate_qa_suite
```
## Phase 2 — QA Platform Foundation
| Milestone | Capability | Status |
|---|---|---|
| P2-S1 | QA Project Context | COMPLETE |
| P2-S2 | SQLite Persistence | COMPLETE |
| P2-S3 | Requirement & Suite Versioning | COMPLETE |
| P2-S4 | Project Import / Export | COMPLETE |
| P2-S5 | Jira Connector | COMPLETE |
| P2-S6 | GitHub Connector | COMPLETE |
| P2-S8 | Automation Pipeline | COMPLETE through current checkpoints |
| P2-S8.6 | Automation Candidate Selection | COMPLETE |
| P2-S8.7 | Candidate → Automation Generation | COMPLETE |
| P2-S8.8 | Automation Case Validation | COMPLETE |
| P2-S8.8+ | Automation Code Generation | COMPLETE |
| P2-S8.9 | Controlled Automation Execution | COMPLETE |
Slack integration exists behind service/client abstractions.
---
# 7. PROJECT CONTEXT AND PERSISTENCE
Conceptually:
```text
QAProject
|
+-- project_id
+-- name
+-- description
+-- application
+-- environment
+-- metadata
+-- requirements
+-- test suites
```
Persistence:
```text
ProjectContext
|
v
ProjectRepository
|
v
SQLiteProjectRepository
|
v
SQLite
```
Database:
```text
data/qa_mcp.db
```
**Important:** Never delete the persistent database merely to make tests pass.
Persistence-focused tests should use isolated database state.
---
# 8. EXTERNAL CONNECTORS
## Jira
Abstraction:
```text
MCP
|
v
JiraService
|
v
JiraClient
+-- MockJiraClient
+-- JiraCloudClient
```
Current real operations are read-only:
```text
get_jira_issue(issue_key)
search_jira_issues(jql, max_results=50)
```
No Jira write operations are part of the completed connector milestone.
## GitHub
Abstraction:
```text
MCP
|
v
GitHubService
|
v
GitHubClient
+-- MockGitHubClient
+-- GitHubCloudClient
```
Current read-only tools:
```text
get_github_repository(owner, repository)
get_github_issue(owner, repository, issue_number)
get_github_pull_request(owner, repository, pull_number)
search_github_issues(query, max_results=50)
```
No GitHub write operations are part of the completed connector milestone.
## Slack
Abstraction:
```text
SlackService
|
v
SlackClient
+-- MockSlackClient
+-- SlackCloudClient
```
Current tools include:
```text
get_slack_channel
get_slack_messages
search_slack_messages
get_slack_thread
```
---
# 9. AUTOMATION PIPELINE
```text
Test Cases
|
v
Automation Candidate Selection
|
v
Automation Case
|
v
Automation Validation
|
v
Framework-specific Code Generation
|
v
GeneratedAutomationArtifact
|
v
Controlled Execution
|
v
AutomationExecutionResult
```
Candidate selection deliberately distinguishes:
```text
Recommended for automation
|
+---- Automated
|
+---- Manual-only
```
Manual-only test cases must not be sent to the automation generator.
---
# 10. AUTOMATION CHECKPOINTS ALREADY COMPLETE
## P2-S8.6 — Candidate Selection
`AutomationCandidateSelector` / `AutomationCandidateService`
Result:
```text
AutomationCandidateResult
+-- candidate_ids
+-- manual_ids
+-- total
```
MCP tool:
```text
select_automation_candidates
```
## P2-S8.7 — Candidate → Automation Generation
Service:
```text
AutomationCandidateGenerationService
```
Flow:
```text
TestCase[]
|
v
Candidate Selection
|
v
candidate_ids
|
v
Generate automation ONLY for candidates
|
v
AutomationCase[]
```
Zero-candidate behavior:
```text
No automation candidates
|
v
[]
|
v
Automation generator is NOT called
```
MCP tool:
```text
generate_automation_for_candidates
```
## P2-S8.8 — Automation Case Validation
Validator:
```text
AutomationValidator
```
Result:
```text
AutomationValidationResult
+-- automation_case_id
+-- test_case_id
+-- valid
+-- errors
+-- warnings
```
Minimum integrity:
- At least one automation step.
- Validation failures are structured errors.
- Non-blocking concerns can be warnings.
- Validation remains separate from generation.
## P2-S8.8+ — Automation Code Generation
Artifact:
```text
GeneratedAutomationArtifact
+-- id
+-- automation_case_id
+-- framework
+-- language
+-- file_name
+-- code
```
Current execution target:
```text
Framework: Playwright
Language: Python
```
Generated automation must be validated before downstream execution.
---
# 11. CONTROLLED AUTOMATION EXECUTION — P2-S8.9 COMPLETE
The committed local execution pipeline is:
```text
GeneratedAutomationArtifact
|
v
AutomationExecutionConfig
|
v
AutomationWorkspace
|
v
AutomationExecutionRunner
|
v
AutomationExecutionService
|
v
AutomationExecutionResult
|
v
MCP execute_automation_code
```
## Execution configuration
```text
AutomationExecutionConfig
+-- timeout_seconds = 60
+-- workspace_root = optional
```
The configuration is immutable.
## Automation workspace
`AutomationWorkspace` creates an isolated temporary directory for the generated artifact.
The workspace is cleaned up after execution unless explicit retention is requested.
The project working tree must not be used as the normal generated-artifact execution directory.
## Controlled subprocess runner
`AutomationExecutionRunner`:
- accepts an explicit command list
- runs from a supplied working directory
- captures stdout
- captures stderr
- captures exit code
- measures execution duration
- enforces a timeout
- reports timeout separately
- reports operating-system execution errors separately
The runner is injectable so tests do not need to execute real automation processes.
## Execution service
Current validation:
```text
Empty code
-> ValueError
Missing framework
-> ValueError
Unsupported framework
-> ValueError
```
Current supported framework:
```text
Playwright
```
Current Python execution command:
```text
python -m pytest <generated_file_name>
```
Status mapping:
```text
exit_code == 0
-> PASSED
exit_code != 0
-> FAILED
timed_out
-> TIMEOUT
runner error
-> ERROR
```
Separation:
```text
Runner
= process mechanics
ExecutionService
= QA execution semantics
AutomationExecutionResult
= stable domain contract
```
Execution IDs are currently deterministic in the service foundation (`EX001`). Durable unique execution IDs belong to the future execution-history/persistence layer.
---
# 12. EXECUTION SAFETY REQUIREMENTS
The current subprocess runner is a controlled local execution boundary, **not** the final production-grade sandbox.
Intended progression:
```text
Current
Local controlled subprocess
|
v
Hardened execution boundary
|
v
Container / isolated execution
|
v
Cloud or CI execution
```
Mandatory safety direction:
- Do not introduce arbitrary shell execution.
- Do not construct unrestricted commands from user input.
- Keep framework support explicit.
- Keep generated filenames and execution paths controlled.
- Keep execution bounded by timeouts.
- Preserve workspace isolation.
- Keep the runner injectable and testable.
- Introduce containerization before exposing execution to untrusted production workloads.
Do not add container/cloud complexity before the local execution contract and orchestration behavior are stable.
---
# 13. MCP AUTOMATION SURFACE
Current automation-related MCP tools:
```text
generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code
```
`execute_automation_code(artifact)`:
1. Validates the incoming artifact through `GeneratedAutomationArtifact`.
2. Delegates to `AutomationExecutionService`.
3. Returns `AutomationExecutionResult.model_dump()`.
4. Converts invalid execution-artifact input into a controlled MCP-facing error.
The MCP layer must not contain subprocess implementation details.
---
# 14. TEST STRATEGY AND CURRENT BASELINE
Test-first development remains mandatory.
Expected sequence:
```text
Write failing test
|
v
Implement smallest production change
|
v
Focused test
|
v
Related tests
|
v
Full regression
|
v
Runtime/MCP verification
|
v
README update
|
v
git diff --check
|
v
Commit + push
```
Current verified baseline before P2-S9.1.a:
```text
pytest -q
190 passed
7 warnings
0 failures
```
P2-S9.1.a verified regression:
```text
pytest -q
204 passed
7 warnings
0 failures
```
P2-S9.1.a focused execution suite:
```text
16 passed
1 warning
```
Execution service suite:
```text
7 passed
```
No test was removed or weakened to obtain the current green baseline.
---
# 15. KNOWN WARNINGS / TECHNICAL DEBT
## Pytest collection warnings
Pydantic models named:
```text
TestCase
TestCaseReview
```
can be interpreted by pytest as possible test classes, producing `PytestCollectionWarning`.
These are non-functional warnings.
Future cleanup may use test-only import aliases. Keep this separate from feature work.
## Pydantic settings warning
Existing:
```text
IncompleteFieldDefinitionWarning
```
related to the `lifespan` forward reference in `pydantic_settings`.
It does not currently cause test failures.
Keep this as separate technical debt unless it blocks development.
---
# 16. ENVIRONMENT / .ENV DOCUMENTATION
## Critical rule
The following is the **documented `.env` template currently used by the development setup**.
**These are placeholders, not real credentials.**
Never commit a real `.env` file, API token, password, or secret to Git.
The actual local `.env` remains developer-machine configuration.
## Current `.env` template
```dotenv
JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token
GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org
# ---------------------------------------------------------
# Slack
# ---------------------------------------------------------
SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=
```
## Variable purpose
| Variable | Purpose | Secret? |
|---|---|---|
| `JIRA_URL` | Jira Cloud base URL | No |
| `JIRA_EMAIL` | Jira API account email | No, but treat as configuration |
| `JIRA_API_TOKEN` | Jira API authentication | **YES** |
| `GITHUB_URL` | GitHub API base URL | No |
| `GITHUB_TOKEN` | GitHub API authentication | **YES** |
| `GITHUB_OWNER` | GitHub username/org used by configuration | No |
| `SLACK_URL` | Slack API base URL | No |
| `SLACK_TOKEN` | Slack API authentication | **YES** |
| `SLACK_DEFAULT_CHANNEL` | Default Slack channel configuration | No |
## Deployment/configuration rule
When configuring a new environment:
1. Copy the documented template into a local `.env`.
2. Replace only the placeholder values required for that environment.
3. Never paste real secrets into this README.
4. Never commit the populated `.env`.
5. Verify `.gitignore` protects `.env`.
6. Keep configuration changes documented here when they materially affect deployment.
7. If new environment variables are introduced, update this section in the same development checkpoint.
---
# 17. CONFIGURATION AND ENVIRONMENT CONTRACT
The primary application configuration file is:
`config/settings.yaml`
The application also supports environment-specific configuration through environment variables.
Environment variables take precedence over default configuration values where supported by the implementation.
## Application configuration
The configuration file contains settings for:
- Application metadata
- LLM provider and model configuration
- Feature flags
- Automation execution
- Jira
- GitHub
- Slack
## Automation environment variables
| Variable | Purpose |
|---|---|
| `DEFAULT_TEST_ENV` | Default execution environment when no environment is explicitly selected |
| `QA_BASE_URL` | Base URL for the QA environment |
| `STAGE_BASE_URL` | Base URL for the staging environment |
| `PROD_BASE_URL` | Base URL for the production environment |
| `QA_AUTOMATION_TIMEOUT_SECONDS` | Automation execution timeout in seconds |
| `QA_AUTOMATION_WORKSPACE_ROOT` | Root directory used for automation execution workspaces |
| `QA_AUTOMATION_KEEP_WORKSPACE` | Controls whether automation workspaces are retained after execution |
## Integration environment variables
| Variable | Purpose | Secret? |
|---|---|---|
| `JIRA_URL` | Jira Cloud base URL | No |
| `JIRA_EMAIL` | Jira API account email | No, but treat as configuration |
| `JIRA_API_TOKEN` | Jira API authentication | Yes |
| `GITHUB_URL` | GitHub API base URL | No |
| `GITHUB_TOKEN` | GitHub API authentication | Yes |
| `GITHUB_OWNER` | GitHub username or organization | No |
| `SLACK_URL` | Slack API base URL | No |
| `SLACK_TOKEN` | Slack API authentication | Yes |
| `SLACK_DEFAULT_CHANNEL` | Default Slack channel | No |
## Example local configuration
The following values are examples only:
~~~yaml
application:
name: qa-mcp
environment: local
llm:
provider: mock
region: ""
model_id: ""
features:
requirement_analyzer: true
testcase_generator: true
testcase_reviewer: true
jira_connector: false
github_connector: false
slack_connector: false
automation_generator: false
automation_execution:
base_url: ""
timeout_seconds: 60
workspace_root: ""
keep_workspace: false
jira:
url: ""
email: ""
api_token: ""
github:
url: "https://api.github.com"
token: ""
owner: ""
slack:
url: "https://slack.com/api"
token: ""
default_channel: ""
~~~
## Example `.env` template
These are placeholders only. Never commit real credentials or secrets.
~~~dotenv
JIRA_URL=https://your-company.atlassian.net
JIRA_EMAIL=your-email
JIRA_API_TOKEN=your-token
GITHUB_URL=https://api.github.com
GITHUB_TOKEN=your-github-token
GITHUB_OWNER=your-github-username-or-org
SLACK_URL=https://slack.com/api
SLACK_TOKEN=
SLACK_DEFAULT_CHANNEL=
DEFAULT_TEST_ENV=qa
QA_BASE_URL=
STAGE_BASE_URL=
PROD_BASE_URL=
QA_AUTOMATION_TIMEOUT_SECONDS=60
QA_AUTOMATION_WORKSPACE_ROOT=
QA_AUTOMATION_KEEP_WORKSPACE=false
~~~
## Secret-handling rules
- Never commit real credentials, access tokens, API keys, passwords, or production secrets.
- Use environment variables or a local ignored `.env` file for secrets.
- Never paste populated secret values into this README.
- Verify that `.env` is protected by `.gitignore`.
- Review `git diff` before committing configuration changes.
- Run `git diff --check`.
- Run focused tests.
- Run the full regression suite.
- Update this README with verified results before committing.
# 18. DEVELOPMENT ENVIRONMENT
Python requirement:
```text
Python >= 3.11
```
Current development environment used during the latest verification:
```text
Python 3.12 virtual environment
.venv/
```
Activate:
```bash
source .venv/bin/activate
```
Install project dependencies according to the repository's `requirements.txt`.
Run all tests:
```bash
pytest -q
```
Run a specific test:
```bash
pytest -q tests/<test_file>.py
```
Check MCP tools:
```bash
python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools()])"
```
Check automation tools:
```bash
python -c "from qa_mcp.server import mcp; print([t.name for t in mcp._tool_manager.list_tools() if 'automation' in t.name])"
```
Check formatting issues:
```bash
git diff --check
```
Check repository state:
```bash
git status
```
Review recent commits:
```bash
git log -5 --oneline
```
---
# 19. GIT CHECKPOINT HISTORY
Important checkpoints:
```text
a288569 Initial commit with configured gitignore
1144ddd Resolve README.md merge conflict
882e149 Jira Connector Added
06dbe61 Complete GitHub connector
71c893e Initial commit with Slack Configured
169c1a1 Complete automation case generator
1d2360b Add automation candidate pipeline
a226e9e Add automation case validation
5ced4e3 Add automation code generation
715ad52 Complete P2-S8.8 automation code generation
703cdb1 Complete automation execution foundation
3bdf761 Implement controlled automation execution
add5ba2 Update project continuity roadmap
```
Every completed checkpoint must contain:
```text
Implementation
Tests
README
Verification evidence
Commit
Push
Clean working tree
```
---
# 20. WHAT HAS ALREADY BEEN COMPLETED — DO NOT REBUILD
These capabilities are already implemented/tested and must not be redesigned or recreated as if they were new:
```text
MCP server foundation
Configuration
LLM abstraction
Mock LLM
Requirement analysis
Test case generation
Test case review
QA suite workflow
Project context
SQLite persistence
Requirement/suite versioning
Import/export
Jira connector
GitHub connector
Slack connector
Automation case generation
Automation candidate selection
Candidate → automation orchestration
Automation case validation
Playwright/Python automation code generation
GeneratedAutomationArtifact contract
AutomationExecutionResult contract
AutomationWorkspace
AutomationExecutionRunner
AutomationExecutionService
execute_automation_code MCP boundary
```
Future work must build on these components.
---
# 21. HISTORICAL EXECUTION-HARDENING ROADMAP
This section is retained for historical continuity.
It is not the current development checkpoint.
The current development checkpoint is defined at the beginning of this README under:
**P2-S9.12 — Automation Execution Configuration Hardening**
The earlier P2-S9.1 execution-hardening roadmap has been superseded by the completed implementation and later P2-S9.12 work.
Historical direction included:
- Safe workspace and file handling
- Stronger command validation
- Execution identity
- Configurable limits
- Failure classification
- Artifact and result retention
- Execution evidence
- Execution history
- Reporting and analysis
- Failure analysis
- Web dashboard integration
- Agent orchestration
The completed historical checkpoints below must be preserved as historical records only.
Future work must follow the current checkpoint at the top of this README and must not return to already-completed functionality without a new, explicitly documented requirement.
# 21A. COMPLETED CHECKPOINT — P2-S9.1.a
## Execution Hardening — Safe Workspace/File Handling
**STATUS: COMPLETE**
P2-S9.1.a hardens the generated automation workspace boundary without changing the established execution contracts or MCP execution flow.
Implemented:
- Strict generated artifact filename validation.
- Rejection of empty and whitespace-only filenames.
- Rejection of `.` and `..`.
- Rejection of POSIX absolute paths and traversal paths.
- Rejection of Windows-style traversal and drive-style paths.
- Explicit resolved-path containment verification before writing.
- Filename validation before workspace creation.
- Generated artifacts remain constrained to the controlled workspace.
Tests:
```text
14 new workspace-hardening tests
Focused execution suite: 28 passed
Full regression: 204 passed, 0 failures, 7 warnings
git diff --check: clean
```
The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15 and are intentionally outside this checkpoint.
No existing Pydantic execution contracts were changed.
**Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening**
---
# 21B. COMPLETED CHECKPOINT — P2-S9.1.b.1
## Execution Hardening — Controlled Automation Command Boundary
**STATUS: COMPLETE**
Implemented:
- Explicit controlled command construction in `AutomationExecutionService`.
- Playwright automation is restricted to `python -m pytest <artifact_file_name>`.
- Unsupported automation frameworks are rejected before command construction.
- Unsafe artifact filenames are rejected before command construction.
- `AutomationExecutionRunner` remains a generic subprocess execution wrapper.
- Existing Pydantic execution contracts and the MCP execution boundary remain unchanged.
Tests:
```text
Focused execution service suite: 10 passed
Full regression: 207 passed, 0 failures, 7 warnings
git diff --check: clean
```
The 7 warnings remain the existing non-blocking pytest/Pydantic technical debt documented in Section 15.
**Next implementation: P2-S9.1.b.2 — Further command/execution policy hardening**
---
# 22. FUTURE EXECUTION ARCHITECTURE
Target:
```text
GeneratedAutomationArtifact
|
v
Execution Policy / Safety Validation
|
v
Isolated Execution Environment
|
v
Framework Runner
|
v
Execution Evidence
|
v
Execution Result
|
v
Persistent Execution History
|
v
Reporting / AI Analysis
```
Potential isolation progression:
```text
Local hardened process
|
v
Docker/container
|
v
CI worker
|
v
Cloud execution
```
Do not implement all layers at once.
---
# 23. EVENTUAL AGENT-DRIVEN QA WORKFLOW
The eventual product experience:
```text
Understanding requirement...
|
v
Analyzing scenarios...
|
v
Generating test cases...
|
v
Reviewing coverage...
|
v
Identifying automation candidates...
|
v
Generating automation...
|
v
Validating automation...
|
v
Executing automation...
|
v
Analyzing results...
|
v
Preparing QA report...
```
MCP is intended to become the capability layer underneath an agent-driven QA product.
---
# 24. EVENTUAL PRODUCT / UI DIRECTION
The eventual UI should make the agent's progress, generated artifacts, execution state, and results visible and understandable.
```text
User
|
v
QA MCP UI
|
v
Agent / MCP Orchestration
|
+-- Requirement analysis
+-- Test generation
+-- Test review
+-- Candidate selection
+-- Automation generation
+-- Automation validation
+-- Automation execution
+-- Results / reporting
+-- Jira
+-- GitHub
+-- Slack
```
The UI and hosted product layer should be introduced only after the core QA-agent capabilities are sufficiently stable.
---
# 25. LONG-TERM PRODUCT DIRECTION
The final product should evolve toward:
```text
Understand
|
v
Plan
|
v
Generate
|
v
Validate
|
v
Execute
|
v
Observe
|
v
Analyze
|
v
Report
|
v
Learn / Improve
```
Long-term capabilities:
- Requirements intelligence
- Test design
- Test review
- Automation selection
- Automation generation
- Automation validation
- Safe execution
- Execution evidence
- Failure analysis
- Coverage analysis
- Regression intelligence
- External engineering-system context
- CI/CD integration
- Agent orchestration
- Interactive UI
- Hosted/cloud execution
These are future goals, not permission to prematurely implement everything.
---
# 26. DEVELOPMENT PRINCIPLES
The following principles must remain unchanged:
1. Build incrementally.
2. Write tests before implementation where practical.
3. Keep services small and composable.
4. Keep MCP tools thin.
5. Keep external integrations behind infrastructure abstractions.
6. Avoid destabilizing existing workflows.
7. Preserve structured Pydantic contracts.
8. Keep secrets outside source control.
9. Run full regression before every feature checkpoint.
10. Update this README whenever a meaningful feature checkpoint is committed.
11. Commit code, tests and README together for each completed checkpoint.
12. Prefer explicit contracts over implicit behavior.
13. Prefer deterministic behavior over clever behavior.
14. Keep execution safety ahead of execution convenience.
15. Keep production concerns separated from prototype convenience.
16. Do not duplicate completed capabilities.
17. Do not silently change established contracts.
18. Maintain requirement → test case → automation case → artifact → execution result traceability.
19. Treat deployment/configuration documentation as part of the implementation.
20. Treat this README as the continuity record, not optional documentation.
---
# 27. CURRENT RESUME POINT
## Resume from
**P2-S9.1.b.2 — Further Command/Execution Policy Hardening**
P2-S9.1.a — Safe Workspace/File Handling and P2-S9.1.b.1 — Controlled Automation Command Boundary are complete and must not be recreated.
Previous completed checkpoints:
```text
P2-S8.6 Automation Candidate Selection COMPLETE
P2-S8.7 Candidate → Automation Generation COMPLETE
P2-S8.8 Automation Case Validation COMPLETE
P2-S8.8+ Automation Code Generation COMPLETE
P2-S8.9 Controlled Automation Execution COMPLETE
```
Verified baseline after P2-S9.1.b.1:
```text
207 passed
7 warnings
0 failures
```
Latest repository implementation commit:
```text
b6190d9 Harden automation workspace file handling
```
Current checkpoint commit:
```text
Pending user commit
```
Previous implementation checkpoint:
```text
3bdf761 Implement controlled automation execution
```
Current automation MCP surface:
```text
generate_automation
select_automation_candidates
generate_automation_for_candidates
execute_automation_code
```
---
# 28. CRITICAL CONTINUITY INSTRUCTION FOR A NEW CHAT
A future development session must:
1. Read this README first.
2. Inspect the current GitHub `main` branch:
`https://github.com/sanumenon/qa-mcp/tree/main`
3. Confirm the latest commit and test baseline.
4. Inspect the existing implementation before proposing changes.
5. Start from **P2-S9.1.b.2 — Further Command/Execution Policy Hardening**.
6. Treat **P2-S9.1.a — Safe Workspace/File Handling** as complete.
7. Treat **P2-S9.1.b.1 — Controlled Automation Command Boundary** as complete.
8. Do not recreate candidate selection.
9. Do not recreate automation generation.
10. Do not recreate automation validation.
11. Do not recreate controlled local execution.
12. Do not silently replace established architecture/contracts.
13. Add tests first wherever practical.
14. Keep the architecture layered.
15. Verify focused tests.
16. Verify the full regression suite.
17. Verify the MCP/runtime path for major capabilities.
18. Update this README at the end of every verified checkpoint.
19. Include deployment/configuration changes in this README.
20. Never commit real secrets or a populated `.env`.
21. Commit and push code + tests + README together.
22. Verify the working tree is clean after the checkpoint.
23. Never make the user repeat already-completed development work when the repository and README contain it.
24. Never use a new chat as a reason to restart the project from an earlier phase.
**This README is part of the implementation and must be treated as the project's authoritative continuity record.**
---
## P2-S9.2 — Executable Playwright Code Generation
Status: COMPLETE
Implementation commit:
```text
176ab59 Implement executable Playwright code generation
```
Implemented in:
```text
src/qa_mcp/core/automation/code_generation_service.py
```
Generated Playwright/Python artifacts now translate the controlled automation DSL into executable Playwright code.
Supported automation steps:
```text
goto: <url>
fill: <selector> = <value>
click: <selector>
press: <selector> = <key>
```
Supported assertions:
```text
visible: <selector>
text: <selector> = <expected text>
url: <expected url>
```
The generator now:
- Produces executable Playwright/Python code instead of comments.
- Generates `Page` and `expect` based Playwright code.
- Rejects unsupported automation steps.
- Rejects unsupported automation assertions.
- Validates malformed step/assertion expressions.
- Preserves the existing `GeneratedAutomationArtifact` contract.
- Keeps the existing `execute_automation_code` MCP boundary unchanged.
- Supports generated-artifact → execution-service integration.
Verification:
```text
pytest -q tests/test_automation_code_generation_service.py tests/test_automation_code_generation_empty.py tests/test_automation_code_generation_result.py
8 passed
0 failures
pytest -q tests/test_automation_execution_service.py
17 passed
0 failures
pytest -q
216 passed
7 warnings
0 failures
git diff --check
clean
```
The 7 pytest/Pydantic warnings are existing non-blocking technical debt and are not part of P2-S9.2.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
```
### Next implementation checkpoint
```text
P2-S9.3 — Real Playwright Execution Validation
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, or controlled execution functionality.
---
---
## P2-S9.3 — Execution History and Persistence
Status: COMPLETE
Implementation commit:
```text
7b8a67b Add automation execution history
```
Verification:
```text
Focused execution/history suite: 26 passed
Full regression suite: 227 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Unique execution IDs are generated for every automation execution.
- Automation execution results are persisted in SQLite.
- Execution history can be retrieved by execution ID.
- Execution history can be listed with optional automation-case filtering and result limits.
- `execute_automation_code` now persists execution results.
- Added MCP tool: `get_automation_execution`.
- Added MCP tool: `list_automation_executions`.
- Added repository, application-service, execution-service, and MCP-tool test coverage.
- Existing execution behavior and contracts remain intact.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Reporting / Analysis
```
### Next implementation checkpoint
```text
P2-S9.4 — Reporting / Analysis
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, or execution-history functionality.
---
## P2-S9.4 — Reporting / Analysis
Status: COMPLETE
Implementation commit: 55ddd09 Add automation execution reporting
Verification:
```text
Focused P2-S9.4 tests: 12 passed
Full regression suite: 234 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added aggregated automation execution reporting.
- Added execution totals by result status.
- Added pass-rate calculation.
- Added total and average execution duration metrics.
- Added latest execution identification and status.
- Added optional automation-case filtering.
- Added persistent SQLite-backed reporting.
- Added reporting application-service support.
- Added MCP reporting capability.
- Added repository, service, and MCP-tool test coverage.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
```
### Next implementation checkpoint
```text
P2-S9.5 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, or reporting functionality.
---
## P2-S9.5 — Failure Analysis
Status: COMPLETE
Implementation commit: 77a8d9b Add automation execution failure analysis
Verification:
```text
Focused P2-S9.5 tests: 15 passed
Full regression suite: 242 passed
Warnings: 7 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added structured automation execution failure analysis.
- Added failure-analysis models and application-service support.
- Added persisted failure-analysis retrieval from execution history.
- Added analysis of failed and errored executions.
- Added execution identifiers, automation artifact identifiers, and automation case traceability.
- Added MCP failure-analysis capability.
- Added repository, service, and MCP-tool test coverage.
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
```
### Next implementation checkpoint
```text
P2-S9.6 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, or failure-analysis functionality.
---
## P2-S9.6 — Web Dashboard
Status: COMPLETE
Implementation commit: 79e3c3f Add web dashboard
Verification:
```text
Focused Web Dashboard tests: 5 passed
Full regression suite: 247 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Implementation delivered
- Added FastAPI-based web application.
- Added browser-accessible QA automation dashboard.
- Added execution reporting view.
- Added execution history view.
- Added failure-analysis view.
- Added REST endpoints for execution reporting, execution history, and failure analysis.
- Added dedicated web entrypoint through `run_web.py`.
- Added web dashboard automated test coverage.
- Added FastAPI to `requirements.txt`.
### Start the Web Dashboard
From the project root with the virtual environment activated:
```bash
cd ~/pythonPrograms/qa-mcp
source .venv/bin/activate
python run_web.py
```
The dashboard is then available at:
```text
http://127.0.0.1:8000
```
### Web API endpoints
```text
GET /
GET /api/executions/report
GET /api/executions?limit=20
GET /api/executions/failures?limit=20
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
```
### Next implementation checkpoint
```text
P2-S9.7 — Next functional capability
```
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, or web-dashboard functionality.
---
## P2-S9.7 — AI QA Workspace
Status: COMPLETE
Implementation scope:
- Added a browser-based AI QA Workspace to the existing web dashboard.
- Added persistent QA project creation and retrieval.
- Added project-aware requirement analysis.
- Added project-aware QA test-suite generation.
- Added AI-generated test-case review.
- Added requirement version persistence.
- Added QA-suite version persistence.
- Added workspace REST APIs.
- Added deterministic MockLLM support for workspace development and tests.
- Added automated coverage for project, workspace service, and MockLLM behavior.
- Preserved all existing execution, reporting, failure-analysis, and dashboard functionality.
### AI QA Workspace flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### Workspace API endpoints
```text
POST /api/projects
GET /api/projects/{project_id}
POST /api/projects/{project_id}/qa-suite
```
### Workspace behavior
A QA project must exist before a QA suite can be generated for that project.
Existing projects can be retrieved using:
```text
GET /api/projects/{project_id}
```
A requirement can then be submitted using:
```text
POST /api/projects/{project_id}/qa-suite
```
The generated response contains:
```text
project
requirement_version
suite_version
requirement
analysis
test_cases
review
```
Requirement versions and QA-suite versions are persisted independently so that generated QA work remains traceable to the project and requirement history.
### Verification
```text
Focused S9.7 workspace tests: 19 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
```
### P2-S9.8 — Automation Candidate Selection
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing Automation Candidate Service.
- Added automation candidate selection to QA-suite generation.
- Added automation candidate results to the generated QA-suite response.
- Preserved manual-test identification through the existing candidate-selection workflow.
- Added focused regression coverage for automation candidate selection.
- Preserved all existing QA Workspace, execution, reporting, failure-analysis, and dashboard functionality.
### Automation Candidate flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### QA-suite response
The generated QA-suite response now includes:
```text
automation_candidates
candidate_ids
manual_ids
total
```
Candidate selection is performed through the existing:
```text
AutomationCandidateService
↓
AutomationCandidateSelector
```
### Verification
```text
Focused S9.8 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
```
### P2-S9.9 — AI QA Workspace Automation Generation
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing automation candidate generation service.
- Added dependency injection for automation candidate generation.
- Added automation case generation during QA-suite creation.
- Added generated automation cases to the QA-suite response.
- Preserved the existing automation candidate selection flow.
- Added focused regression coverage for automation generation.
- Preserved all existing QA Workspace, candidate selection, execution, reporting, failure-analysis, and dashboard functionality.
### AI QA Workspace automation flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Case Generation
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### QA-suite automation response
The generated QA-suite response now includes:
```text
automation_candidates
candidate_ids
manual_ids
total
automation_cases
test_case_id
automation_type
```
Automation generation is performed through the existing service boundary:
```text
AutomationCandidateGenerationService
↓
Generated Automation Cases
```
### Verification
```text
Focused S9.9 workspace-service tests: 2 passed
Full regression suite: 261 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
↓
Automation Case Generation
```
### P2-S9.10 — AI QA Workspace Automation Wiring
Status: COMPLETE
Implementation scope:
- Connected the production web application to the existing automation-generation pipeline.
- Added the workspace automation case generator using the workspace LLM.
- Added the workspace automation service.
- Added the existing automation candidate service and selector to the production workspace.
- Added the automation candidate generation service to the AI QA Workspace.
- Connected `QAWorkspaceService` to the production automation candidate generation service.
- Added focused regression coverage confirming the production dependency is injected correctly.
- Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.
### Automation generation flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
↓
Requirement Version
↓
QA Suite Version
↓
Persisted QA Workspace Result
```
### Production wiring
```text
Workspace LLM
↓
AutomationCaseGenerator
↓
AutomationService
↓
AutomationCandidateGenerationService
↑
AutomationCandidateService
↑
AutomationCandidateSelector
↓
QAWorkspaceService
↓
AI QA Workspace API
```
The web application now constructs the real automation-generation dependencies instead of relying on test-only or implicit service construction.
### Verification
```text
Focused S9.10 workspace-service tests: 3 passed
Full regression suite: 262 passed
Warnings: 8 existing non-blocking warnings
Failures: 0
git diff --check: clean
```
### Current implementation state
```text
Requirement/Test Case
↓
Automation Candidate
↓
Automation Case
↓
Generated Playwright/Python Artifact
↓
Controlled Automation Execution
↓
Execution Result
↓
Persistent Execution History
↓
Execution Reporting / Analysis
↓
Failure Analysis
↓
Web Dashboard
↓
AI QA Workspace
↓
Project-aware Requirement Analysis
↓
Versioned QA Suite
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
```
#
## P2-S9.12 — Project Workspace Execution History and Result Review
Implemented:
- Added project-scoped execution history listing.
- Added project-scoped execution lookup.
- Added project-scoped execution reporting.
- Added project-scoped failure analysis.
- Added artifact ownership filtering through the project artifact repository.
- Added API coverage for project execution history and result review.
Verification:
- Focused tests: 4 passed.
- Full regression: 310 passed, 8 warnings.
- git diff --check: passed.
Next implementation:
- Audit remaining backend-to-UI integration.
- Define and implement the QA Agent `skills.md` capability.
## P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
Status: IN PROGRESS
Current implementation state:
- Persistent Project QA Workspace is implemented.
- Persisted project requirements, requirement versions, saved QA suite versions, test cases, automation candidates, and generated automation artifacts are exposed through the workspace.
- Automation candidates are now actionable from the Project QA Workspace.
- Added `QAProjectAutomationGenerationRequest` for project-level automation generation requests.
- Added `QAWorkspaceService.generate_automation_from_project()` to continue from persisted test cases.
- Persisted test-case dictionaries are reconstructed through the existing `TestCase` model before candidate processing.
- Requested test-case IDs are validated against the persisted project workspace.
- Selected test cases are rechecked through the existing `AutomationCandidateService`.
- Non-candidate selections are rejected instead of bypassing the existing candidate policy.
- Existing `AutomationCandidateGenerationService` is reused to generate automation cases.
- Existing `AutomationCodeGenerationService` is reused to generate automation artifacts.
- Added `POST /api/projects/{project_id}/automation`.
- Added Project QA Workspace UI controls for selecting automation candidates.
- Added a Generate Automation action in the Project QA Workspace.
- Browser-side UI wiring invokes the new project automation endpoint and refreshes the persisted workspace state.
- Existing Generate QA Suite functionality remains separate and unchanged.
- Added dashboard/API regression coverage for the new workflow.
- Focused workspace/dashboard tests: 34 passed.
- Full regression suite: 302 passed, 8 known non-blocking warnings, 0 failures.
- `git diff --check` is clean.
P2-S9.12 product flow:
Project
-> Requirement
-> Requirement Analysis
-> Test Case Generation
-> AI Review
-> Preview
-> User selects test cases
-> Save
-> QASuiteVersioningService
-> SQLite qa_suite_versions
-> Project QA Workspace
-> Existing Requirements
-> Requirement Versions
-> Saved Suite Versions
-> Existing Test Cases
-> Automation Candidates
-> Select Automation Candidates
-> Generate Automation
-> Automation Case Generation
-> Automation Artifact Generation
-> Controlled Automation Execution
-> Execution History / Reporting
-> Failure Analysis
Implemented user capability:
- View requirements already provided or prepared for the project.
- View requirement analysis and version information already prepared.
- View test cases already generated and persisted for the project.
- View automation candidates derived from persisted test cases.
- Select one or more persisted automation candidates.
- Generate automation from the selected candidates.
- Generate automation artifacts through the existing automation code-generation pipeline.
- Keep generated automation associated with the project workspace.
- Preserve the existing active Generate QA Suite workflow independently.
Remaining P2-S9.12 scope:
Project QA Workspace
-> View persisted test cases
-> View automation candidates
-> Select automation candidates
-> Generate/use automation cases
-> Validate automation cases
-> Generate automation artifacts
-> Controlled automation execution
-> Execution result/history/reporting
-> Failure analysis
Completed automation candidate selection, candidate generation, automation case generation, validation, Playwright code generation, artifact generation, command-boundary enforcement, execution configuration, controlled execution, execution history, reporting, and failure-analysis services must continue to be reused rather than rebuilt.
Current checkpoint:
P2-S9.12 — Controlled Automation Execution from Project QA Workspace
The Project QA Workspace now supports controlled execution of persisted automation artifacts. Execution is scoped to the selected project, reuses the existing controlled execution pipeline, and persists results through the existing execution-history service.
Implemented in this continuation:
- Added project-scoped persisted automation-artifact lookup.
- Added controlled execution through the existing `AutomationExecutionService`.
- Added execution-result persistence through `AutomationExecutionHistoryService`.
- Added `POST /api/projects/{project_id}/automation/{artifact_id}/execute`.
- Added Execute actions for generated automation artifacts in the Project QA Workspace.
- Added workspace feedback for execution status, stdout, stderr, and errors.
- Preserved existing automation generation, validation, artifact generation, command-boundary enforcement, execution configuration, reporting, and failure-analysis functionality.
Validation completed:
```text
Python compilation: passed
Full regression suite: 306 passed
Warnings: 8 known non-blocking warnings
Failures: 0
git diff --check: clean
```
The Project QA Workspace QA-suite generation flow has been hardened to require a complete test-case suite covering the supplied positive scenarios, negative scenarios, and edge cases. The generator now rejects malformed or incomplete LLM payloads instead of silently normalizing them into a single test case.
The LLM generation boundary now distinguishes unusable provider output from application validation failures. In particular, a provider/guardrail refusal is surfaced as `LLMGenerationError` and returned by the QA-suite API as HTTP 502 rather than being incorrectly classified as HTTP 404.
During validation, the configured Bedrock provider returned the following non-JSON response:
`The response was blocked by dev-guardrails policy. If this looks like a false positive, ping #ai-guardrails.`
This confirmed that the observed single-test-case UI symptom was not caused by application-side test-case truncation. The provider response was blocked before a test-case suite could be produced.
Validation completed:
```text
Focused generator/dashboard tests: 30 passed
Full regression suite: 304 passed
Warnings: 8 known non-blocking warnings
Failures: 0
git diff --check: clean
```
The provider-failure diagnostic path now preserves the raw LLM provider response on `LLMGenerationError.provider_response` and logs that response at error level for IT troubleshooting. The API continues to return only the safe generic HTTP 502 message and does not expose the provider response to the end user.
The diagnostic regression test verifies that the exact provider response is retained. Focused diagnostic validation passed with 39 tests, and the full regression suite remains at 304 passed with 8 known non-blocking warnings and 0 failures.
The implementation is intentionally limited to prompt hardening, strict test-case response validation, LLM-generation error classification, API error mapping, and regression coverage. Do not bypass or weaken provider/dev-guardrail policy as part of this fix.
Next implementation:
P2-S9.12 continuation — Project Workspace Execution History and Result Review
The next implementation should expose persisted execution history and result review from the Project QA Workspace by reusing the existing execution-history, reporting, and failure-analysis services.
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, automation artifact generation, or the QA-suite generation/error-boundary hardening completed in this checkpoint.
### P2-S9.11 — AI QA Workspace Artifact Generation
Status: COMPLETE
Implementation scope:
- Connected the AI QA Workspace to the existing automation code-generation service.
- Added production construction of `AutomationCodeGenerationService`.
- Connected the service to `QAWorkspaceService` through dependency injection.
- Added automation artifact generation for generated automation cases.
- Added `automation_artifacts` to the QA-suite workspace response.
- Added persistent SQLite repository support for QA Workspace automation artifacts.
- Added focused regression coverage for artifact persistence, artifact generation, and dependency injection.
- Added browser-level Playwright regression coverage for the AI QA Workspace dashboard.
- Verified the rendered dashboard preserves the AI QA Workspace and existing Automation Execution Overview.
- Preserved all existing QA Workspace, candidate selection, automation generation, execution, reporting, failure-analysis, and dashboard functionality.
### Automation artifact flow
```text
QA Project
↓
Requirement
↓
Requirement Analysis
↓
Test Case Generation
↓
AI Test Case Review
↓
Automation Candidate Selection
↓
Automation Candidate Generation
↓
Automation Case Generation
↓
Automation Code Generation
↓
Automation Artifact
↓
Persistent Artifact Repository
↓
QA Workspace Response
```
### Workspace response
The generated QA-suite response now includes:
```text
automation_candidates
automation_cases
automation_artifacts
requirement_version
suite_version
```
### Production wiring
```text
Workspace LLM
↓
AutomationCaseGenerator
↓
AutomationService
↓
AutomationCandidateGenerationService
↓
AutomationCase
↓
AutomationCodeGenerationService
↓
AutomationArtifact
↓
QAWorkspaceService
↓
AI QA Workspace API
```
### Dashboard regression protection
The AI QA Workspace is protected by both structural and browser-level regression tests.
The dashboard regression coverage verifies:
```text
AI QA Workspace
↓
Create QA Project controls
↓
Generate QA Suite controls
↓
Project / requirement inputs
↓
JavaScript action wiring
↓
Workspace API wiring
↓
Rendered dashboard
↓
Existing Automation Execution Overview preserved
```
The browser regression test uses Playwright with Chromium against a live Uvicorn instance and verifies the rendered dashboard and the important workspace controls.
The browser regression test is now part of the permanent dashboard regression suite and must remain green during future UI changes.
### Verification
```text
Focused dashboard/workspace tests: 12 passed
Browser regression test: 1 passed, 11 deselected
Full regression suite: 269 passed
Warnings: 8 known non-blocking warnings
Failures: 0
git diff --check: clean
```
**P2-S9.11 is complete. Do not recreate or redesign this capability.**
### Next implementation checkpoint
```text
P2-S9.12 — Test Case Persistence and Automation Candidate Workflow
```
P2-S9.12 has progressed from the persistent Project QA Workspace to actionable automation-candidate selection and generation. The workspace can now select persisted automation candidates and invoke the existing automation generation and artifact-generation pipeline. This implementation is committed as `e186391`. The remaining work is to continue from generated automation artifacts into the completed controlled execution, execution-history, reporting, and failure-analysis pipeline.
Do not redesign or rebuild completed automation generation, validation, artifact generation, workspace handling, command boundary, execution configuration, controlled execution, execution history, reporting, failure analysis, web-dashboard functionality, AI QA Workspace functionality, automation candidate selection functionality, automation case generation functionality, production automation-generation wiring, or automation artifact generation.
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues