# Website QA Agent

Standalone Jira-to-Playwright QA agent for testing any website URL, then posting results to Jira, Slack, and GitHub when those integrations are configured. External integrations run through a local MCP gateway with allowlisted operations and redacted SQLite audit logging.

## Project Status

This repo is now in paused placeholder mode. It remains a working reference implementation for an autonomous QA framework, but scheduled GitHub Actions automation is disabled and GitHub publishing is off by default.

What this project achieved:

- autonomous Jira QA intake through label, watch, explicit issue, or webhook flows
- MCP-gated Jira, GitHub, Slack, and optional Playwright integrations
- LangGraph-controlled inspect, plan, generate, validate, repair flow
- Playwright generated-test execution with scrubbed secrets
- SQLite and JSONL run history for learning from prior failures
- Slack notifications and optional GitHub failure issue / passing-test PR publishing
- OpenTelemetry tracing hooks for local console or OTLP backends
- safety guardrails for generated code, off-target navigation, and mutation-like actions

Older red GitHub Actions runs were mostly caused by stale generated tests `SCRUM-2` and `SCRUM-3` using brittle Hacker News selectors. Those files were removed from normal CI. `SCRUM-4` remains as the stable generated-test example.

## Setup

1. Copy `.env.example` to `.env`.
2. Fill in `TARGET_URL`, Jira credentials, and `OPENAI_API_KEY`.
3. Optional: fill `SLACK_WEBHOOK_URL`, `GITHUB_REPO`, `GITHUB_TOKEN`, `JIRA_TRANSITION_DONE_ID`, and `JIRA_WEBHOOK_SECRET`.
4. Install dependencies:

```powershell
npm install
npx playwright install chromium
```

## Commands

Run the baseline target smoke test:

```powershell
npm run test
```

Check any Jira issue before generation:

```powershell
npm run agent:check-ticket -- ISSUE-KEY
```

Run autonomous QA once for all ready Jira tickets:

```powershell
npm run agent:auto
```

Run autonomous QA once for one explicit issue:

```powershell
npm run agent:auto -- --issue ISSUE-KEY
```

Run continuously, polling Jira for ready tickets:

```powershell
npm run agent:auto -- --watch
```

Start the Jira webhook listener:

```powershell
npm run agent:webhook
```

Production webhook requests should send a `POST` request to `/jira/webhook` with this JSON body:

```json
{
  "issueKey": "{{issue.key}}"
}
```

Use these headers:

```text
X-QA-Agent-Timestamp: <unix epoch milliseconds>
X-QA-Agent-Signature: sha256=<HMAC_SHA256(JIRA_WEBHOOK_SECRET, timestamp + "." + raw_body)>
```

If you connect directly from Jira Automation and cannot compute an HMAC signature, set `AGENT_WEBHOOK_ALLOW_SHARED_SECRET=true` and send `X-QA-Agent-Secret: <JIRA_WEBHOOK_SECRET>`. Do not put the secret in the URL.

Find and save a Done-like Jira transition id:

```powershell
npm run agent:find-transition
npm run agent:find-transition -- ISSUE-KEY
```

Autonomous mode looks for Jira issues matching `JIRA_LABEL`, skips issues already labeled with `JIRA_AUTONOMOUS_FAILURE_LABEL`, and requires no approval prompt. It still guard-checks generated code before writing or running it. Failed or unsafe tickets are labeled with `JIRA_AUTONOMOUS_FAILURE_LABEL` so the watcher does not retry the same failing ticket forever.

Webhook mode uses the same autonomous flow as `agent:auto -- --issue ISSUE-KEY`. Duplicate webhooks for the same currently running issue are accepted but skipped.

On an autonomous run, the agent now:

- runs the generated Playwright test
- writes run history to SQLite and `metrics/ledger.jsonl`
- comments the result back to Jira
- sends a Slack message when `SLACK_WEBHOOK_URL` is set, or when `SLACK_BOT_TOKEN` and `SLACK_CHANNEL_ID` are set
- creates a GitHub issue for failed tests only when `GITHUB_PUBLISH_MODE=issue` or `GITHUB_PUBLISH_MODE=all`
- creates or updates a branch like `ai-qa/SCRUM-4`, commits the generated passing test, opens/reuses a PR, and links it back to Jira only when `GITHUB_PUBLISH_MODE=pr` or `GITHUB_PUBLISH_MODE=all`
- transitions passed Jira tickets only when `AGENT_TRANSITION_ON_PASS=true` and `JIRA_TRANSITION_DONE_ID` are set

Generated Playwright runs use a scrubbed environment so Jira, Slack, GitHub, and OpenAI secrets are not exposed to generated test code. Blank optional integration values are skipped, so local test generation still works while Slack, GitHub, or Jira transition credentials are being added. Public comments and notifications do not include model token counts.

The AI generation path is a LangGraph state machine: inspect target, load lessons, plan, generate, validate, repair once when needed, then return either safe test code or a blocked reason. LangChain remains the model adapter inside those graph nodes.

For production isolation, set `AGENT_TEST_RUNNER=docker` to run generated Playwright tests in a Playwright container. Local development defaults to `AGENT_TEST_RUNNER=process`, which still runs with a scrubbed environment and static guard checks against external navigation.

OpenTelemetry is disabled by default. Enable local console traces with:

```env
OTEL_ENABLED=true
OTEL_TRACES_EXPORTER=console
```

For an observability backend, set `OTEL_TRACES_EXPORTER=otlp` and `OTEL_EXPORTER_OTLP_ENDPOINT`.

Check stored run history:

```powershell
npm run agent:history
```

## GitHub Actions CI

The workflow in `.github/workflows/qa-agent.yml` is manual-only in placeholder mode. It can be triggered with `workflow_dispatch` from GitHub Actions, but it no longer runs on a schedule.

It runs:

- `npm run typecheck`
- `npm test`
- `npm run agent:auto`

The workflow is split into a read-only validation job and a write-enabled autonomous publishing job. The generated Playwright test process still receives only a scrubbed environment.

Configure these repository secrets before running the workflow manually or re-enabling automation:

```text
TARGET_URL
JIRA_BASE_URL
JIRA_EMAIL
JIRA_API_TOKEN
JIRA_PROJECT_KEY
JIRA_LABEL
JIRA_TRANSITION_DONE_ID
JIRA_AUTONOMOUS_FAILURE_LABEL
AGENT_TRANSITION_ON_PASS
TARGET_TEST_HINTS
OPENAI_API_KEY
SLACK_WEBHOOK_URL
SLACK_BOT_TOKEN
SLACK_CHANNEL_ID
```

The workflow uses `GITHUB_REPO=tamzidnahian/Full-MCP-QA` and the built-in GitHub Actions `GITHUB_TOKEN` when GitHub publishing is explicitly enabled. It uploads Playwright reports, test results, metrics, and `state/agent.sqlite` as artifacts, and caches `state/` so future manual runs can reuse QA history.

## GitHub Placeholder Mode

GitHub publishing is off by default:

```env
GITHUB_PUBLISH_MODE=off
```

Supported values:

- `off`: no GitHub API calls
- `pr`: create/update generated-test PRs for passing runs
- `issue`: create GitHub issues for failed runs
- `all`: enable both PRs and failure issues

The GitHub MCP server, config, and scripts stay in the repo as reference architecture. With `off`, the runtime skips GitHub MCP calls entirely.

## MCP

The autonomous workflow uses `mcp.config.json` to route integration calls through allowlisted operation aliases:

- `jira.getIssue`, `jira.findReadyIssues`, `jira.commentIssue`, `jira.addLabel`, `jira.removeLabel`, `jira.transitionIssue`
- `github.createIssue`, `github.createBranch`, `github.createOrUpdateFile`, `github.findPullRequest`, `github.createPullRequest`
- `slack.postMessage`
- optional Playwright MCP inspection through `playwright.navigate`, `playwright.snapshot`, and `playwright.close`

Run the redacted agent status MCP server:

```powershell
npm run mcp:server
```

It exposes `qa_agent_status`, which reports configured integrations and a redacted latest QA metric.

The local stdio MCP servers used by the gateway can also be started directly for debugging:

```powershell
npm run mcp:jira
npm run mcp:github
npm run mcp:slack
```

Playwright MCP inspection is optional. To use it, run a Playwright MCP server on `PLAYWRIGHT_MCP_URL` and set:

```env
PLAYWRIGHT_MCP_ENABLED=true
PLAYWRIGHT_MCP_URL=http://localhost:8931/mcp
```

If Playwright MCP is disabled or unavailable, target inspection falls back to the local Playwright browser path. Generated tests still execute through the local Playwright CLI so reports and artifacts continue to work.

Every MCP call is audited in `state/agent.sqlite` table `mcp_audit` with redacted input, output, errors, duration, and timestamp.

## Example Target

Default target:

```env
TARGET_URL=https://news.ycombinator.com
```

Good first Jira issue description:

```text
As a reader, I want to open Hacker News so that I can browse current stories.

Acceptance criteria:
- Open the homepage.
- The Hacker News navigation/header is visible.
- At least one story link is visible.
- The More link is visible.
- The test runs in Chromium.
- The test must not log in, vote, comment, or modify data.
```