Website QA Agent

Standalone Jira-to-Playwright QA agent for testing any website URL, then posting results to Jira, Slack, and GitHub when those integrations are configured. External integrations run through a local MCP gateway with allowlisted operations and redacted SQLite audit logging.

Project Status

This repo is now in paused placeholder mode. It remains a working reference implementation for an autonomous QA framework, but scheduled GitHub Actions automation is disabled and GitHub publishing is off by default.

What this project achieved:

• autonomous Jira QA intake through label, watch, explicit issue, or webhook flows

• MCP-gated Jira, GitHub, Slack, and optional Playwright integrations

• LangGraph-controlled inspect, plan, generate, validate, repair flow

• Playwright generated-test execution with scrubbed secrets

• SQLite and JSONL run history for learning from prior failures

• Slack notifications and optional GitHub failure issue / passing-test PR publishing

• OpenTelemetry tracing hooks for local console or OTLP backends

• safety guardrails for generated code, off-target navigation, and mutation-like actions

Older red GitHub Actions runs were mostly caused by stale generated tests SCRUM-2 and SCRUM-3 using brittle Hacker News selectors. Those files were removed from normal CI. SCRUM-4 remains as the stable generated-test example.

Related MCP server: VerifyAX MCP

Setup

1. Copy .env.example to .env.

2. Fill in TARGET_URL, Jira credentials, and OPENAI_API_KEY.

3. Optional: fill SLACK_WEBHOOK_URL, GITHUB_REPO, GITHUB_TOKEN, JIRA_TRANSITION_DONE_ID, and JIRA_WEBHOOK_SECRET.

4. Install dependencies:

npm install
npx playwright install chromium

Commands

Run the baseline target smoke test:

npm run test

Check any Jira issue before generation:

npm run agent:check-ticket -- ISSUE-KEY

Run autonomous QA once for all ready Jira tickets:

npm run agent:auto

Run autonomous QA once for one explicit issue:

npm run agent:auto -- --issue ISSUE-KEY

Run continuously, polling Jira for ready tickets:

npm run agent:auto -- --watch

Start the Jira webhook listener:

npm run agent:webhook

Production webhook requests should send a POST request to /jira/webhook with this JSON body:

{
  "issueKey": "{{issue.key}}"
}

Use these headers:

X-QA-Agent-Timestamp: <unix epoch milliseconds>
X-QA-Agent-Signature: sha256=<HMAC_SHA256(JIRA_WEBHOOK_SECRET, timestamp + "." + raw_body)>

If you connect directly from Jira Automation and cannot compute an HMAC signature, set AGENT_WEBHOOK_ALLOW_SHARED_SECRET=true and send X-QA-Agent-Secret: <JIRA_WEBHOOK_SECRET>. Do not put the secret in the URL.

Find and save a Done-like Jira transition id:

npm run agent:find-transition
npm run agent:find-transition -- ISSUE-KEY

Autonomous mode looks for Jira issues matching JIRA_LABEL, skips issues already labeled with JIRA_AUTONOMOUS_FAILURE_LABEL, and requires no approval prompt. It still guard-checks generated code before writing or running it. Failed or unsafe tickets are labeled with JIRA_AUTONOMOUS_FAILURE_LABEL so the watcher does not retry the same failing ticket forever.

Webhook mode uses the same autonomous flow as agent:auto -- --issue ISSUE-KEY. Duplicate webhooks for the same currently running issue are accepted but skipped.

On an autonomous run, the agent now:

• runs the generated Playwright test

• writes run history to SQLite and metrics/ledger.jsonl

• comments the result back to Jira

• sends a Slack message when SLACK_WEBHOOK_URL is set, or when SLACK_BOT_TOKEN and SLACK_CHANNEL_ID are set

• creates a GitHub issue for failed tests only when GITHUB_PUBLISH_MODE=issue or GITHUB_PUBLISH_MODE=all

• creates or updates a branch like ai-qa/SCRUM-4, commits the generated passing test, opens/reuses a PR, and links it back to Jira only when GITHUB_PUBLISH_MODE=pr or GITHUB_PUBLISH_MODE=all

• transitions passed Jira tickets only when AGENT_TRANSITION_ON_PASS=true and JIRA_TRANSITION_DONE_ID are set

Generated Playwright runs use a scrubbed environment so Jira, Slack, GitHub, and OpenAI secrets are not exposed to generated test code. Blank optional integration values are skipped, so local test generation still works while Slack, GitHub, or Jira transition credentials are being added. Public comments and notifications do not include model token counts.

The AI generation path is a LangGraph state machine: inspect target, load lessons, plan, generate, validate, repair once when needed, then return either safe test code or a blocked reason. LangChain remains the model adapter inside those graph nodes.

For production isolation, set AGENT_TEST_RUNNER=docker to run generated Playwright tests in a Playwright container. Local development defaults to AGENT_TEST_RUNNER=process, which still runs with a scrubbed environment and static guard checks against external navigation.

OpenTelemetry is disabled by default. Enable local console traces with:

OTEL_ENABLED=true
OTEL_TRACES_EXPORTER=console

For an observability backend, set OTEL_TRACES_EXPORTER=otlp and OTEL_EXPORTER_OTLP_ENDPOINT.

Check stored run history:

npm run agent:history

GitHub Actions CI

The workflow in .github/workflows/qa-agent.yml is manual-only in placeholder mode. It can be triggered with workflow_dispatch from GitHub Actions, but it no longer runs on a schedule.

It runs:

• npm run typecheck

• npm test

• npm run agent:auto

The workflow is split into a read-only validation job and a write-enabled autonomous publishing job. The generated Playwright test process still receives only a scrubbed environment.

Configure these repository secrets before running the workflow manually or re-enabling automation:

TARGET_URL
JIRA_BASE_URL
JIRA_EMAIL
JIRA_API_TOKEN
JIRA_PROJECT_KEY
JIRA_LABEL
JIRA_TRANSITION_DONE_ID
JIRA_AUTONOMOUS_FAILURE_LABEL
AGENT_TRANSITION_ON_PASS
TARGET_TEST_HINTS
OPENAI_API_KEY
SLACK_WEBHOOK_URL
SLACK_BOT_TOKEN
SLACK_CHANNEL_ID

The workflow uses GITHUB_REPO=tamzidnahian/Full-MCP-QA and the built-in GitHub Actions GITHUB_TOKEN when GitHub publishing is explicitly enabled. It uploads Playwright reports, test results, metrics, and state/agent.sqlite as artifacts, and caches state/ so future manual runs can reuse QA history.

GitHub Placeholder Mode

GitHub publishing is off by default:

GITHUB_PUBLISH_MODE=off

Supported values:

• off: no GitHub API calls

• pr: create/update generated-test PRs for passing runs

• issue: create GitHub issues for failed runs

• all: enable both PRs and failure issues

The GitHub MCP server, config, and scripts stay in the repo as reference architecture. With off, the runtime skips GitHub MCP calls entirely.

MCP

The autonomous workflow uses mcp.config.json to route integration calls through allowlisted operation aliases:

• jira.getIssue, jira.findReadyIssues, jira.commentIssue, jira.addLabel, jira.removeLabel, jira.transitionIssue

• github.createIssue, github.createBranch, github.createOrUpdateFile, github.findPullRequest, github.createPullRequest

• slack.postMessage

• optional Playwright MCP inspection through playwright.navigate, playwright.snapshot, and playwright.close

Run the redacted agent status MCP server:

npm run mcp:server

It exposes qa_agent_status, which reports configured integrations and a redacted latest QA metric.

The local stdio MCP servers used by the gateway can also be started directly for debugging:

npm run mcp:jira
npm run mcp:github
npm run mcp:slack

Playwright MCP inspection is optional. To use it, run a Playwright MCP server on PLAYWRIGHT_MCP_URL and set:

PLAYWRIGHT_MCP_ENABLED=true
PLAYWRIGHT_MCP_URL=http://localhost:8931/mcp

If Playwright MCP is disabled or unavailable, target inspection falls back to the local Playwright browser path. Generated tests still execute through the local Playwright CLI so reports and artifacts continue to work.

Every MCP call is audited in state/agent.sqlite table mcp_audit with redacted input, output, errors, duration, and timestamp.

Example Target

Default target:

TARGET_URL=https://news.ycombinator.com

Good first Jira issue description:

As a reader, I want to open Hacker News so that I can browse current stories.

Acceptance criteria:
- Open the homepage.
- The Hacker News navigation/header is visible.
- At least one story link is visible.
- The More link is visible.
- The test runs in Chromium.
- The test must not log in, vote, comment, or modify data.