@pluswithai/cli

Run your Pluswithai tests from CI and the terminal. It starts the runs, waits for them and exits non-zero when one fails.

• No runtime dependencies. Node 18.3 or later.

• Readable CI logs, a GitHub step summary, failure annotations and JUnit XML.

• Your API key is only read from the environment, and it is never printed.

Quick start

1. In the app, open API keys in the sidebar, choose Create key, then pick CI / CLI.

2. Export the key, then run your project's tests:

export PLUSWITHAI_API_KEY=pwai_…
npx @pluswithai/cli run --project shop

Started 2 runs in shop. Waiting for results…
✓ tests/home.spec.ts  passed  8.4s
✗ tests/checkout.spec.ts  failed  21.0s
Pluswithai · shop: 1 passed, 1 failed
  ✗ tests/checkout.spec.ts  failed  21.0s
Details: https://pluswithai.com/app?project=shop

Related MCP server: TestPilot MCP

GitHub Actions

Store the key as the repository secret PLUSWITHAI_API_KEY and use the action:

name: Pluswithai tests
on: [push, workflow_dispatch]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: pluswithai/cli@v0
        with:
          api-key: ${{ secrets.PLUSWITHAI_API_KEY }}
          project: shop
          environment: staging   # optional; default is production (the project URL)

To test every pull request's preview deploy (Vercel, Netlify, Cloudflare Pages) and block the merge when a test fails:

on: deployment_status
jobs:
  e2e:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: pluswithai/cli@v0
        with:
          api-key: ${{ secrets.PLUSWITHAI_API_KEY }}
          project: shop
          base-url: ${{ github.event.deployment_status.environment_url }}

The preview's host must match a preview host pattern declared in the project's Settings, e.g. myapp-*.vercel.app. See Environments and PR checks.

Inside GitHub Actions, the CLI also:

• writes a table of the results to the job's summary page (turn it off with --no-summary);

• adds an error annotation for every failed test.

Commands

Every command takes --help, and every command except mcp takes --json.

Run tests on your machine

pluswithai local downloads the project's tests, installs Playwright the first time and runs them here — useful against a local server before you deploy:

pluswithai local --project shop --base-url http://localhost:3000
pluswithai local --project shop -t tests/checkout.spec.ts --headed

The files go to pluswithai/<project> (--dir to change it): plain Playwright plus the project's config and fixture, so npx playwright test works there on its own too (pluswithai pull only downloads). The site's HTTP auth and extra headers stay on Pluswithai: set PLUSWITHAI_HTTP_USERNAME, PLUSWITHAI_HTTP_PASSWORD and PLUSWITHAI_EXTRA_HEADERS (a JSON object) when the site needs them. Needs Node.js 18+.

AI agents (MCP)

Pluswithai is also an MCP server. With it, an agent like Claude Code, Cursor or Codex can run your tests, read the failures, push new tests and schedule them.

Clients that support remote MCP servers connect straight to https://api.pluswithai.com/mcp. Claude.ai and ChatGPT sign in with OAuth, so you don't copy a key. Clients that only start local servers can use this CLI as a stdio bridge:

{
  "mcpServers": {
    "pluswithai": {
      "command": "npx",
      "args": ["-y", "@pluswithai/cli", "mcp"],
      "env": { "PLUSWITHAI_API_KEY": "pwai_…" }
    }
  }
}

Use a key with Agent access to get every tool. Logs go to stderr; stdout carries only the protocol.

run options

Environment

Exit codes

If several apply, the CLI reports the first one in this order: a failed run, then an error, then the timeout.

Development

npm install
npm test        # builds, runs unit + end-to-end tests, enforces 100% coverage

The design is documented in SPEC.md.

License

MIT