# Deploy Test

A CLI tool for running Playwright test fixtures against live deployments, with
optional MCP (Model Context Protocol) interface for AI agents"

## Features

- ✅ Run test fixtures against live deployments
- ✅ Capture screenshots, console errors, and network failures
- ✅ Support for parallel or sequential test execution
- ✅ Tag-based test filtering
- ✅ Structured test results with assertions
- ✅ Reusable test fixture interface


## Why Deploytest? Reusable Tests vs Ad-Hoc Verification

### The Problem with Ad-Hoc Testing

When verifying a deployment, it's tempting to test things manually or write throwaway verification code. AI agents often try to create one-off bash scripts or Playwright commands to check if something works. This is inefficient for several reasons:

- **Wasteful**: Every test requires re-discovering what to check and how to check it
- **Error-prone**: Manual steps are forgotten, commands have typos
- **Not repeatable**: Can't easily re-run the same verification later
- **Token-inefficient**: Agents spend tokens re-figuring out the same verification logic

The same applies to humans doing manual testing in a browser.

### The Deploytest Approach

Deploytest follows a better pattern:

1. **Write test fixtures once** - Codify your verification logic as reusable test fixtures
2. **Run locally during development** - Test against localhost as part of your normal workflow
3. **Run against deployments** - Use the **same test code** to verify staging/production deploys
4. **Get structured results** - Receive pass/fail, errors, screenshots, and assertions

Your test fixtures are proper code that lives in your repo and can be maintained over time.

### Comparison to Browser MCPs

**[Chrome DevTools MCP](https://github.com/ChromeDevTools/chrome-devtools-mcp)** - For **exploration and discovery**
- General-purpose browser inspection and DOM manipulation
- Great for: Understanding a site, debugging issues, exploring unknown pages
- Trade-off: Ad-hoc usage, requires many tokens to figure out what to check

**[Playwright MCP](https://github.com/microsoft/playwright-mcp)** - For **scripted browser automation**
- Low-level browser automation commands
- Great for: One-off automation tasks, quick checks
- Trade-off: Each verification requires writing/running commands from scratch

**Deploytest** - For **deployment verification with reusable tests**
- Test fixtures that encode domain knowledge about your site
- Great for: Verifying deployments, regression testing, CI/CD integration
- Benefit: Write once, run many times; token-efficient; structured results

### Suggested Workflow

1. Use Chrome DevTools MCP or Playwright MCP to **explore** and understand what needs testing
2. **Codify** your findings into deploytest fixtures (proper test code in your repo)
3. Run those fixtures locally during development
4. Use deploytest (CLI or MCP) to verify staging/production deployments

### CLI First, MCP as a Subcommand

Deploytest is primarily a **CLI tool**. Run `deploytest mcp` to start it as an MCP server for AI agents:

```bash
# Direct CLI usage
deploytest run-fixture --url https://staging.example.com --fixture login-flow

# MCP server mode (for Claude Desktop/Code)
deploytest mcp
```

The **MCP mode** simply makes it convenient for AI agents to run your existing test fixtures. The agent doesn't write ad-hoc test code - it runs your well-tested, reusable fixtures.

## Installation

```bash
npm install
npm run build
```

Or with bun:

```bash
bun install
bun run build
```

## Install Playwright Browsers

```bash
npx playwright install chromium
```

## Usage

### As a CLI Tool

```bash
# List all available fixtures
deploytest list-fixtures

# List fixtures with specific tags
deploytest list-fixtures --tags smoke,critical

# Run a specific fixture against a deployment
deploytest run-fixture --url https://staging.myapp.com --fixture basic-page-load

# Run all fixtures
deploytest run-all-fixtures --url https://staging.myapp.com

# Run tagged fixtures in parallel
deploytest run-all-fixtures --url https://staging.myapp.com --tags smoke --parallel
```

### As an MCP Server (for Claude Desktop / Claude Code)

Add this to your MCP settings:

**MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
**Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "deploytest": {
      "command": "deploytest",
      "args": ["mcp"]
    }
  }
}
```

Or if using npx (for published packages):

```json
{
  "mcpServers": {
    "deploytest": {
      "command": "npx",
      "args": ["-y", "deploytest", "mcp"]
    }
  }
}
```

## Available Tools

### `list_fixtures`

List all available test fixtures, optionally filtered by tags.

**Parameters:**
- `tags` (optional): Array of tags to filter by

**Example:**
```typescript
// List all fixtures
await mcp.list_fixtures();

// List only smoke tests
await mcp.list_fixtures({ tags: ["smoke"] });
```

### `run_fixture`

Run a specific test fixture against a deployment URL.

**Parameters:**
- `baseUrl` (required): URL of the deployment to test
- `fixtureName` (required): Name of the fixture to run
- `headless` (optional): Run in headless mode (default: true)

**Example:**
```typescript
await mcp.run_fixture({
  baseUrl: "https://staging.myapp.com",
  fixtureName: "basic-page-load"
});
```

### `run_all_fixtures`

Run all test fixtures (or filtered by tags) against a deployment URL.

**Parameters:**
- `baseUrl` (required): URL of the deployment to test
- `tags` (optional): Only run fixtures with these tags
- `parallel` (optional): Run in parallel (default: false)
- `headless` (optional): Run in headless mode (default: true)

**Example:**
```typescript
// Run all smoke tests
await mcp.run_all_fixtures({
  baseUrl: "https://staging.myapp.com",
  tags: ["smoke"],
  parallel: true
});
```

## Creating Custom Test Fixtures

Test fixtures are defined in `src/fixtures.ts`. Each fixture implements the `TestFixture` interface:

```typescript
import type { TestFixture, TestResult } from './types.js';

export const myCustomTest: TestFixture = {
  name: 'my-custom-test',
  description: 'Description of what this tests',
  tags: ['smoke', 'critical'],
  timeout: 30000,
  
  run: async (page, baseUrl) => {
    const start = Date.now();
    const consoleErrors: string[] = [];
    
    page.on('console', msg => {
      if (msg.type() === 'error') consoleErrors.push(msg.text());
    });
    
    try {
      await page.goto(`${baseUrl}/my-page`);
      
      // Your test logic here
      const isVisible = await page.isVisible('[data-testid="my-element"]');
      
      return {
        passed: isVisible,
        duration: Date.now() - start,
        artifacts: {
          screenshot: await page.screenshot({ encoding: 'base64' }),
          consoleErrors
        },
        assertions: [
          {
            selector: '[data-testid="my-element"]',
            expected: 'visible',
            actual: isVisible,
            passed: isVisible
          }
        ]
      };
      
    } catch (error: any) {
      return {
        passed: false,
        duration: Date.now() - start,
        error: {
          message: error.message,
          stack: error.stack
        },
        artifacts: {
          screenshot: await page.screenshot({ encoding: 'base64' }).catch(() => undefined),
          consoleErrors
        }
      };
    }
  }
};

// Add to the exported array
export const testFixtures: TestFixture[] = [
  // ... existing fixtures
  myCustomTest
];
```

## Integration with Your App

To use this MCP with your own application's test fixtures:

1. **Option 1: Direct Import** (if fixtures are in same repo)
   ```typescript
   // src/fixtures.ts
   import { myAppFixtures } from '../my-app/tests/fixtures.js';
   export const testFixtures = [...myAppFixtures];
   ```

2. **Option 2: Symlink** (if fixtures are in separate repo)
   ```bash
   ln -s /path/to/my-app/tests/fixtures.ts src/app-fixtures.ts
   ```
   
   ```typescript
   // src/fixtures.ts
   import { testFixtures as appFixtures } from './app-fixtures.js';
   export const testFixtures = appFixtures;
   ```

3. **Option 3: Dynamic Import** (load at runtime)
   ```typescript
   // Configure path via environment variable
   const fixturesPath = process.env.FIXTURES_PATH || './fixtures.js';
   const { testFixtures } = await import(fixturesPath);
   ```

## Usage with Claude Code

Once configured, Claude Code can use this MCP to verify deployments:

```
User: "Test my staging deployment at https://staging.myapp.com"

Claude Code will:
1. Call list_fixtures to see available tests
2. Call run_all_fixtures with the URL
3. Analyze the results (screenshots, errors, assertions)
4. Report any issues and suggest fixes
```

## Example Claude Code Workflow

```
# Claude Code automatically uses the MCP like this:

1. User deploys to staging
2. Claude Code calls: run_all_fixtures({ 
     baseUrl: "https://staging.myapp.com",
     tags: ["smoke"]
   })
3. Gets back:
   - Test summary (10/12 passed)
   - Failed test details with screenshots
   - Console errors
   - Network failures
4. Claude analyzes and reports:
   "Deployment verification: 2 tests failed
   - login-flow: Button selector changed
   - checkout-flow: 404 on /api/cart"
5. Claude can fix the issues or alert you
```

## Development

```bash
# Watch mode for development
npm run watch

# Run directly with tsx (for testing)
npm run dev

# Build for production
npm run build
```

## Testing the MCP

You can test the MCP using the MCP Inspector:

```bash
npx @modelcontextprotocol/inspector node dist/index.js
```

## Demo & Examples

The `demo/` directory contains a complete example with working and broken demo sites that you can use to verify the tool works correctly.

**Run the automated demo:**
```bash
npm run demo
```

This will:
1. Start local servers for both working and broken example sites
2. Launch Chromium via Playwright
3. Run all test fixtures against both sites
4. Generate a comprehensive report showing detected issues

**Manual exploration:**
```bash
# Start working site on http://localhost:3000
npm run demo:working

# Start broken site on http://localhost:3001
npm run demo:broken
```

See [demo/README.md](demo/README.md) for detailed documentation.

## License

MIT

## Contributing

To add new fixtures:
1. Create your fixture in `src/fixtures.ts`
2. Follow the `TestFixture` interface
3. Add it to the exported `testFixtures` array
4. Rebuild: `npm run build`

The fixture will automatically be available to Claude Code!