# Googlebot Simulator MCP

An MCP server that simulates Googlebot crawling pages and captures analytics events. Use this to verify your site's bot detection and analytics tracking work correctly.

## Features

- **Authentic Googlebot User-Agents**: Uses real Googlebot desktop and mobile user-agent strings
- **Idle Detection Simulation**: Mimics Googlebot's `requestIdleCallback`-based idle detection
- **Event Interception**: Captures analytics/tracking requests matching your pattern
- **Detailed Reporting**: Returns timing info, captured payloads, and screenshots

## Installation

### Add to Claude Code

```bash
claude mcp add googlebot-sim -- npx googlebot-simulator-mcp
```

### Or add to `.claude.json`

```json
{
  "mcpServers": {
    "googlebot-sim": {
      "type": "stdio",
      "command": "npx",
      "args": ["googlebot-simulator-mcp"]
    }
  }
}
```

### From source

```bash
git clone https://github.com/RichardDillman/googlebot-simulator-mcp.git
cd googlebot-simulator-mcp
npm install
npm run build
```

## Usage

Once added to Claude, you can use natural language:

> "Test https://example.com/jobs as Googlebot, watching for requests containing 'page_viewed'"

Or be more specific:

> "Simulate mobile Googlebot on https://talent.com/jobs, capture events to **/serp-event-producer/send"

## Tool: `simulate_googlebot`

### Input

```typescript
{
  // Required: URL(s) to test
  urls: string | string[];

  // Required: Pattern to match analytics events
  eventPattern: {
    urlPattern?: string;    // e.g., "**/analytics/**"
    bodyContains?: string;  // e.g., "page_viewed"
  };

  // Optional configuration
  options?: {
    userAgent?: "desktop" | "mobile";  // Default: "desktop"
    idleTimeout?: number;              // Default: 5000ms
    maxWaitTime?: number;              // Default: 10000ms
    captureScreenshot?: boolean;       // Default: true
    screenshotDir?: string;            // Default: cwd
  };
}
```

### Output

```typescript
{
  results: Array<{
    url: string;
    success: boolean;              // true if matching event was captured
    eventsFired: Array<{
      timestamp: number;
      url: string;
      method: string;
      payload: object;
      matchedPattern: boolean;
      responseStatus?: number;
    }>;
    timing: {
      navigationStart: number;
      domContentLoaded: number;
      idleDetected: number;
      totalTime: number;
      firstIdleCallback?: number;
    };
    screenshot?: string;           // Path to screenshot file
    errors: string[];
    idleCallbacksObserved: number;
  }>;
  summary: {
    totalUrls: number;
    passed: number;
    failed: number;
    totalEventsCaptured: number;
  };
}
```

## Example Output

```json
{
  "results": [
    {
      "url": "https://talent.com/jobs",
      "success": true,
      "eventsFired": [
        {
          "timestamp": 1704825600000,
          "url": "https://events.talent.com/serp-event-producer/send",
          "method": "POST",
          "payload": {
            "event_type": "page_viewed_bot",
            "bot_type": "google",
            "page_name": "serp"
          },
          "matchedPattern": true,
          "responseStatus": 200
        }
      ],
      "timing": {
        "navigationStart": 1704825595000,
        "domContentLoaded": 1704825596500,
        "idleDetected": 1704825600000,
        "totalTime": 5000,
        "firstIdleCallback": 3200
      },
      "screenshot": "/tmp/googlebot_desktop_jobs_1704825600000.png",
      "errors": [],
      "idleCallbacksObserved": 3
    }
  ],
  "summary": {
    "totalUrls": 1,
    "passed": 1,
    "failed": 0,
    "totalEventsCaptured": 1
  }
}
```

## How It Works

1. **Launches headless Chromium** with Googlebot user-agent and viewport settings
2. **Injects tracking script** to monitor `requestIdleCallback` usage
3. **Sets up network interception** to capture matching requests
4. **Navigates to URL** and waits for page to reach idle state
5. **Captures screenshot** at the "idle" moment (what Googlebot sees)
6. **Returns detailed report** with all captured events and timing

## Use Cases

- **Verify bot detection**: Ensure your analytics correctly identifies Googlebot
- **Test event timing**: Confirm events fire before Googlebot's idle timeout
- **Debug rendering**: See what content is visible when Googlebot considers the page "done"
- **Compare desktop vs mobile**: Test both Googlebot variants

## License

MIT