# ChatGPT Escalation MCP Server

An MCP (Model Context Protocol) server that enables autonomous coding agents to escalate complex questions to the ChatGPT Desktop app automatically — **ToS-compliant** via native UI automation.

**What this does:** This tool lets autonomous coding agents (Copilot, Claude, Cline, Roo, etc.) escalate hard questions to the *ChatGPT Desktop app* on your computer. It automates ChatGPT the same way a human would — clicking the UI, sending the question, waiting for the response, copying it — then returns the answer to your agent so it can continue working *without you*.

> ### 🖥️ Windows 10/11 Only
> **This tool supports only Windows.** macOS and Linux are not supported and there are no plans to add support.
>
> ### ⚠️ Important Requirements
> - **ChatGPT Desktop app** (Microsoft Store version)
> - Automation controls your ChatGPT window — **don't touch it** during escalations
> - Only **one escalation at a time** (requests are queued)
> - UI changes in ChatGPT may break automation — [open an issue](../../issues) if this happens
>
> ### ✅ ToS Compliant
> This tool only automates your *local* ChatGPT Desktop application. It does **not** automate the web UI, bypass security features, or scrape data.

## Features

- **Two MCP Tools**:
  - `escalate_to_expert` - Send questions to ChatGPT and receive detailed responses
  - `list_projects` - Discover available project IDs from your configuration
- **100% Accurate UI Detection** - Pixel-based detection for sidebar state and response completion
- **OCR-Based Navigation** - PaddleOCR v5 for reliable text extraction and fuzzy matching
- **Async Model Loading** - OCR models preload in background for faster response times
- **Project Organization** - Map multiple projects to different ChatGPT conversations

## How It Works

```
┌─────────────────┐    MCP Protocol    ┌──────────────────┐
│  Coding Agent   │◄──────────────────►│   MCP Server     │
│ (Copilot/Roo)   │                    │  (This Project)  │
└─────────────────┘                    └────────┬─────────┘
                                                │
                                                │ spawn
                                                ▼
                                       ┌──────────────────┐
                                       │  Python Driver   │
                                       │  (Windows)       │
                                       └────────┬─────────┘
                                                │
                                                │ UI Automation
                                                ▼
                                       ┌──────────────────┐
                                       │  ChatGPT Desktop │
                                       │       App        │
                                       └──────────────────┘
```

### Automation Flow

1. **Kill ChatGPT** - Ensures clean state
2. **Open ChatGPT** - Fresh start
3. **Focus Window** - Bring to foreground
4. **Open Sidebar** - Click hamburger menu (pixel detection for state)
5. **Click Project** - OCR + fuzzy matching to find folder
6. **Click Conversation** - OCR + fuzzy matching to find chat (Ctrl+K fallback if not found)
7. **Focus Input** - Click text input area
8. **Send Prompt** - Paste and submit
9. **Wait for Response** - Pixel-based stop button detection
10. **Copy Response** - Robust button probing to find copy button

**Automatic Retry Logic**: If any step fails, the entire flow restarts (up to 4 attempts total). Each retry gets a fresh ChatGPT instance. Most failures are transient (focus lost, window minimized) and succeed on retry.

## System Requirements

| Requirement | Version | Notes |
|-------------|---------|-------|
| **Windows** | 10 or 11 | macOS/Linux not supported |
| **ChatGPT Desktop** | Latest | Microsoft Store version |
| **Node.js** | 18+ | For the MCP server |
| **Python** | 3.10+ | For UI automation driver |
| **GPU** | Not required | CPU-only OCR works fine |

### Python Packages
```
pywinauto        # Windows UI automation
pyperclip        # Clipboard access
paddleocr        # Text recognition
paddlepaddle     # PaddleOCR backend
```

### Why Windows Only?

ChatGPT Desktop exposes fully accessible UI elements on Windows via UI Automation APIs. The pixel-based detection and keyboard/mouse automation work reliably on Windows.

macOS has different automation APIs (Accessibility API) that would require a complete rewrite of the driver. Linux doesn't have a ChatGPT Desktop app.

### Tested Environment

| Component | Version | Status |
|-----------|---------|--------|
| **ChatGPT Desktop** | 1.2025.112 | ✅ Tested |
| **Windows 11** | 24H2 (Build 26100.2605) | ✅ Tested |
| **Last Verified** | December 2, 2025 | |

### Robustness Features

- **Automatic Retries**: Up to 4 attempts per escalation with intelligent failure detection
- **Structured Observability**: Every escalation gets a unique `run_id` for correlation and debugging
- **Error Reason Codes**: 12+ specific error codes (e.g., `focus_failed`, `project_not_found`, `empty_response`)
- **Chaos Tested**: Passes aggressive chaos testing (random focus stealing, window minimization, mouse interference)
- **Smart Fallbacks**: Ctrl+K search if conversation not visible in sidebar

> 💡 **After ChatGPT Updates:** UI automation may break if ChatGPT significantly changes their layout. If you encounter issues after an update, please [open an issue](../../issues) with your ChatGPT version.

## Installation

### Option 1: Install from npm (Recommended)

```powershell
# Install globally
npm install -g chatgpt-escalation-mcp

# Install Python dependencies
pip install pywinauto pyperclip paddleocr paddlepaddle

# Run setup wizard
chatgpt-escalation-mcp init
```

### Option 2: Install from GitHub Release

1. Download the latest release from [GitHub Releases](https://github.com/Dazlarus/chatgpt-escalation-mcp/releases)
2. Extract the ZIP file
3. Run:

```powershell
cd chatgpt-escalation-mcp
npm install
npm run build
pip install pywinauto pyperclip paddleocr paddlepaddle
```

### Option 3: Install from Source

```powershell
# Clone the repository
git clone https://github.com/Dazlarus/chatgpt-escalation-mcp.git
cd chatgpt-escalation-mcp

# Install Node.js dependencies
npm install

# Build the project
npm run build

# Install Python dependencies
pip install pywinauto pyperclip paddleocr paddlepaddle
```

## Quick Start

### Step 1: Install ChatGPT Desktop

```powershell
winget install --id=9NT1R1C2HH7J --source=msstore --accept-package-agreements --accept-source-agreements
```

Or install from the Microsoft Store: search "ChatGPT" by OpenAI.

### Step 2: Create a Conversation in ChatGPT

1. Open ChatGPT Desktop and sign in
2. Create a new **Project** (folder) called `Agent Expert Help`
3. Inside that project, create a new conversation called `Copilot Escalations`
4. Send this initial message to set the context:

```
You are an expert software architect. I'll send you technical questions from my coding agent (GitHub Copilot, Claude, etc.) when it gets stuck. 

For each question:
1. Analyze the problem thoroughly
2. Provide specific, actionable guidance
3. Include code examples when helpful
4. Explain WHY a solution works, not just what to do

The questions will include context about what the agent already tried.
```

### Step 3: Configure the MCP Server

Create the config file at `~/.chatgpt-escalation/config.json`:

```powershell
# Create config directory
New-Item -ItemType Directory -Path "$env:USERPROFILE\.chatgpt-escalation" -Force

# Create config file (edit the path in notepad)
notepad "$env:USERPROFILE\.chatgpt-escalation\config.json"
```

Paste this configuration:

```json
{
  "chatgpt": {
    "platform": "win",
    "responseTimeout": 600000,
    "projects": {
      "default": {
        "folder": "Agent Expert Help",
        "conversation": "Copilot Escalations"
      }
    }
  },
  "logging": {
    "level": "info"
  }
}
```

### Step 4: Add to Your MCP Client

**For VS Code with GitHub Copilot** (`%APPDATA%\Code\User\mcp.json`):
```json
{
  "servers": {
    "chatgpt-escalation": {
      "command": "node",
      "args": ["N://AI Projects//chatgpt-escalation-mcp//dist//src//server.js"]
    }
  }
}
```

**For Claude Desktop** (`%APPDATA%\Claude\claude_desktop_config.json`):
```json
{
  "mcpServers": {
    "chatgpt-escalation": {
      "command": "node",
      "args": ["C://path//to//chatgpt-escalation-mcp//dist//src//server.js"]
    }
  }
}
```

> ⚠️ Use double forward slashes `//` in paths for JSON, or escape backslashes as `\\\\`

### Step 5: Teach Your Agent When to Escalate

Add escalation instructions to your agent. Choose the format that matches your tool:

<details>
<summary><strong>GitHub Copilot</strong> (.github/copilot-instructions.md)</summary>

```markdown
## Expert Escalation Protocol

You have access to the `escalate_to_expert` MCP tool that sends questions to ChatGPT for expert guidance.

### When to Escalate
- You've tried 3+ approaches without success
- The problem requires specialized domain knowledge
- You're unsure if the fundamental approach is correct
- You're hitting consistent failure patterns you can't diagnose

### How to Escalate
Use the `escalate_to_expert` tool with:
- `project`: "default" (or specific project ID)
- `reason`: Why you're stuck (be specific)
- `question`: The technical question
- `attempted`: What you already tried and results
- `artifacts`: Relevant code snippets

### After Escalation
Read the full response before implementing. ChatGPT often provides multiple approaches - pick the most appropriate one for the context.
```

</details>

<details>
<summary><strong>Cline / Roo Code</strong> (.clinerules or .roo/rules)</summary>

```markdown
## Expert Escalation Protocol

You have access to the `escalate_to_expert` MCP tool. Use it when stuck.

### Escalation Triggers
1. **Accuracy plateau** - 3+ attempts with no improvement
2. **Consistent failures** - Same error pattern despite different approaches
3. **Domain gap** - Problem needs specialized knowledge you lack
4. **Architecture uncertainty** - Unsure if approach is fundamentally correct

### Before Escalating
Stop and ask the user: "I've tried [X approaches] but I'm hitting [limitation]. Should I escalate to ChatGPT?"

If yes, call `escalate_to_expert` with:
- `project`: "default"
- `reason`: Brief description of why you're stuck
- `question`: Specific technical question
- `attempted`: Numbered list of what you tried and results
- `artifacts`: Relevant code snippets

### Question Format
Structure your question clearly:
- **Problem:** One sentence description
- **Context:** Technical details, frameworks, constraints
- **What I tried:** Numbered list with results
- **Specific questions:** What you need answered

### After Response
1. Read the FULL response before implementing
2. Identify the recommended approach (there may be multiple)
3. Implement incrementally - test each suggestion
4. If unclear, ask user for clarification before proceeding
```

</details>

<details>
<summary><strong>OpenAI Codex CLI</strong> (AGENTS.md or instructions)</summary>

```markdown
## Expert Escalation via ChatGPT

The `escalate_to_expert` MCP tool lets you ask ChatGPT for help on complex problems.

### When to Use
- Multiple failed attempts on a problem
- Need domain expertise (ML, systems, security, etc.)
- Debugging issues that don't make sense
- Architecture or design decisions

### Tool Usage
```
escalate_to_expert({
  project: "default",
  reason: "Brief explanation of the blocker",
  question: "Specific technical question",
  attempted: "What was tried and what happened",
  artifacts: [{type: "file_snippet", pathOrLabel: "file.py", content: "..."}]
})
```

### Best Practices
- Be specific about what you tried and exact error messages
- Include relevant code snippets in artifacts
- Ask focused questions, not "help me fix this"
- After receiving response, implement suggestions step by step
```

</details>

<details>
<summary><strong>Claude Desktop / Other MCP Clients</strong></summary>

```markdown
## Expert Escalation Protocol

You have access to the `escalate_to_expert` MCP tool that sends questions to ChatGPT.

### When to Escalate
- Tried 3+ approaches without success
- Problem requires specialized domain knowledge
- Unsure if fundamental approach is correct
- Hitting consistent failure patterns

### Tool Parameters
| Parameter | Required | Description |
|-----------|----------|-------------|
| project | Yes | Project ID (usually "default") |
| reason | Yes | Why you're escalating |
| question | Yes | The technical question |
| attempted | No | What you tried and results |
| artifacts | No | Code snippets [{type, pathOrLabel, content}] |

### After Response
Read fully before implementing. Pick the most appropriate suggestion for the context.
```

</details>

#### Example Escalation Call

```json
{
  "project": "default",
  "reason": "Authentication flow failing silently, can't identify root cause",
  "question": "Why would JWT refresh tokens work in development but fail in production with no error messages?",
  "attempted": "1. Checked token expiry (valid), 2. Verified CORS (correct), 3. Tested with Postman (works)",
  "artifacts": [{"type": "file_snippet", "pathOrLabel": "auth.ts", "content": "..."}]
}
```

## Configuration Reference

Config file location: `%USERPROFILE%\.chatgpt-escalation\config.json`

```json
{
  "chatgpt": {
    "platform": "win",
    "responseTimeout": 120000,
    "projects": {
      "my-project": {
        "folder": "My Project Folder",
        "conversation": "Expert Help Chat"
      },
      "simple-project": "Just a Conversation Title"
    }
  },
  "logging": {
    "level": "info"
  }
}
```

### Project Configuration

Projects can be configured two ways:

**Simple** (conversation at root level in ChatGPT sidebar):
```json
"project-id": "Conversation Title"
```

**With Folder** (conversation inside a ChatGPT project folder):
```json
"project-id": {
  "folder": "Project Folder Name",
  "conversation": "Conversation Title"
}
```

### Multiple Projects

You can map different coding projects to different ChatGPT conversations:

```json
"projects": {
  "webapp": {
    "folder": "Web Projects",
    "conversation": "React App Help"
  },
  "api": {
    "folder": "Backend Projects", 
    "conversation": "API Design Help"
  },
  "default": "General Coding Help"
}
```

Then agents can escalate to the right context:
```json
{"project": "webapp", "question": "How to optimize React re-renders?"}
{"project": "api", "question": "Best practices for REST pagination?"}
```

## MCP Tools Reference

#### `escalate_to_expert`

Send a question to ChatGPT via the desktop app.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `project` | string | Yes | Project ID from config (use `list_projects` to discover) |
| `reason` | string | Yes | Why you're escalating (helps ChatGPT understand context) |
| `question` | string | Yes | The specific technical question |
| `attempted` | string | No | What you've already tried and the results |
| `projectContext` | string | No | Additional context about the codebase |
| `artifacts` | array | No | Code snippets, logs, or notes (see below) |

**Artifact format:**
```json
{
  "type": "file_snippet" | "log" | "note",
  "pathOrLabel": "src/auth.ts",
  "content": "// the actual code or content"
}
```

#### `list_projects`

Discover available project IDs from your configuration. Call this first if you don't know what projects are available.

**Returns:**
```json
{
  "projects": ["default", "webapp", "api"],
  "count": 3
}
```

## Important Notes

### ChatGPT Conversation Setup

For best results, start each project's ChatGPT conversation with a system prompt that establishes the expert role:

> You are the dedicated expert escalation endpoint for autonomous coding agents working on this project.  
>  
> Your role:  
> - Provide clear, technically correct, implementation-ready guidance.  
> - Assume the agent will immediately act on your instructions.  
> - Avoid asking the agent follow-up questions unless absolutely necessary.  
> - Be concise, direct, and practical.  
>  
> Response Format:  
> 1. Begin with a brief explanation of the issue and the recommended solution.  
> 2. End every response with a strict JSON object in the following format:  
>  
> {  
>   "guidance": "one-sentence summary of what the agent should do next",  
>  "action_plan": ["step 1", "step 2", "step 3"],  
>  "priority": "low | medium | high",  
>  "notes_for_user": "optional message for the human"  
> }  
>  
> Important Rules:  
> - The JSON must be the final content in your message.  
> - Do NOT wrap the JSON in code fences.  
> - Do NOT include any commentary after the JSON.  
> - Do NOT use placeholders or incomplete structures.  
> - Always return syntactically valid JSON.  

### During Use

- **Keep ChatGPT Desktop installed** (it will be opened/closed automatically)
- **Don't interact with ChatGPT** while escalation is in progress
- Automation takes ~30-120 seconds depending on response length
- Works best when you're AFK or focused on other tasks

### Version Compatibility

| ChatGPT Desktop Version | Status | Notes |
|-------------------------|--------|-------|
| 1.2025.112 | ✅ Supported | Last tested Nov 30, 2025 |
| Older versions | ⚠️ Unknown | May work, not tested |
| Future versions | ⚠️ Unknown | May break if UI changes significantly |

If a ChatGPT update breaks automation, [open an issue](../../issues) with your version number.

## What Happens During Escalation

When your agent calls `escalate_to_expert`, the server launches ChatGPT fresh, navigates to the configured conversation, sends the question, waits for completion, copies the response, and returns structured JSON — matching the high‑level flow diagram above. Typical time: **30–120 seconds**.

For implementation details (pixel detection, OCR, copy logic), see `docs/internals-detection.md` and `docs/sidebar-selection.md`.

## Detection Internals

Looking for the low‑level heuristics (sidebar state, response generation, copy button)? They’re documented for contributors in:
- `docs/internals-detection.md`
- `docs/sidebar-selection.md`

## Development

```powershell
# Watch mode
npm run dev

# Build
npm run build
```

## Troubleshooting

### "ChatGPT window not found"
- Make sure ChatGPT Desktop app is installed
- The automation will start it automatically

### "Conversation not found"
- Verify the conversation title in config matches exactly
- Check that the project folder name is correct
- The conversation must exist before first use

### "Response timeout"
- Increase `responseTimeout` in config for longer responses
- Check if ChatGPT is rate-limited or experiencing issues

### OCR not working
```powershell
# Reinstall PaddleOCR
pip install --upgrade paddleocr paddlepaddle
```

### Windows automation issues
```powershell
# Reinstall automation dependencies
pip install --upgrade pywinauto pyperclip pywin32
```

### Logs

Logs are written to stderr and can be captured by your MCP client. Set `logging.level` to `"debug"` in config for verbose output.

### Common Driver Error: NoneType window rect

If you see an error like:

```
TypeError: 'NoneType' object is not subscriptable
```

This typically means the Python driver could not find or access the ChatGPT Desktop window. Try the following:

- Make sure ChatGPT Desktop is open and not minimized
- Set `headless` to `false` in your config if it is `true` (some environments hide the window)
- Move ChatGPT Desktop to your primary monitor and ensure it isn't occluded by other apps
- Confirm the conversation and folder titles match your config exactly
- Run `npm run doctor` to validate the configuration and dependencies
- Re-run the MCP smoke-test: `node tools/mcp_smoke_test.js`

If the issue persists, check the backend logs (stdout/stderr) for more details and open an issue with the log snippet and your ChatGPT Desktop version.

## Verification Checklist

Before your first escalation, confirm:

- [ ] Windows 10 or 11
- [ ] ChatGPT Desktop installed (Microsoft Store version)
- [ ] ChatGPT Desktop opens and you're logged in
- [ ] Created the project folder in ChatGPT (e.g., "Agent Expert Help")
- [ ] Created the conversation inside that folder (e.g., "Copilot Escalations")
- [ ] Conversation title in config matches **exactly** (case-sensitive)
- [ ] Config file exists at `%USERPROFILE%\.chatgpt-escalation\config.json`
- [ ] MCP client configured with correct path to `dist/src/server.js`
- [ ] Node.js 18+ installed (`node --version`)
- [ ] Python 3.10+ installed (`python --version`)
- [ ] Python packages installed (`pip list | findstr pywinauto`)

## FAQ

<details>
<summary><strong>Can I keep working while it runs?</strong></summary>

Yes, but **don't interact with the ChatGPT window**. The automation controls mouse/keyboard input to that specific window. You can use other apps normally.
</details>

<details>
<summary><strong>Can I use this for multiple agents simultaneously?</strong></summary>

No. Only one escalation at a time. If you have multiple agents, they'll queue up and be processed sequentially.
</details>

<details>
<summary><strong>Can this escalate to multiple ChatGPT conversations?</strong></summary>

Yes! Configure multiple projects in your config, each pointing to different folders/conversations. Your agent specifies which project to use.
</details>

<details>
<summary><strong>Will this work on macOS in the future?</strong></summary>

Unlikely. macOS has different automation APIs (Accessibility API) that would require a complete driver rewrite. The Windows-only scope is intentional to keep the project maintainable.
</details>

<details>
<summary><strong>Can I use Ollama or a local LLM instead?</strong></summary>

Not with this tool — it specifically automates the ChatGPT Desktop app. For local LLMs, use a different MCP server that calls Ollama's API directly.
</details>

<details>
<summary><strong>How long does an escalation take?</strong></summary>

Typically 30-120 seconds:
- ~10s to open ChatGPT and navigate
- ~5-90s for ChatGPT to generate response (depends on length)
- ~5s to copy and return
</details>

<details>
<summary><strong>The first run is really slow. Why?</strong></summary>

PaddleOCR downloads its model files (~100MB) on first use. Subsequent runs are much faster, and the model preloads in the background.
</details>

## Uninstall

```powershell
# Remove config directory
Remove-Item -Recurse -Force "$env:USERPROFILE\.chatgpt-escalation"

# Remove from your MCP client config
# (edit your settings.json or claude_desktop_config.json)

# Optionally uninstall Python dependencies
pip uninstall pywinauto pyperclip paddleocr paddlepaddle
```

## Security

This tool **never** automates anything outside the ChatGPT Desktop window. It never reads unrelated windows, captures screens of other apps, or interacts with other applications. All automation is scoped to the ChatGPT process.

## Author

Created by **Darien Hardin** ([@Dazlarus](https://github.com/Dazlarus))

## License

MIT

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for version history.

## Additional Docs

- Protocol probe usage and troubleshooting: `docs/protocol-probe.md`
- Sidebar selection internals and tuning: `docs/sidebar-selection.md`
- Safety guardrails and interruption recovery: `docs/safety-guardrails.md`

## Chaos / Antagonistic Testing

Test safety guardrails by running commands under an antagonist that randomly steals focus, minimizes ChatGPT, moves/clicks the mouse, opens occluding windows, and scrolls.

**Quick commands:**

```pwsh
# Run any command with chaos (60s, medium intensity)
npm run chaos -- <your-command>

# Run protocol probe with aggressive chaos (90s)
npm run chaos:probe

# Run full escalation test under chaos (90s, aggressive)
npm run chaos:escalate
```

**Customize chaos parameters:**

```pwsh
# Gentle chaos for 120 seconds
node tools/with_antagonist.js --duration=120 --intensity=gentle -- npm run probe

# Custom duration and intensity for escalation test
node tools/chaos_escalation_test.js --duration=60 --intensity=medium
```

**Intensities:**
- `gentle`: Fewer disruptions, longer delays between actions
- `medium`: Balanced (default)
- `aggressive`: Heavy focus stealing, frequent minimize/occlude

**What the antagonist does:**
- Random mouse moves and clicks
- Steals focus to Notepad
- Opens Notepad windows on top of ChatGPT
- Minimizes ChatGPT window
- Random scroll events

**Note:** This intentionally disrupts your desktop session. Run on non-critical environments or VMs.

**Chaos escalation test (`npm run chaos:escalate`):**

Runs a full end-to-end test:
1. Starts antagonist (default 90s, aggressive)
2. Connects to MCP server
3. Lists projects
4. Calls `escalate_to_expert` with a test question
5. Validates response
6. Reports pass/fail

This verifies that safety guardrails successfully recover from interruptions during a real escalation flow.

**Current Test Results:**
- ✅ **Gentle**: Passes consistently
- ✅ **Medium**: Passes consistently
- ✅ **Aggressive**: Passes with retry logic (may take 2-4 attempts)

**Seeded Tests**: Use `--seed=12345` for reproducible chaos patterns:
```pwsh
node tools/chaos_escalation_test.js aggressive --duration=120 --seed=99999
```