# GUI2MCP

### Turn browser-only legacy software into tools that AI can call.

[![Node.js 20+](https://img.shields.io/badge/Node.js-20%2B-339933?logo=node.js&logoColor=white)](https://nodejs.org/)
[![Model Context Protocol](https://img.shields.io/badge/MCP-STDIO-5A45FF)](https://modelcontextprotocol.io/)
[![Playwright](https://img.shields.io/badge/Playwright-powered-2EAD33?logo=playwright&logoColor=white)](https://playwright.dev/)
[![Hackathon Prototype](https://img.shields.io/badge/status-hackathon%20prototype-ff5c35)](#prototype-scope)

![GUI2MCP Studio showing generated CRM tools and a real invoice created in the legacy application](artifacts/gui2mcp-studio.png)

**[Watch the 27-second demo (MP4)](artifacts/gui2mcp-demo-before-teach.mp4)**

**[View the GUI2MCP pitch deck on Canva](https://canva.link/piada3pdccjjl5v)**

> Legacy GUI -> GUI2MCP -> generated MCP server -> Codex uses it like a native API

## The problem

Companies still depend on internal and legacy software that has no usable API. Humans can search records, create customers, and submit invoices through a browser, but AI agents cannot access those capabilities without reasoning through the entire interface on every run.

Rebuilding the vendor software is expensive. Writing and maintaining one-off automation scripts does not scale.

## The insight

**The GUI is already a machine-readable description of what the software can do.**

GUI2MCP observes that interface, discovers useful actions, compiles them into typed MCP tools, and implements every call by operating the original GUI with Playwright.

For the included API-less CRM, GUI2MCP generates:

```text
search_customer(customer_name)
create_customer(name, email, company)
create_invoice(customer, amount_jpy, description)
```

Codex no longer needs to find menus, inspect the DOM, or decide which button to press. It calls one domain-level tool:

```json
{
  "name": "create_invoice",
  "arguments": {
    "customer": "Aiko Tanaka",
    "amount_jpy": 120000,
    "description": "AI integration workshop"
  }
}
```

GUI2MCP opens the real invoice screen, fills the fields, submits the form, verifies the visible result, and returns `INV-0001` with a screenshot.

## Why not call Playwright directly?

Playwright is the actuator. GUI2MCP is the **interface compiler**.

| Direct browser control | GUI2MCP |
| --- | --- |
| Re-observe and reason about the page every run | Discover once, call a stable domain tool repeatedly |
| Low-level `navigate`, `snapshot`, `fill`, and `click` calls | One `create_invoice(...)` call |
| Broad browser permissions | Explicit allow-listed business capabilities |
| UI details consume agent context | Small JSON Schema exposed to the agent |
| Workflow knowledge stays in a prompt | Workflow becomes a portable MCP contract |
| Success is inferred each time | Visible success state and structured results are captured |

For a one-off task, direct Playwright is enough. GUI2MCP is for recurring operations that should become reusable, reviewable, and available to every MCP client.

## How it works

```mermaid
flowchart LR
    A[Legacy web GUI<br/>No JSON API] -->|Observe DOM and forms| B[GUI Inspector]
    B -->|Compile labels, fields, actions| C[generated/tools.json]
    C --> D[Generic STDIO<br/>MCP server]
    D -->|tools/list and tools/call| E[Codex]
    E -->|Typed business action| D
    D -->|Replay through Playwright| A
    A -->|Visible result + evidence| D
```

1. **Observe:** Playwright crawls a bounded set of same-origin pages.
2. **Understand:** Headings, labels, required fields, input types, options, and submit actions are extracted.
3. **Compile:** GUI actions become MCP names, descriptions, JSON Schema, and safety annotations.
4. **Execute:** The generic MCP runtime loads the generated specification and replays calls through the GUI.
5. **Verify:** Visible alerts and result tables are returned as structured output with a screenshot.

The generated MCP runtime contains **no CRM-specific handlers**. Its behavior comes from [`generated/tools.json`](generated/tools.json).

## 90-second judge demo

1. Open the Legacy CRM and point out that it exposes browser pages only.
2. Open GUI2MCP Studio and click **Discover tools**.
3. Show the three generated tools and their typed parameters.
4. Ask Codex:

   > Search for the Acme customer, then create a JPY 120,000 invoice for an AI integration workshop.

5. Codex calls `search_customer`, followed by `create_invoice`.
6. Open the CRM invoice register and show `INV-0001` created through the original GUI.

The demo deliberately uses separate origins:

- GUI2MCP control plane: `http://127.0.0.1:4310`
- Unmodified legacy target: `http://127.0.0.1:4311/legacy`
- `http://127.0.0.1:4311/api/health` returns `404`

The JSON endpoints on port 4310 belong to GUI2MCP Studio, not the legacy application.

## Run locally

Prerequisites: Node.js 20+ and pnpm.

```powershell
corepack pnpm install
corepack pnpm exec playwright install chromium
corepack pnpm dev
```

Open:

- GUI2MCP Studio: [http://127.0.0.1:4310](http://127.0.0.1:4310)
- Legacy CRM: [http://127.0.0.1:4311/legacy](http://127.0.0.1:4311/legacy)

In Studio, click **Discover tools**, select any generated tool card, provide arguments, and choose **Run via Playwright**.

The Studio supports English and Japanese from the `EN / JP` control in the top-right corner.

## Connect Codex

Generate the tool specification in Studio first. Keep the Studio and legacy target running, then build and register the STDIO server:

```powershell
corepack pnpm build
codex mcp add gui2mcp -- node "C:\absolute\path\to\GUI2MCP\dist\src\mcp.js"
codex mcp list
```

Restart Codex and ask:

```text
Search for the Acme customer, then create a JPY 120,000 invoice
for an AI integration workshop.
```

Project-scoped configuration is also available in [`.codex/config.toml.example`](.codex/config.toml.example).

## CLI workflow

With the demo servers running:

```powershell
corepack pnpm discover -- --url http://127.0.0.1:4311/legacy
corepack pnpm mcp
```

Useful environment variables:

- `GUI2MCP_SPEC`: load another generated tool specification.
- `GUI2MCP_HEADFUL=1`: show the browser while an MCP tool executes.
- `PORT`: change the GUI2MCP Studio port.
- `LEGACY_PORT`: change the demo target port.

## Technical highlights

- Dynamic MCP `tools/list` generated at server startup
- JSON Schema inferred from real form controls
- MCP safety annotations such as `readOnlyHint` and `idempotentHint`
- Label-first field resolution with name and ID fallbacks
- Structured extraction of visible tables and success alerts
- Evidence screenshots for every completed tool call
- Browser fallback across bundled Chromium, Edge, and Chrome
- English/Japanese Studio interface
- No model API key required for deterministic discovery and replay

## Verification

The repository includes unit and end-to-end MCP checks:

```powershell
corepack pnpm check
corepack pnpm test
corepack pnpm build
node dist/tests/mcp-smoke.js
```

The smoke test launches the generated STDIO MCP server, runs `tools/list`, calls `search_customer`, and verifies that the result contains `Acme Industries`.

## Prototype scope

This two-hour prototype optimizes for a convincing end-to-end proof, not arbitrary-site automation.

It currently targets conventional server-rendered forms. CAPTCHA, complex SPAs, iframe workflows, arbitrary authentication, and selector self-healing are intentionally out of scope.

The next product step is **Teach Workflow**:

> A human demonstrates a multi-page operation once; GUI2MCP turns that demonstration, its parameters, and its success condition into a governed MCP tool.

That moves the product from form discovery to durable business-workflow compilation.

## Repository map

```text
src/discovery.ts       GUI inspection and tool compilation
src/executor.ts        Playwright-based generated tool runtime
src/mcp.ts             Generic STDIO MCP server
src/legacy-crm.ts      Separate-origin, API-less demo target
public/                GUI2MCP Studio
generated/tools.json   Generated machine interface
tests/                 Unit and MCP smoke tests
Plan.md                Japanese implementation plan and demo script
```

## Re-record the demo

```powershell
corepack pnpm record:demo
```

This produces English-captioned WebM and H.264 MP4 versions in `artifacts/`.

---

**An API no longer has to be built by the software vendor. AI can derive a machine interface from the GUI that already exists.**