GUI2MCP

Turn browser-only legacy software into tools that AI can call.

Watch the 27-second demo (MP4)

View the GUI2MCP pitch deck on Canva

Legacy GUI -> GUI2MCP -> generated MCP server -> Codex uses it like a native API

The problem

Companies still depend on internal and legacy software that has no usable API. Humans can search records, create customers, and submit invoices through a browser, but AI agents cannot access those capabilities without reasoning through the entire interface on every run.

Rebuilding the vendor software is expensive. Writing and maintaining one-off automation scripts does not scale.

The insight

The GUI is already a machine-readable description of what the software can do.

GUI2MCP observes that interface, discovers useful actions, compiles them into typed MCP tools, and implements every call by operating the original GUI with Playwright.

For the included API-less CRM, GUI2MCP generates:

search_customer(customer_name)
create_customer(name, email, company)
create_invoice(customer, amount_jpy, description)

Codex no longer needs to find menus, inspect the DOM, or decide which button to press. It calls one domain-level tool:

{
  "name": "create_invoice",
  "arguments": {
    "customer": "Aiko Tanaka",
    "amount_jpy": 120000,
    "description": "AI integration workshop"
  }
}

GUI2MCP opens the real invoice screen, fills the fields, submits the form, verifies the visible result, and returns INV-0001 with a screenshot.

Why not call Playwright directly?

Playwright is the actuator. GUI2MCP is the interface compiler.

For a one-off task, direct Playwright is enough. GUI2MCP is for recurring operations that should become reusable, reviewable, and available to every MCP client.

How it works

flowchart LR
    A[Legacy web GUI<br/>No JSON API] -->|Observe DOM and forms| B[GUI Inspector]
    B -->|Compile labels, fields, actions| C[generated/tools.json]
    C --> D[Generic STDIO<br/>MCP server]
    D -->|tools/list and tools/call| E[Codex]
    E -->|Typed business action| D
    D -->|Replay through Playwright| A
    A -->|Visible result + evidence| D

1. Observe: Playwright crawls a bounded set of same-origin pages.

2. Understand: Headings, labels, required fields, input types, options, and submit actions are extracted.

3. Compile: GUI actions become MCP names, descriptions, JSON Schema, and safety annotations.

4. Execute: The generic MCP runtime loads the generated specification and replays calls through the GUI.

5. Verify: Visible alerts and result tables are returned as structured output with a screenshot.

The generated MCP runtime contains no CRM-specific handlers. Its behavior comes from generated/tools.json.

90-second judge demo

1. Open the Legacy CRM and point out that it exposes browser pages only.

2. Open GUI2MCP Studio and click Discover tools.

3. Show the three generated tools and their typed parameters.

4. Ask Codex:

   Search for the Acme customer, then create a JPY 120,000 invoice for an AI integration workshop.

5. Codex calls search_customer, followed by create_invoice.

6. Open the CRM invoice register and show INV-0001 created through the original GUI.

The demo deliberately uses separate origins:

• GUI2MCP control plane: http://127.0.0.1:4310

• Unmodified legacy target: http://127.0.0.1:4311/legacy

• http://127.0.0.1:4311/api/health returns 404

The JSON endpoints on port 4310 belong to GUI2MCP Studio, not the legacy application.

Run locally

Prerequisites: Node.js 20+ and pnpm.

corepack pnpm install
corepack pnpm exec playwright install chromium
corepack pnpm dev

Open:

• GUI2MCP Studio: http://127.0.0.1:4310

• Legacy CRM: http://127.0.0.1:4311/legacy

In Studio, click Discover tools, select any generated tool card, provide arguments, and choose Run via Playwright.

The Studio supports English and Japanese from the EN / JP control in the top-right corner.

Connect Codex

Generate the tool specification in Studio first. Keep the Studio and legacy target running, then build and register the STDIO server:

corepack pnpm build
codex mcp add gui2mcp -- node "C:\absolute\path\to\GUI2MCP\dist\src\mcp.js"
codex mcp list

Restart Codex and ask:

Search for the Acme customer, then create a JPY 120,000 invoice
for an AI integration workshop.

Project-scoped configuration is also available in .codex/config.toml.example.

CLI workflow

With the demo servers running:

corepack pnpm discover -- --url http://127.0.0.1:4311/legacy
corepack pnpm mcp

Useful environment variables:

• GUI2MCP_SPEC: load another generated tool specification.

• GUI2MCP_HEADFUL=1: show the browser while an MCP tool executes.

• PORT: change the GUI2MCP Studio port.

• LEGACY_PORT: change the demo target port.

Technical highlights

• Dynamic MCP tools/list generated at server startup

• JSON Schema inferred from real form controls

• MCP safety annotations such as readOnlyHint and idempotentHint

• Label-first field resolution with name and ID fallbacks

• Structured extraction of visible tables and success alerts

• Evidence screenshots for every completed tool call

• Browser fallback across bundled Chromium, Edge, and Chrome

• English/Japanese Studio interface

• No model API key required for deterministic discovery and replay

Verification

The repository includes unit and end-to-end MCP checks:

corepack pnpm check
corepack pnpm test
corepack pnpm build
node dist/tests/mcp-smoke.js

The smoke test launches the generated STDIO MCP server, runs tools/list, calls search_customer, and verifies that the result contains Acme Industries.

Prototype scope

This two-hour prototype optimizes for a convincing end-to-end proof, not arbitrary-site automation.

It currently targets conventional server-rendered forms. CAPTCHA, complex SPAs, iframe workflows, arbitrary authentication, and selector self-healing are intentionally out of scope.

The next product step is Teach Workflow:

A human demonstrates a multi-page operation once; GUI2MCP turns that demonstration, its parameters, and its success condition into a governed MCP tool.

That moves the product from form discovery to durable business-workflow compilation.

Repository map

src/discovery.ts       GUI inspection and tool compilation
src/executor.ts        Playwright-based generated tool runtime
src/mcp.ts             Generic STDIO MCP server
src/legacy-crm.ts      Separate-origin, API-less demo target
public/                GUI2MCP Studio
generated/tools.json   Generated machine interface
tests/                 Unit and MCP smoke tests
Plan.md                Japanese implementation plan and demo script

Re-record the demo

corepack pnpm record:demo

This produces English-captioned WebM and H.264 MP4 versions in artifacts/.

An API no longer has to be built by the software vendor. AI can derive a machine interface from the GUI that already exists.