<p align="center">
  <img src="assets/brand-mark.svg" width="120" alt="SoftAuto logo">
</p>

<h1 align="center">Lingheyi SoftAuto — Open-source Windows RPA & MCP Server</h1>

<p align="center">
  <strong>Desktop UI Automation, Browser Automation, and ERP Automation for AI Agents</strong><br>
  Let Computer Use understand the interface. Let SoftAuto execute it fast and reliably.
</p>

<p align="center">
  <a href="https://github.com/guangfubill-crypto/SoftAuto-MCP/actions/workflows/ci.yml"><img src="https://github.com/guangfubill-crypto/SoftAuto-MCP/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://github.com/guangfubill-crypto/SoftAuto-MCP/releases/latest"><img src="https://img.shields.io/github/v/release/guangfubill-crypto/SoftAuto-MCP" alt="Release"></a>
  <img src="https://img.shields.io/badge/platform-Windows%2010%2F11-0078D4" alt="Windows 10/11">
  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-2FA36B" alt="MIT License"></a>
</p>

<p align="center">
  <a href="https://github.com/guangfubill-crypto/SoftAuto-MCP/releases/latest"><strong>Download for Windows</strong></a>
  · <a href="README_ZH.md">简体中文</a>
</p>

<p align="center">
  <img src="https://github.com/guangfubill-crypto/SoftAuto-MCP/releases/download/v0.5.3/SoftAuto-0.5.3-responsive-narrow-en.png" alt="SoftAuto English interface" width="760">
</p>

## What is SoftAuto?

SoftAuto is an open-source Windows RPA application and MCP server for AI agents. It locates desktop controls through Microsoft UI Automation (UIA), locates web elements through the Chrome DOM, and exposes click, type, read, highlight, and validation actions as Model Context Protocol (MCP) tools. It is designed for ERP automation, desktop automation, browser automation, GUI testing, and repeatable business workflows.

SoftAuto acts as a deterministic execution layer for Computer Use. Capture and validate an element once, then let an Agent invoke it by name without repeating screenshot analysis and coordinate clicking for every step.

## The problem

Computer Use can operate almost any application, but repeatedly running the loop of screenshot,
visual reasoning, coordinate clicking, and another screenshot is slow and sensitive to window
movement or layout changes.

SoftAuto separates understanding from execution. A person captures and validates an element once.
The Agent can then invoke that named element through MCP, using Windows UIA or browser DOM actions
instead of repeating visual inference for every step.

| | Traditional Computer Use | SoftAuto + MCP |
|---|---|---|
| Targeting | Re-analyze screenshots and coordinates | Reuse a validated UIA/DOM locator |
| Execution | Vision model → coordinate input | Agent → MCP → native element action |
| Window movement | Coordinates may fail | Stable properties are resolved again |
| Dynamic text | Requires another visual pass | Wildcards, prefixes, and MCP variables |
| Best fit | Unknown interfaces and exploration | ERP, line-of-business apps, repeatable workflows |

```mermaid
flowchart LR
    A[Capture and validate once] --> B[(Project element library)]
    B --> C[SoftAuto MCP]
    D[Agent understands and decides] --> C
    C --> E[Windows UIA]
    C --> F[Browser DOM]
    E --> G[Desktop applications]
    F --> H[Web applications]
```

## Highlights

- Capture and highlight desktop or web elements with `Ctrl + left-click`.
- Organize elements in project-scoped folder trees and move projects between computers.
- Edit recommended locator properties, wildcards, prefixes, and `${variable}` values. Runtime `ProcessId` and `NativeWindowHandle` remain diagnostic-only and cannot be selected for matching.
- Resolve elements again after applications restart or windows move.
- Expose 19 MCP query and action tools, including find, highlight, click, input, focus, and read.
- Use Windows UIA and browser DOM backends from one element library; desktop capture automatically falls back through the bundled FlaUI UIA3 → UIA2 bridge when the primary provider cannot inspect a control.
- Switch between Simplified Chinese and English; responsive UI supports Windows DPI scaling.
- No arbitrary shell execution and no unrestricted coordinate clicking through MCP.

## How the element-to-MCP workflow works

SoftAuto is intentionally split into two phases:

1. **Capture phase (human/RPA-style):** create a project, pick a desktop or web element with `Ctrl + left-click`, review the recommended properties, and press **Validate**. The element is stored in the current project's tree.
2. **Execution phase (Agent/MCP):** the Agent calls an element by name or path. SoftAuto resolves the saved locator and performs the native UIA or DOM action without another screenshot/vision loop.

The recommended locator properties are chosen for stability. `AutomationId`, control type, framework, window title, and an ancestor path are preferred when available. Volatile `ProcessId` and `NativeWindowHandle` values are diagnostics only and are deliberately excluded from matching, because they change after a restart. For dynamic labels, use a name prefix, `*`/`?` wildcards, or a runtime variable such as `${reference}`.

Each project owns its own folders and element library. Export the project JSON to move it to another Windows computer; after importing, validate the elements against the target application. For difficult legacy controls, use the bundled FlaUI bridge (UIA3 with UIA2 fallback) and the **Deep Inspect** action.

## Agent execution and performance

Use `run_sequence` to send a group of dependent actions in one MCP request. This keeps the Agent in the decision loop while removing unnecessary per-click round trips:

```json
{
  "steps": [
    {"action": "set_value", "element": "Staffnumber", "value": "admin"},
    {"action": "set_value", "element": "Password", "value": "admin"},
    {"action": "click", "element": "Sign In"}
  ],
  "confirm": true
}
```

Use `confirm: true` for action tools. Keep `highlight_element` and `validate_saved_element` for capture, diagnostics, and recovery; normal production runs should call the action directly. SoftAuto does not execute arbitrary shell commands or allow MCP to bypass the saved element library with unrestricted coordinates.

## Example case: TrainingOrderSystem ERP batch entry

The repository includes a complete [TrainingOrderSystem example](examples/training-order). It models a common legacy ERP flow:

1. Log in with `admin` / `admin`.
2. Select `Option 1` and click `Go` before each order.
3. Read `Orders.csv`, fill Product Code, Quantity, Unit Price, and Cost Centre, then submit.
4. Read the dynamically generated Order Reference.
5. Click `Continue` to return to the selection page and process the next row.

The example deliberately does **not** use an Option 2 recovery shortcut. It stops on the first failed stage and reports the row and action, so a missed order cannot be silently skipped. Its declarative workflow is in [`examples/training-order/workflow.json`](examples/training-order/workflow.json), and the latest benchmark is in [`benchmark-latest.json`](examples/training-order/benchmark-latest.json).

The latest five-row run completed in **45.493 seconds** (about **8.7 seconds per row**) without Computer Use. The [v0.5.7 Release](https://github.com/guangfubill-crypto/SoftAuto-MCP/releases/tag/v0.5.7) contains a ready-to-run `SoftAuto-Demo-Bundle-0.5.7.zip` with the installer, training application, CSV, captured element library, workflow, and Codex Skill.

## Troubleshooting legacy applications

- **Validation succeeds but a later run fails:** re-check that the project points to the intended application window; do not select process IDs or native handles as locator properties.
- **A changing label cannot be found:** replace the full `Name` with a stable prefix or wildcard and pass dynamic text through `variables`.
- **The first action is slow:** use `run_sequence` for consecutive actions and avoid a preflight highlight on every step.
- **A WinForms/ERP control is not exposed by the primary provider:** run **Deep Inspect** or let the bundled FlaUI UIA3 → UIA2 fallback inspect it.
- **MCP connects but actions are rejected:** use the installed path shown by **MCP Config** and keep `confirm: true`; action execution is intentionally gated.

## Quick start

### Windows installer

1. Download `Lingheyi-SoftAuto-Setup-*.exe` from [Releases](https://github.com/guangfubill-crypto/SoftAuto-MCP/releases/latest).
2. Install SoftAuto and create a project and folder.
3. Click **Pick Desktop Element** or **Pick Web Element**, then press `Ctrl + left-click` on the target.
4. Adjust locator properties and click **Validate**.
5. Click **MCP Config** and paste the copied configuration into an MCP-compatible Agent.

### Run from source

Requirements: Windows 10/11 and Python 3.11–3.13.

```powershell
uv sync --extra dev
uv run softauto-inspector
```

Run the MCP server directly:

```powershell
$env:SOFTAUTO_ALLOW_ACTIONS = "1"
uv run mcp run src/softauto/server.py:mcp
```

## MCP configuration

The installed application can copy the exact configuration for the current computer. A typical
configuration looks like this:

```json
{
  "mcpServers": {
    "software-automation": {
      "command": "C:\\Users\\<user>\\AppData\\Local\\Programs\\SoftAuto\\mcp\\SoftAutoMCP.exe"
    }
  }
}
```

MCP follows the project currently selected in SoftAuto. Set `SOFTAUTO_ELEMENT_LIBRARY` when you
want it to use an exported standalone element library instead.

## Development

```powershell
uv sync --frozen --extra dev
uv run ruff check src tests
uv run pytest -q
```

The current release has 47 automated tests. See [CONTRIBUTING.md](CONTRIBUTING.md) before sending a
pull request.

## Security and scope

SoftAuto is designed for systems and accounts you are authorized to automate. Saved locators may
contain application titles, URLs, and control metadata, so review exported libraries before sharing
them. See [SECURITY.md](SECURITY.md) for the action policy and vulnerability reporting process.

## License

SoftAuto's original source code is available under the [MIT License](LICENSE). Bundled and optional
third-party components retain their own licenses; see [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).