SoftAuto

What is SoftAuto

SoftAuto is an open-source Windows RPA and MCP Server designed for AI Agents. It locates desktop software elements through Microsoft UI Automation (UIA) and web page elements through the Chrome DOM, then exposes click, input, read, highlight, and validate operations as Model Context Protocol (MCP) tools. It is suitable for ERP automation, desktop software automation, web business system automation, GUI testing, and repetitive office workflows.

It is not another macro tool that relies on screenshots and coordinates, but rather the deterministic execution layer for Computer Use: elements only need to be picked and validated once, and the Agent can then call them by name, reducing repeated screenshots, visual reasoning, and coordinate clicks.

Related MCP server: Browser[X]MCP

Core Pain Points Solved

Computer Use can operate almost any software, but if every step goes through "screenshot → visual reasoning → coordinate click → screenshot again," the workflow is slow and easily affected by window movement, resolution, and UI changes.

SoftAuto splits this execution chain: the first time, a human picks, validates, and saves elements like using RPA; afterward, the Agent calls Windows UIA or browser DOM actions directly by element name through MCP. The agent continues to handle understanding and decision-making, while repetitive software operations follow the deterministic element automation path.

flowchart LR
    A[首次拾取并验证元素] --> B[(项目元素库)]
    B --> C[SoftAuto MCP]
    D[Agent 理解任务并决策] --> C
    C --> E[Windows UIA]
    C --> F[Browser DOM]
    E --> G[桌面软件]
    F --> H[网页系统]

Key Capabilities

• Ctrl + Left-click to pick and highlight desktop or web elements, managing the element library by project and folder.

• The system recommends stable properties; you can also edit property values, wildcards, and ${variable} dynamic variables.

• After reopening or moving a window, elements can still be re-located by window anchor, control path, and target characteristics.

• MCP provides 19 query and action tools, including find, highlight, click, input, focus, and read.

• Dual channels for Windows UIA and browser DOM; the web extension can be installed with one click from within the software.

• Project element libraries can be imported, exported, and migrated to other Windows computers.

• Simplified Chinese / English instant switching; the interface adapts to resolution and Windows DPI.

• No arbitrary command execution, and MCP is not allowed to bypass the element library for arbitrary coordinate clicks.

Five-Minute Quick Start

Installer Version (Recommended)

1. Download Lingheyi-SoftAuto-Setup-*.exe from Releases.

2. Install and open SoftAuto, then create a project and folder.

3. Click "Pick Desktop Element" or "Pick Web Element," move to the target control, and press Ctrl + Left-click to save.

4. Adjust the locating properties on the right and click "Validate."

5. Click "MCP Configuration" and paste the copied configuration into an MCP-capable Agent.

Running from Source

uv sync --extra dev
uv run softauto-inspector

Workflow:

1. Select the current project at the top; clicking "New Project" creates a completely independent empty element library.

2. The first time you use web mode, click "Install Web Extension"; the software will open the Chrome extensions page, open the extension directory, and copy the path. Follow the prompts to enable Developer Mode and select "Load unpacked."

3. Click "New Folder" to create levels such as application, page, and module; you can continue creating subfolders within folders.

4. Select the target folder, then click "Pick Desktop Element" or "Pick Web Element."

5. Move the mouse; the target element will show an orange real-time border.

6. Press Ctrl + Left mouse button; the click will be intercepted and not passed to the target software.

7. Enter a custom name and save to the current folder.

8. After selecting an element in the element library, check the properties to participate in locating directly in the "Element Details" panel on the right.

9. The system pre-selects stable properties marked with ★. For dynamic text, NamePrefix is selected by default, not the full Name containing business values; users can add or remove properties themselves and click "Save Properties" or "Validate."

10. Select an element in the element library and click "Validate"; the system will re-locate and highlight the element.

11. Both folders and elements can be renamed by double-clicking, and can be dragged and dropped to other folders; non-empty folders will not be accidentally deleted.

12. Click "Export Project" to generate a portable JSON; after installing SoftAuto on another computer, click "Import Project" to migrate all folders, elements, and locators.

13. Click "Copy MCP Configuration" to get the complete Agent configuration with SoftAutoMCP.exe installed.

The project registry is stored in data/projects.json, and each project's element library is stored independently in data/projects/<project ID>/elements.json. The legacy data/elements.json is copied to the first-created "Default Project," and the original file is kept. MCP follows the picker's current project by default; you can also point SOFTAUTO_ELEMENT_LIBRARY to the JSON generated by "Export for MCP." MCP can retrieve elements using stable IDs, unique names, or full paths (for example, Training System/Login/Account).

Property values can be edited directly and support:

• Wildcards: Please take note of your order reference: *

• Single-character wildcards: Order ?

• MCP variables: Order reference: ${reference}

Locators containing ${reference} pass values through variables when calling get_element, validate_saved_element, highlight_element, or action tools:

{
  "variables": {
    "reference": "842"
  }
}

Deep Inspection

When normal picking cannot distinguish complex controls, click "Deep Inspection" to launch the mature FlaUInspect.

uv run softauto-flauinspect

The project includes the SHA-256 verified official FlaUInspect 3.1.0 Release.

Download package checksum: a3e571e783af1cf84661332e5229714b7ac81f908ea44b454e5c1aa0398e20d5.

Building the Windows Installer

After installing PyInstaller and Inno Setup 6, run:

uv sync --extra packaging
.\packaging\build_installer.ps1

The build result is Lingheyi-SoftAuto-Setup-0.5.3.exe. The installer includes:

• SoftAuto.exe: Element picker and project element library interface.

• mcp/SoftAutoMCP.exe: stdio MCP service that Agents can connect to directly.

• FlaUInspect: Deep element inspection tool installed with the GUI.

• Lingheyi Web Connector: Manifest V3 Chrome extension directory installed with the GUI.

Installer user data is stored by default in %LOCALAPPDATA%\SoftAuto\data; uninstalling or upgrading over the top will not delete project elements. The 安装与MCP配置.txt file in the installation directory provides MCP configuration examples.

Connecting the Full MCP Stack

mcp-stack.json is configured with:

• This project's Windows UIA MCP

• Microsoft Playwright MCP 0.0.79

• Ui.Vision MCP Bridge 1.1.1

Merge the mcpServers from that file into your Agent's MCP configuration. Playwright provides the browser Accessibility Snapshot and action tools directly to the Agent; Ui.Vision requires installing the browser extension and completing local pairing first.

OpenAdapt requires the user's own workflow bundle, so mcp-openadapt.example.json is provided separately; fill in the bundle directory before merging.

Starting the Windows UIA MCP Standalone

Read-only mode:

uv run mcp run src/softauto/server.py:mcp

Explicitly allow changing the UI:

$env:SOFTAUTO_ALLOW_ACTIONS = "1"
uv run mcp run src/softauto/server.py:mcp

MCP client configuration:

{
  "mcpServers": {
    "software-automation": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "C:/absolute/path/to/software-automation-mcp",
        "mcp",
        "run",
        "src/softauto/server.py:mcp"
      ]
    }
  }
}

MCP Tools

Read-only tools:

• automation_status

• integration_catalog

• list_saved_elements

• list_element_tree

• list_element_projects

• get_active_element_project

• get_saved_element

• validate_saved_element

• list_windows

• inspect_point

• get_element

• get_children

• find_elements

• highlight_element

Action tools:

• focus_element

• click_element

• invoke_element

• set_element_value

• send_element_keys

Action tools require the server to be started with SOFTAUTO_ALLOW_ACTIONS=1; Click, Invoke, SetValue, and SendKeys also require explicitly passing confirm=true on every call.

Locator Examples

{
  "backend": "windows-uia",
  "version": 1,
  "window": {
    "name": "业务系统",
    "class_name": "MainWindow",
    "control_type": "WindowControl"
  },
  "path": [
    {
      "automation_id": "orderPanel",
      "control_type": "PaneControl",
      "sibling_index": 0
    },
    {
      "automation_id": "saveButton",
      "name": "保存",
      "control_type": "ButtonControl",
      "sibling_index": 0
    }
  ],
  "target": {
    "automation_id": "saveButton",
    "name": "保存",
    "control_type": "ButtonControl"
  }
}

Locating prioritizes AutomationId, then combines name, ClassName, ControlType, FrameworkId, and structural path. Process ID and window handle are only weak hints, because they change when the software restarts.

Backend Routing Principles

The Agent should choose in this order:

1. Browser pages: Playwright MCP.

2. Windows native software: Windows UIA MCP, with FlaUInspect for manual exploration.

3. Element tree not visible: Ui.Vision OCR/image locating.

4. Stable repetitive business processes: compile into an OpenAdapt workflow, then call through the OpenAdapt Agent MCP.

Known Limitations

Standard Win32, WPF, WinForms, and UWP/Electron accessibility quality is generally good. The following interfaces may not return a meaningful element tree:

• Custom-drawn Canvas, DirectX/OpenGL interfaces

• Games and some industrial software

• Remote applications in Citrix, RDP, or video streams

• Legacy controls that do not implement Accessibility

• Software with higher privileges than the Inspector/MCP process

These scenarios are delegated to the integrated Ui.Vision/OpenAdapt backends; this project does not re-implement a vision engine.