OmniFlow

基于 MCP (Model Context Protocol) 协议的 Windows 自动化全能工具

English | | 中文| 日本語 | Deutsch

OmniFlow encapsulates Windows desktop automation capabilities (window operations, image/color recognition, OCR, keyboard/mouse simulation, background binding, memory operations, etc.) as a standard MCP Server. Any AI client supporting the MCP protocol (such as Claude Desktop, VS Code, CodeBuddy, Cursor, Windsurf, Continue, Cline, Cody, Crayfish, Hermes, Trae, Kiro, etc.) can directly call Windows automation capabilities.

✨ Features

🎯 Use Cases

OmniFlow turns Windows desktop applications into "AI-callable tools". Common use cases:

• 🎨 Image Processing Automation — Drive Photoshop/AI for batch image processing (resizing, color grading, filters, batch export)

• 📸 Screenshot Pipeline — Screen capture → OCR text extraction / PIL secondary processing / annotation back to screen

• ✏️ Drawing Automation — Use mspaint to draw diagrams, add annotations, flowcharts

• 🎮 Game Automation — Background AFK, auto-battle, dungeon recording and playback

• 📝 Office Automation — WPS/Office document auto-fill, report generation, batch formatting

• 🎬 Video Control — Player auto-pause/resume, subtitle recognition, scheduled recording

🖥️ Window Operations

• Window finding and enumeration (by title, class name, PID, etc.)

• Get window status (position, size, visibility, minimized state, etc.)

• Window topmost / cancel topmost / show / hide

• Window binding and unbinding (prepare for background operations)

🎯 Image & Color Recognition

• Find Image: Search for image position by similarity in specified screen area, supports batch search

• Find Color: Single point color search, multi-point color search, area color search, supports color tolerance

• Compare Color: Compare color at specified coordinate

• Screenshot: Capture specified area and save / return image data

📝 OCR (Optical Character Recognition)

• Find Text: Search for text coordinates on screen based on preset dictionary

• Recognize Text: Recognize text content in specified area and return

• Supports standard characters, multi-color characters, color-shifted characters

• Supports dictionary-free OCR recognition

⌨️🖱️ Keyboard & Mouse Simulation

• Foreground Input: Simulate keyboard keys, key combinations, string input; mouse move / click / scroll

• Background Input: Send background messages to bound windows without stealing focus

• Supports key state control (press / release)

🧠 Background Binding

• Multiple image/color binding modes: gdi, dx, dx2, opengl, etc.

• Multiple keyboard/mouse binding modes: windows, normal, etc.

• Suitable for game background AFK, multi-window concurrent automation

🧬 Memory Operations

• Read / Write process memory

• Memory search and pattern scanning

• Get process module base address

📁 File & System

• File read/write operations

• System information retrieval (CPU usage, memory, screen resolution, etc.)

• Process enumeration and management

🔧 Thread Safety

• Supports multi-threaded concurrent calls

• Each thread has independent window binding, no interference

🔄 Workflow

• Visual / script-based automation task orchestration

• Supports control structures like conditionals, loops, delay waiting, sub-flows

• Task chain execution, output of previous step as input for next step

• Supports workflow saving, loading and reuse

🧩 Plugin System

• Plugin hot-loading to extend OmniFlow capabilities

• Supports custom Tools, custom workflow nodes

• Community plugin ecosystem for sharing and reusing automation capabilities

Related MCP server: computer-control-mcp-lands

🏗️ Architecture

┌────────────────────────────────────────┐
│           AI Client (MCP Host)          │
│  Claude Desktop / CodeBuddy / VS Code   │
│  Cursor / Windsurf / Continue / Cline   │
│  Crayfish / Hermes / Trae / Kiro ...     │
└──────────────┬─────────────────────────┘
               │  MCP Protocol (stdio)
┌──────────────▼─────────────────────────┐
│           OmniFlow MCP Server           │
│  ┌───────────────────────────────────┐  │
│  │         MCP Tools Layer           │  │
│  │  window / image / text / input    │  │
│  │  / binding / memory / system ...  │  │
│  └──────────────┬────────────────────┘  │
│  ┌──────────────▼────────────────────┐  │
│  │        Automation Engine          │  │
│  │     Win32 API / COM / ctypes      │  │
│  └──────────────┬────────────────────┘  │
└─────────────────┼───────────────────────┘
┌─────────────────▼───────────────────────┐
│          Windows System Layer           │
│     GDI / DX / Win32 API               │
└────────────────────────────────────────┘

📥 Installation

Prerequisites

• Windows OS

• Python 3.10+

Installation Steps

# 1. Clone repository
git clone <this-repo-url>
cd OmniFlow

# 2. Create virtual environment
python -m venv .venv
.venv\Scripts\activate   # Windows
# source .venv/bin/activate  # Linux/Mac (some features not supported)

# 3. Install dependencies (Important: must use `pip install -e .` to make omniflow package importable via `python -m omniflow`)

# Method 1: requirements.txt + editable install (Recommended)
pip install -r requirements.txt
pip install -e .

# Method 2: One-liner
pip install -r requirements.txt && pip install -e .

🚀 Quick Start

Configure MCP Client

Add OmniFlow to your MCP client configuration file:

{
  "mcpServers": {
    "omniflow": {
      "command": "<venv-python>",
      "args": ["-m", "omniflow"],
      "cwd": "<local OmniFlow repo path>"
    }
  }
}

Notes:

• command points to python.exe in venv (e.g., D:/OmniFlow/.venv/Scripts/python.exe)

• cwd points to OmniFlow repository root directory

• Some MCP clients support the cwd field (e.g., Hermes); for those that don't, use env.PYTHONPATH instead

Example: Let AI Find an Image and Click It

Through natural language description in AI client, OmniFlow automatically calls corresponding MCP Tools:

1. Capture screen → screenshot

2. Find image in screenshot → find_image

3. Move mouse and click → mouse_click

AI clients automatically orchestrate these MCP Tool calls.

📖 MCP Tools Reference

Window Tools

Binding Tools

Image Tools

Text Tools

Input Tools

Memory Tools

System Tools

Workflow Tools

Plugin Tools

📁 Project Structure

OmniFlow/
├── .gitignore               # Git ignore configuration
├── README.md                # Project documentation (Chinese)
├── README.en.md             # Project documentation (English)
├── README.ja.md             # Project documentation (Japanese)
├── README.de.md             # Project documentation (German)
├── OmniFlow 使用示例.md      # Usage examples (by scenario: PS / screenshot / drawing)
├── requirements.txt         # Python dependencies
├── pyproject.toml          # Project metadata
├── scripts/                 # Helper scripts
├── src/
│   └── omniflow/
│       ├── __init__.py
│       ├── __main__.py       # Entry point (python -m omniflow)
│       ├── server.py         # MCP Server main logic
│       ├── engine/
│       │   ├── __init__.py
│       │   ├── com.py        # COM / Win32 API wrapper
│       │   └── types.py      # Type definitions
│       └── tools/
│           ├── __init__.py
│           ├── window.py     # Window Tools (includes window_close)
│           ├── binding.py    # Binding Tools
│           ├── image.py      # Image/Color Recognition Tools
│           ├── ocr.py        # OCR Tools
│           ├── input.py      # Keyboard/Mouse Simulation Tools
│           ├── memory.py     # Memory Operation Tools
│           ├── system.py     # System Tools (includes run_program)
│           ├── workflow.py   # Workflow Tools (v2 engine: IF / WAIT_FOR_WINDOW / variable system)
│           └── plugin.py     # Plugin System Tools
├── examples/
│   ├── open_photoshop.py    # Photoshop automation example
│   └── workflows/
│       └── infinite_fish_auto_sell.json   # Complete v2 engine workflow example (self-contained)
└── tests/
    ├── __init__.py
    ├── test_tools.py        # Basic tool tests
    └── test_workflow_v2.py  # Workflow v2 engine regression tests (8 cases)

Note: OmniFlow engine v2 improvements (variable system / IF node / WAIT_FOR_WINDOW / safe condition / true MCP calls) can be found in docs/workflows/translation-notes.md and the OmniFlow 0.2.0 entry in known-integrations.md.

🛠️ Development

# Install development dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Code formatting
ruff format src/
ruff check src/

📄 License

MIT License

⚠️ Disclaimer

This tool is for legal purposes only (such as automation testing, office automation, etc.). Do not use it for scenarios that violate game terms of service, infringe on others' rights, or engage in any illegal activities. Users are responsible for their own actions.