reflex

Screen-reading macros and bots for Windows, written in YAML and driven by people or by AI agents.

reflex watches a window, finds images, colours and text in it, and sends keys, clicks and gamepad input back. It can do that while the window is behind other windows and without touching your mouse, so you can keep using the PC. Scripts run step by step ("wait for the Play button, click it, attack until Victory shows"). Watchers react at frame rate ("whenever the heal icon lights up, press 4"). Both work together.

It also runs as an MCP server, so an AI agent can look at a window, act on it, and hand the fast work to a script it writes. An agent takes seconds to think. A boss telegraph lasts 300 ms. reflex is the part that doesn't think.

reflex doctor
PASS  window capture (works while covered)  384x261, 76.0 fps, p95 18.5 ms
PASS  background keys (PostMessage)         key down/up delivered without focus
PASS  background clicks (PostMessage)       click routed to the child window under the point
PASS  screen capture                        available
PASS  text recognition (OCR)                Windows OCR (en-GB)

Install

Python 3.10+ on Windows 10 or 11.

pip install -e ".[all]"     # OCR (the engine built into Windows) and the MCP server
reflex doctor               # proves capture and input work on your machine

pip install -e . gives you the engine without OCR or MCP. The engine itself is plain Python and runs anywhere. Capture and input are Windows-only.

Related MCP server: OpenSky

Five minutes

Find your window, grab a frame of it, and cut a template out of it:

reflex windows --title "My Game"
reflex capture --title "My Game" -o shot.png
reflex capture --title "My Game" -o img/play.png --region 600,580,80,40

Write a script:

name: daily claim
target:
  window: {title: "My Game"}
  input: postmessage            # background input; use sendinput if the game ignores it
steps:
  - wait_for: {image: img/play.png}
    timeout_s: 20
    as: play
  - click: $play
  - wait_for: {text: {contains: "Reward"}}
  - click: {text: {contains: "Claim"}}
  - log: "claimed"

Check it, try it without sending input, then run it:

reflex validate daily.yaml
reflex run daily.yaml --dry-run
reflex run daily.yaml
reflex run daily.yaml --at 09:00 --at 21:00      # or --every 30m

ctrl+alt+end stops any run. ctrl+alt+home pauses it. Both work from any window, full-screen games included, and stopping releases every key and button reflex was holding.

Scripts

A script is a list of steps. reflex schema prints all 29 with their fields. The ones you'll use most:

Conditions are screen checks, expressions, or both:

{image: img/boss.png}                              # template match (multi-scale, transparent PNGs)
{image: {template: img/coin.png, count_min: 3}}    # at least three on screen
{text: {region: [20, 20, 160, 30], number_below: 300}}   # "HP 240/1200" -> 240 < 300
{color_search: {rgb: [255, 40, 40], region: [0, 0, 1280, 80]}}
{stable: {for_ms: 800}}                            # the screen stopped moving
"${battles < 20 and gold > 500}"
{any_of: [...]}  {all_of: [...]}  {not: ...}

Expressions use ${...}: arithmetic, comparisons, and/or/not, fields of stored matches (${loot.x + 10}) and a few functions (min max abs round int float str len). They're evaluated against a whitelist, so a macro file can't run code.

Mistakes are caught before the script runs, with their location:

FAIL farm.yaml
     farm.yaml: steps[3]<start battle>.clik: a step needs exactly one action, found ['clik'] (did you mean 'click'?)

The validator catches unknown key names (in steps and watcher actions), missing template files, malformed expressions and calls to undefined functions. Values built from ${...} variables can only be checked when they run. Runs are bounded too: a script stops at timeout_s (default one hour), and every wait_for has its own timeout.

humanize: {jitter_ms: 40, click_jitter_px: 2} adds small random delays and click offsets. examples/scripts/auto_battle.yaml uses most of the above.

Watchers

Watchers are the reactive half: a condition, the input to send, and rules for how often. They run on every frame a script grabs, so a script can be waiting for a loading screen while a watcher closes popups.

watchers:
  - id: close_popup
    kind: on_match
    cooldown_s: 1
    params:
      when: {predicate: text, region: [300, 200, 400, 60], contains: "Daily bonus"}
      do: [{click: [680, 215]}]
    expect: {kind: gone, within_ms: 800}     # check the click worked
    trip_after_unverified: 3                 # three misses in a row: stop pressing

After each fire, expect checks that the screen changed the way it should. If the input keeps doing nothing (wrong window, game lagging), the watcher trips instead of spamming. A watcher file on its own runs with reflex run watchers.yaml --title "My Game" --duration 30m.

Several windows at once

reflex run farm.yaml --title "Client" --all-windows

One copy per matching window, in parallel. Background capture and input are what make this work: no instance needs focus.

Recording

reflex record --title "My Game" -o farm.yaml --images

Play normally, then press ctrl+alt+end. You get an ordinary script you can edit. Typing becomes one type step using your real keyboard layout, held keys keep their hold time, press-move-release becomes a drag, and pauses become wait steps.

With --images, each click also saves a 48 px crop of what was under the pointer and clicks that instead of a fixed pixel, so the script survives the window moving. Input reflex sends itself is never recorded.

For AI agents (MCP)

claude mcp add reflex -- reflex mcp --base-dir C:/path/to/macros

19 tools. screenshot returns the window as an image (covered windows too) with the scale to map coordinates back. find_image, find_text, read_text and find_color look things up. click, press, type_text, drag and scroll act, in the background by default. save_template turns a region the agent can see into an image condition.

Then the part that matters. schema, validate_script and run_script (or watch for watchers) let the agent write a script and run it in the background; the call returns a run id at once. The agent polls run_status for the log and result and calls stop_run when it's done. dry_run reports the input a script would have sent.

The tools are plain methods on reflex.agent.ReflexTools, so other agent frameworks can call them without MCP.

Python

from reflex.backends.windows import WindowSource, PostMessageSink, find_window
from reflex.vision import find_image
from reflex.ocr import read_text

win = find_window(title="My Game")
frame = WindowSource(win).grab()              # BGR numpy array of the client area
hits = find_image(frame, "img/coin.png", max_results=10)
hp = read_text(frame, (20, 20, 160, 30)).text
PostMessageSink(win).click(*hits[0].center)

Scripts: reflex.script.load_script() and run_script(). Watchers: reflex.runtime.ReflexRuntime. Predicates, recipes and expectations are plug-ins (reflex list), and other packages can register their own through the reflex.plugins entry point.

Input: which one to use

If clicks or keys do nothing with postmessage, switch to sendinput.

Gamepad: target: {gamepad: true} adds a virtual Xbox 360 controller. pad_a, pad_rt, pad_up and the rest work anywhere a key name does, and stick: left with x/y moves a stick. It needs vgamepad and the ViGEmBus driver, which you install yourself.

Limits, and what reflex won't do

• Minimized windows can't be captured. There's nothing to render. Covered is fine. reflex raises an error rather than handing your conditions black frames.

• A locked PC refuses screen copies and real input. Window capture and background input keep working.

• A few games render black to window capture. reflex notices and falls back to copying the screen area, which needs the window visible.

• reflex sends ordinary Windows input and makes no attempt to hide from anti-cheat. No kernel drivers, no detection evasion, and that won't change. Many online games ban automation in their terms; check before running a bot on an account you care about.

Development

pip install -e ".[all,test]"
pytest                       # the Windows tests drive real windows the suite creates
REFLEX_LIVE_INPUT=1 pytest   # also tests SendInput, which moves your cursor

The tests never touch your own windows. reflex.backends.windows.testwindow makes a real Win32 window that paints known colours and records every message it gets, so capture and input are tested end to end. The replay fixtures are drawn by a script, not captured from a game.

MIT licence.