Skip to main content
Glama

ElfDeckAI

tests PyPI License: MIT Python

Let an AI play games on your Steam Deck.

ElfDeckAI gives an AI assistant such as Claude a virtual controller, keyboard and mouse on your Steam Deck, plus the ability to see the screen. The AI looks at the game, decides what to do, and presses the buttons itself. It works in both Gaming Mode and Desktop Mode, and on other Linux machines too.

Start here: Steam Deck (about 5 minutes)

You need a Steam Deck, and a PC on the same Wi-Fi with an AI app that supports MCP, such as Claude Desktop.

1. Install on the Deck. Switch the Deck to Desktop Mode (Steam button → Power → Switch to Desktop), open Konsole, and paste:

curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/install-on-deck.sh | sh

At the end it prints the Deck's IP address. Keep it for step 3.

2. Turn on SSH on the Deck (once), so your PC can reach it. In the same Konsole window:

passwd                            # choose a password for the Deck, if you haven't
sudo systemctl enable --now sshd

3. Connect your PC. On the PC, paste one line. It asks for the Deck's IP address and its password (once), then sets everything up, including Claude Desktop.

  • Windows (in PowerShell):

    irm https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/setup-pc.ps1 | iex
  • Mac or Linux (in Terminal):

    curl -fsSL https://raw.githubusercontent.com/Garthnesss/ElfDeckAI/main/setup-pc.sh | sh

4. Try it. Restart Claude Desktop and ask: "Take a screenshot of my Steam Deck." When you see your Deck's screen, start a game and ask the AI to play.

Tips

  • Turn-based games, RPGs, puzzles and menus work best. The AI needs a second or more per move, so fast action games are out of reach.

  • In Gaming Mode, if a game ignores the AI's controller, ask the AI to make its controller Player 1. It knows how, using Steam's Reorder Controllers menu while the game runs.

  • Something not working? Give AI_SETUP_GUIDE.md to your AI and ask it to help. It covers setup and troubleshooting step by step.

Related MCP server: dual-wield-mcp

How it works

Three small programs on the Deck create virtual input devices through the Linux kernel's /dev/uinput, so games and Steam see them as real hardware:

  • ElfKey: a virtual keyboard

  • ElfPad: a virtual Xbox 360-class gamepad (buttons, analog triggers, two sticks, D-pad)

  • ElfMouse: a virtual mouse, which can also click an exact point on the screen

A fourth program, an MCP server, exposes them, together with a screenshot tool, as tools any MCP client can call. Your PC's AI app starts that server on the Deck over SSH. Nothing depends on a particular game or emulator: ElfDeckAI has played Pokémon FireRed in RetroArch on a physical Steam Deck, from the intro through the first rival battle, with the agent seeing only screenshots. See notes/driving.md for what was learned.

Everything is single-file Python using only the standard library, and the Deck install needs no sudo.

Other Linux machines

pip install elfdeckai

elfkey &
elfpad &
elfmouse &

echo "key x 150"     > /tmp/elfkey.fifo     # tap X for 150 ms
echo "press a"       > /tmp/elfpad.fifo     # tap A
echo "move 300 -100" > /tmp/elfmouse.fifo   # move the pointer
echo "click left"    > /tmp/elfmouse.fifo   # left click

grep -E "ElfKey|ElfPad|ElfMouse" /proc/bus/input/devices   # confirm the devices exist

Without pip, clone the repository and run python3 elfkey.py (and so on) directly. Runtime state lives in /tmp and is cleared on reboot.

/dev/uinput is user-writable on the Steam Deck by default. On other distributions, install the udev rule first (see Installation).

MCP server

elfdeckai-mcp is a Model Context Protocol server (stdio transport) that exposes 22 tools:

Group

Tools

Keyboard

key_tap, key_down, key_up, key_release

Gamepad

pad_press, pad_down, pad_up, pad_stick, pad_trigger, pad_center, pad_reset

Mouse

mouse_click_at, mouse_move_to, mouse_move, mouse_click, mouse_down, mouse_up, mouse_scroll, mouse_hscroll, mouse_release

Vision

screenshot

Timing

wait

screenshot captures the screen of the machine the server runs on and returns a PNG, so an agent can work in a look, act, look again loop without a separate video stream. It supports the Steam Deck in both Gaming Mode (through gamescope) and Desktop Mode (KDE Plasma), wlroots compositors and X11. Screens larger than 1280 pixels on their long side are scaled down before sending (set ELFDECKAI_SCREENSHOT_MAX to change the limit, or 0 to turn it off), and the result includes the image's size.

mouse_click_at and mouse_move_to take coordinates in the pixels of the most recent screenshot, so an agent can click exactly what it sees, whatever the screen's real resolution or the scaling applied. They assume a single display.

Running on the same machine as the daemons:

{"mcpServers": {"elfdeckai": {"command": "elfdeckai-mcp"}}}

Running on a Steam Deck from an AI client on your PC, over SSH (the setup-pc scripts in Start here write this for you):

{"mcpServers": {"elfdeckai": {
    "command": "ssh",
    "args": ["-i", "~/.ssh/elfdeckai_ed25519", "-o", "IdentitiesOnly=yes",
             "-o", "BatchMode=yes", "deck@DECK-IP-ADDRESS", ".local/bin/elfdeckai-mcp"]}}}

It logs in with an SSH key, because the MCP client cannot answer a password prompt. Without pip, use "command": "python3", "args": ["/path/to/elfdeckai_mcp.py"].

Every tool call is validated against the tool's schema and the daemon's own parser before anything is written to a FIFO. If a daemon is not running, the call returns an error immediately instead of blocking.

Command reference

ElfKey (/tmp/elfkey.fifo)

Command

Effect

key <name> [ms]

Tap a key (default 120 ms)

down <name> / up <name>

Hold / release a key

release

Release every held key

quit

Release everything and exit

Keys: a–z, 0–9, arrows, enter, esc, space, tab, backspace, shift, rshift, ctrl, rctrl, alt, ralt, capslock, f1–f10, home, end, pageup, pagedown, insert, delete and common punctuation. The full table is KEYS in elfkey.py.

ElfPad (/tmp/elfpad.fifo)

Command

Effect

press <btn> [ms]

Tap a button or D-pad direction (default 120 ms)

down <btn> / up <btn>

Hold / release

stick left|right <x> <y> [ms]

Set an analog stick, −100 to 100 per axis; with ms, recenter afterwards

trigger left|right <0..100> [ms]

Set an analog trigger; with ms, release afterwards

center

Return sticks, triggers and D-pad to neutral

reset

Release everything

quit

Release everything and exit

Buttons: a b x y lb rb l2 r2 select start guide l3 r3, plus D-pad directions up down left right.

ElfPad identifies as an Xbox 360 controller, so SDL, Steam Input and RetroArch map it without configuration. l2 and r2 drive the analog trigger axes, as on the real controller; there are intentionally no digital trigger buttons, because declaring them shifts Select, Start and Guide off the standard button indices.

ElfMouse (/tmp/elfmouse.fifo)

Command

Effect

move <dx> <dy> [glide_ms]

Move the pointer relatively; with glide_ms, spread the move over that time

moveto <x> <y> [<width> <height>]

Place the pointer at an absolute position: pixel (x, y) of a width×height frame, or device units 0–32767 across the screen when no frame is given

click <btn> [ms]

Click left, right or middle (default 100 ms)

down <btn> / up <btn>

Hold / release a button (for drags: down, move, up)

scroll <ticks> / hscroll <ticks>

Vertical / horizontal wheel (positive is up / right)

release

Release every held button

quit

Release everything and exit

move is relative: negative dy moves up. Its units are motion counts, not guaranteed screen pixels, because pointer acceleration applies. Each command is limited to ±10,000 counts and ±120 wheel ticks.

moveto uses a second virtual device, ElfMouse Absolute, which compositors treat like a virtual machine's tablet pointer, so the position is exact. Buttons are pressed on whichever device moved last.

Safety model

Because the caller may be a language model, the daemons are built so that bad input cannot leave the machine in a bad state:

  • Malformed input is refused, not fatal. Every line passes a strict parser; invalid commands are logged and ignored.

  • Taps are bounded. A timed hold lasts at most 5 seconds. Longer holds use down/up.

  • Nothing stays stuck. Timed actions restore in a finally block, held inputs are tracked, and on quit, SIGTERM, SIGINT or SIGHUP every input is released before the device is removed. If the MCP client disconnects mid-hold, the server releases every held input as it exits. key_release, mouse_release and pad_reset release everything on demand.

  • Owner-only access. Each FIFO is created with mode 0600.

  • Audit log. Every accepted command is written to stderr (the journal, when run as a service). That includes every key of anything typed — passwords included, one line per keystroke. Do not have an agent type secrets you are not willing to see in the journal.

  • Robust protocol handling. The MCP server returns proper JSON-RPC errors for malformed or oversized requests and never blocks on a stopped daemon.

Scope: these are real OS-level input devices. They act on whichever window has focus, including terminals and browsers. ElfDeckAI keeps input bounded, validated and releasable; deciding where an agent is allowed to act is up to the operator. Only run agents you trust, in sessions you can supervise.

Manipulated callers: the hardening above covers a caller that hallucinates or sends malformed input. It does not cover a caller that is manipulated — and the agent's eyes are a screenshot, so anything on screen (a web page, a game dialog, a chat message) is untrusted input that could try to talk it into typing into a terminal. There is no key or window allowlist and no physical kill switch; the mitigations are the operator's: supervise the session, keep terminals and browsers you care about out of focus, and stop the daemons (systemctl --user stop elfkey elfpad elfmouse) when the agent is not driving. If the MCP client disconnects, the server now releases every held input on its way out — but a live, manipulated agent is an operator problem, not one this validation layer can solve.

Comparison with other approaches

Approach

Limitation

Emulator APIs (PyBoy, mGBA scripting)

Only work inside a specific emulator.

Sending keys through Moonlight or other streaming clients

Mouse clicks are forwarded, but synthetic keyboard events are not.

xdotool

Cannot reach native Wayland clients.

ydotool, dotool

Same uinput approach for keyboard and mouse, but no gamepad, raw keycodes, and no agent-facing interface (MCP, input bounds, release commands).

Hardware input emulators (KVM gadgets, controller adapters)

Work, but require extra hardware.

Installation

Requirements: 64-bit Linux with /dev/uinput and Python 3.9 or newer.

Where /dev/uinput is not writable by your user, install the udev rule once:

sudo cp 70-elfdeckai-uinput.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules && sudo udevadm trigger
getfacl /dev/uinput   # your user should appear in the ACL

To run the daemons as systemd user services after pip install --user elfdeckai (the unit files expect the commands in ~/.local/bin):

mkdir -p ~/.config/systemd/user
cp systemd/*.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now elfkey.service elfpad.service elfmouse.service

Development

pip install -e ".[test]"
python3 -m pytest

The test suite covers each daemon's input parser, device behaviour against a fake uinput device, and the MCP server over real JSON-RPC. No /dev/uinput is required to run it.

Roadmap

See ROADMAP.md.

Changelog

See CHANGELOG.md.

License

MIT. See LICENSE.

Author

Created and maintained by Garth Elfstrom (@Garthnesss). Issues and pull requests are welcome.

Available Tools

22 tools
key_downA

Hold a key down (release with key_up, or key_release for all).

Inputs: key (required).

key_releaseA

Release every held key at once — the unstick button, for when a caller died mid-hold or state is unknown.

Inputs: no named parameters.

key_tapC

Tap a key on the virtual keyboard.

Inputs: key (required), hold_ms.

key_upC

Release a held key.

Inputs: key (required).

mouse_clickC

Click a mouse button at the pointer's current position.

Inputs: button (required), hold_ms.

mouse_click_atA

Move the pointer to (x, y) in the pixel coordinates of the most recent screenshot, then click (left by default).

Inputs: x (required), y (required), button, hold_ms.

mouse_downB

Hold a mouse button down (start a drag).

Inputs: button (required).

mouse_hscrollB

Scroll horizontally: positive clicks right.

Inputs: clicks (required).

mouse_moveA

Move the pointer relatively by (dx, dy). Pointer acceleration applies, so prefer mouse_move_to for targeting. Optional glide_ms spreads the move over time.

Inputs: dx (required), dy (required), glide_ms.

mouse_move_toA

Move the pointer to (x, y), in the pixel coordinates of the most recent screenshot (0,0 is its top-left corner). Takes a screenshot first if none has been taken yet.

Inputs: x (required), y (required).

mouse_releaseA

Release every held mouse button at once — the unstick button, for when a caller died mid-drag.

Inputs: no named parameters.

mouse_scrollC

Scroll the wheel: positive clicks up, negative down.

Inputs: clicks (required).

mouse_upB

Release a mouse button (drop the drag).

Inputs: button (required).

pad_centerB

Return sticks, triggers and D-pad to neutral.

Inputs: no named parameters.

pad_downB

Hold a gamepad button or D-pad direction down. In Steam Deck Gaming Mode, if the game ignores the pad, ElfPad is not Player 1: open Steam > Controller Settings > Reorder Controllers while the game runs and move the non-Deck controller to the top.

Inputs: button (required).

pad_pressC

Tap a gamepad button or D-pad direction. In Steam Deck Gaming Mode, if the game ignores the pad, ElfPad is not Player 1: open Steam > Controller Settings > Reorder Controllers while the game runs and move the non-Deck controller to the top.

Inputs: button (required), hold_ms.

pad_resetB

Release every button and recenter everything.

Inputs: no named parameters.

pad_stickA

Move an analog stick (x/y in percent, -100..100). Without hold_ms the stick stays; with it, it recenters after.

Inputs: x (required), y (required), side (required), hold_ms.

pad_triggerB

Set an analog trigger (percent 0..100). Without hold_ms it stays; with it, it releases after.

Inputs: side (required), percent (required), hold_ms.

pad_upB

Release a held gamepad button or D-pad direction.

Inputs: button (required).

screenshotA

Capture the screen of the machine this server runs on (e.g. a Steam Deck in Desktop Mode) and return it as a PNG image, scaled down if the screen is large. mouse_move_to and mouse_click_at take coordinates in this image's pixels. Requires a graphical session on the server's machine.

Inputs: no named parameters.

waitA

Wait before the next tool call, in milliseconds. Use it for timing between inputs: queued taps land back-to-back with essentially no gap, and a frame-polled game can merge them into one hold. Bounded like every tap.

Inputs: ms (required).

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    B3.3/5.0

    Scored across 22 tools

    Disambiguation4/5

    Tools are mostly distinct by device and action, with clear separation between tap, hold, release, and safety-release operations. A few pairs (pad_center vs pad_reset, mouse_up vs mouse_release) overlap conceptually, but the descriptions help disambiguate them.

    Naming Consistency4/5

    Names predominantly follow a snake_case device_action pattern (key_tap, pad_press, mouse_click), which is predictable. Minor deviations like mouse_move_to/mouse_click_at suffix forms and single-word screenshot/wait, plus the abbreviation hscroll, keep it from perfect consistency.

    Tool Count3/5

    At 22 tools, the surface is on the heavy side for a general MCP server, and several granular hold/release/reset variants could feel excessive. Each tool does correspond to a real input primitive, but the count is borderline per typical scoping guidance.

    Completeness4/5

    The core input surface is comprehensive: keyboard, mouse, and gamepad tap/hold/release, analog sticks and triggers, recentering, screenshots, and timing. Minor gaps like text-entry (type_string) and multi-touch gestures exist, but there are no major workflow dead ends.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Linux desktop HAL for AI agents. One daemon, one socket, every desktop action. GNOME, KDE, Hyprland, X11. Ships in one command.
      31
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server for controlling Linux desktops over Wayland, enabling AI agents to perform mouse, keyboard, window, and screenshot operations on Fedora KDE Plasma.
      AGPL 3.0
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables controlling your own Linux desktop machines from any MCP-capable LLM client, with opt-in agent capabilities for shell, filesystem, processes, screenshots, system info, and input injection.
      1
      MIT