xRDP Agent Harness

Give your AI agent control of your Linux desktop.

Screenshots, mouse clicks, scrolling, typing and keyboard shortcuts—across browsers, terminals, file managers and other desktop apps.

Works with Codex and other compatible MCP clients, with a CLI included. Connects over SSH to your existing X11 desktop. No changes to xrdp required.

What can you use it for?

• Check a remote desktop without opening your RDP client. Ask your agent for a screenshot, inspect an application, or see whether a dialog is blocking work.

• Work in apps without an automation API. Navigate menus, change settings or enter text using the same controls a person would. The agent plans the steps; the harness provides screenshots and input.

• Continue in an existing browser session. Use the browser already open on your server, with its current tabs and login state, then switch to a terminal or file manager when the task calls for it.

• Give an agent a separate machine to operate. Point it at a dedicated Linux desktop on a VPS while keeping your everyday desktop out of its control.

For example: “Show me what's on my VPS,” “Open the file manager and find my download,” or “Check the app's settings and show me the current configuration.”

Related MCP server: VNC MCP Server

Install with your agent

Copy and paste this into your agent:

Install https://github.com/rotcev/xrdp-agent-harness and connect it to my desktop.

Ask me only how I normally connect over SSH: my SSH alias or connection command,
plus a local key-file path if needed. Never ask for passwords or private key contents.
Discover the OS, desktop session and other settings yourself. Ask a follow-up only
if something is missing or there is more than one desktop to choose from.

Read the repo's setup instructions. Install locally in ~/tools/xrdp-agent-harness
using a virtual environment, install any missing remote dependencies, and register
MCP in this agent's client if supported. Keep my connection config outside the repo
at ~/.config/xrdp-agent-harness/config.toml with private permissions. Preserve any
existing setup and use strict SSH host verification; ask me to verify an unknown host.

Enable mouse and keyboard control, then capture a screenshot to verify the connection.
Don't restart or log out my desktop. Finish with a short result and any reload step
needed. If this client doesn't support MCP, set up the CLI instead.

You need a macOS or Linux computer and an existing Linux X11 desktop with SSH access. Ubuntu with xrdp and XFCE is the tested setup. Python 3.11+, Git and SSH are required locally; the agent can check dependencies for you.

Use it

Open a new terminal on my desktop, type echo hello from Harness, press Enter, and show me the result.

Or:

Take a screenshot of my desktop without changing anything.

Install manually

git clone https://github.com/rotcev/xrdp-agent-harness.git
cd xrdp-agent-harness
python3 -m venv .venv
.venv/bin/python -m pip install .

Then follow the connection setup guide.

Good to know

• Uses your own SSH access. Keep keys and private configuration out of the repo.

• Controls apps as the logged-in desktop user. Use a trusted agent and one controller at a time.

• Experimental v0.1.0. Windows and Wayland desktops aren't supported. No drag-and-drop, clipboard transfer or automatic session recovery yet.

Documentation · Security · Contributing · MIT License