luda
This server provides tools for an AI agent to observe and control a Linux desktop via accessibility and screenshots.
Check desktop/display health with
desktop_doctor.List, activate, and manage windows with
desktop_windowsanddesktop_activate.Capture screenshots with
desktop_observeand inspect accessibility trees withdesktop_inspect.Read text from controls with
desktop_read_text.Enter or replace text using
desktop_set_textanddesktop_enter_text.Send keyboard shortcuts with
desktop_press_keys.Click, scroll, and drag using screenshot coordinates via
desktop_click,desktop_scroll, anddesktop_drag.Focus and invoke actions on accessible elements with
desktop_focus_elementanddesktop_invoke.
Allows agents to control a Linux desktop environment, including window management, accessibility-tree inspection, clicking, typing, scrolling, and reading text.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@luda take a screenshot of the current desktop and describe what you see"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Luda is deprecated. Use LCU (Linux Computer Use) instead. Install LCU.
Luda
Let your coding agent use a Linux desktop.
Luda gives agents tools to see applications, read their controls, click, type, and check what changed. Use it to fill forms, edit documents, work with files, or test a graphical application.
It runs on the Linux machine that owns the desktop. No hosted service or model API is required by Luda.
How it works
Luda has two parts:
Tools connect your agent to the desktop through MCP, a protocol supported by many coding agents.
A skill teaches the agent how to choose controls, enter text, verify results, and recover when something changes.
The agent can read accessible controls directly or work from screenshots. Actions report whether their result was verified, merely sent, or uncertain. Clicking “Save,” for example, is not itself proof that a file was saved.
Related MCP server: MacWright
Install: tools and skill together
Run these commands on the Linux machine whose desktop the agent will control. You need an existing Linux X11 desktop, Python 3.12+, and Git. Ubuntu 24.04 with XFCE is the tested starting point. Wayland and Xwayland are not supported.
git clone --branch v0.3.4 --depth 1 https://github.com/0xpolarzero/luda.git
cd luda
sudo bash scripts/install.sh --user "$(id -un)"The installer installs the runtime and Linux dependencies, shows detected and supported agents, and lets you select one or several. It then registers the computer-use tools and installs their skill for your account. Restart/reconnect your agent and ask: “Use Luda to inspect my desktop.” The skill is discoverable by the agent; the client decides when to load it.
Already know your agent? Use --agent codex --yes, for example:
sudo bash scripts/install.sh --user "$(id -un)" --agent codex --yesUse bash scripts/install.sh --list-agents for all supported identifiers. Other agents can use a portable tools-and-skill export. Installation delegates to pinned versions of Vercel skills and add-mcp. It preserves unrelated settings, updates Luda’s own skill and MCP entry, and does not install or authenticate the agent itself. Required installer tooling is supplied automatically; no separate Node installation is needed.
Codex connected to a VM through its built-in SSH connection? Run installation inside the VM, selecting the Linux account used by that connection. Codex's backend there loads the skill and tool registration. Running an ordinary ssh command from a local agent does not automatically load the VM's configuration.
Building a VM or machine image? Run the same installer as root with an explicit account and all supported agents:
bash scripts/install.sh --user YOUR_ACCOUNT --agent all --yesThe account must already exist; use --user root explicitly when root is the intended agent account. Supported agents do not need to be installed yet, and existing agent settings can already be present. Use repeated --agent NAME options to select a subset instead. No running desktop or agent credentials are needed during the build. Start the desktop before using the tools. See the image recipe and first-boot checks.
You can also ask your agent:
Read Luda's README and installation guide. Install the released version on the Linux desktop machine, register its MCP tools and complete skill for my agent account, preserve unrelated configuration, and verify readiness. If this is an image build, configure everything without requiring a running desktop.
Installation, upgrades and custom agents · Agent connection details · Release downloads
What can it do?
Work with applications: find and activate windows, inspect controls, operate menus, select items, and manage windows or workspaces.
Enter and check text: Unicode, multiple lines, selections, clipboard paste, and readback where the application supports it.
Use the screen: screenshots, clicks, drags, scrolling, and keyboard shortcuts. Optional OCR, image matching, and recording provide additional ways to observe.
Handle interruptions: wait for changes, cancel work, pause agent input, and recover owned input after a disconnect.
Watch the agent work with as little disruption as possible. Luda shows a distinct agent cursor where supported and tries to leave your mouse, keyboard, and foreground windows undisturbed. The cursor is drawn into the desktop image, so ordinary desktop viewers can display it.
Luda automatically chooses background actions or independent input where supported. When compatibility requires it, Luda uses ordinary foreground mouse/keyboard control, which can move your pointer, redirect keyboard input, and bring windows forward. Application behavior can also change focus. The agent uses the same tools with no mode selection. See input routing and evidence.
The optional browser provider opens a temporary Chromium session for ordinary web fields. Existing browser profiles are not attached automatically.
Optional editor add-on
Editor Bridge adds exact rich-text verification for applications built with ProseMirror. It is a separate installation with its own tools and skill, and the application's developer must also register the adapter. It is not included or enabled by installing core Luda. Most desktop tasks do not need it.
Package it into an environment
You can preinstall Luda in a workstation, container with a graphical session, or VM image. Your integration provisions the desktop and accounts, then runs Luda's installer to configure the chosen agents.
Use the environment packaging guide to make tools and skills available across folders for each agent account. There is no universal installation directory that every agent automatically discovers.
Learn more
I want to… | Read |
Install or attach to a graphical session | |
Connect an agent and install its skill | |
Understand the tools and their parameters | |
Read the agent's operating instructions | |
Check supported backends and known limits | |
Build packages or contribute |
Application accessibility varies, and human input can race with an agent. Luda reports these limits rather than treating every dispatched action as success.
Available Tools
36 toolsdesktop_activateB
Reveal a window from desktop_windows and establish input focus using the automatically selected route. Observe again afterward.
Inputs: window_id (required).
desktop_applicationsARead-only
Find installed desktop applications by name, description or ID. Returns application_id and file/URI support; works while input is paused. Use an exact returned ID with desktop_launch.
Inputs: limit, query.
desktop_chooseA
Choose an observed list/radio/combo option or a visible table cell and verify selection. A table cell selects its whole row. Successful results verify selection only, including already-selected results. If selection alone was requested, stop. Otherwise verify the requested application effect; some controls apply on selection. If the effect did not occur, inspect for an advertised activation action or Apply/Open control, use it, then verify the outcome. Default makes the choice exclusive; extend preserves other list or table-row selections. Scroll offscreen rows into view and inspect again; reacquire after sorting/filtering. Open collapsed options and inspect first. range_end_id selects an inclusive range of at most 50 visible list items or table rows, using two endpoints from the same inspection; reversed endpoints are allowed. Every intermediate item must be inspected in unchanged order, and table endpoints use the same column. extend adds the range; otherwise it replaces the selection. Duplicate range labels, unsupported providers and unloaded gaps are refused. List replacement verifies one clear then each addition; an already exact set is unchanged. Limits: 50 range items and 500 selected items. Selection step receipts are historical verification, never instructions to retry a remainder.
Inputs: element_id (required), extend, range_end_id.
desktop_clickA
Click screenshot-image coordinates in the target window or its observed menus, automatically choosing independent or foreground pointer input after target validation. Rejects expired snapshots, changed window layout/identity and covered targets. Snapshot validity does not prove unchanged application content; observe again after content transitions before selecting a control.
Inputs: x (required), y (required), window_id (required), snapshot_id (required), +2 more.
desktop_controlA
Pause/resume cooperating agent input across servers on this display. Pause interrupts at the next checkpoint; already-delivered input is not undone. Observation remains available. This does not stop arbitrary external input programs.
Inputs: action.
desktop_doctorARead-only
Check actual display access, desktop session, dependencies and accessibility availability. Reports driver version and content identities for tool declarations and the server-bundled skill; the latter does not identify the skill loaded by your agent.
Inputs: no named parameters.
desktop_dragA
Drag between two observed points inside the same window, automatically choosing independent or foreground pointer input after validation; always attempts release of its own button. Use desktop_drag_to for another destination window.
Inputs: x (required), y (required), end_x (required), end_y (required), +3 more.
desktop_drag_toA
Drag from a source into a second observed window, automatically activating the source after validating both endpoints. Coordinates refer to one screenshot. Verify transfer in the applications; dispatch does not prove a drop was accepted.
Inputs: x (required), y (required), end_x (required), end_y (required), +4 more.
desktop_focus_elementA
Request element focus for agent input; owned browser fields refuse active/unknown composition before focus. Native application callbacks can still change human focus. Inspect to confirm focused state.
Inputs: element_id (required).
desktop_hoverA
Move the pointer to a recent observed point without clicking, automatically activating the target after validation; observe tooltips/submenus afterward.
Inputs: x (required), y (required), window_id (required), snapshot_id (required).
desktop_inspectARead-only
Inspect a window or find controls by name/role substring and required states. Returns bounded tree, parent IDs, supported actions and 60-second element IDs. Owned browser text_fields have distinct provider-bound IDs for ordinary HTML fields; role="entry" filters for fields. Field metadata describes line breaks and write scope. When extra owned pages/windows or frames make the owned provider unavailable, native nodes remain independently inspected; owned_browser reports unavailable/code and text_fields is empty. Cached owned fields still refuse unsupported scope; no mutation fallback. Empty matches and unavailable accessibility are distinct.
Inputs: window_id (required), name, role, limit, +2 more.
desktop_invokeA
Invoke the sole action returned by inspect, or supply its exact action name. Native actions address the control without activating its window; application callbacks may bring windows forward. Multiple actions require an explicit choice; no click/press naming guess is needed for a single-action button. Completion means dispatch, not verified application outcome.
Inputs: element_id (required), action.
desktop_launchA
Launch an installed application by its desktop_applications ID, optionally opening absolute existing paths or URIs. Returns dispatched with process-bound window candidates observed for wait_timeout seconds (default 1, range 0–3; 0 skips observation). Candidates do not prove document readiness; singleton association is never guessed. Inspect candidates or desktop_windows before acting; never blindly retry an uncertain launch.
Inputs: application_id (required), wait_timeout, files_or_uris.
desktop_match_imageARead-only
Find historical visual candidates from a selected screenshot crop in another retained screenshot. template_bounds requires x, y, width and height in source returned-image pixels; x/y are nonnegative and width/height are 8–512. Both IDs must be retained and fresh, from the same server and image scale; only the target must still have its captured layout. Historical source crops may come from a window that moved. Results use target returned-image pixels. Threshold is finite 0–1, limit 1–100; scores are uncalibrated correlation, never semantic identity or click permission. Returns non-overlapping candidates, preserving distinct duplicates; flat templates are refused. No new capture or input. Optional system OpenCV required.
Inputs: snapshot_id (required), template_bounds (required), template_snapshot_id (required), limit, +1 more.
desktop_observeBRead-only
Return screenshot plus window layout and a 15-second snapshot ID. Pointer coordinates and window/popup image_bounds use returned-image pixels; bounds/frame_bounds remain native X11 root pixels. image_bounds is null when no integer screenshot pixel maps into the client. Use integer image points.
Inputs: max_width.
desktop_ocrARead-only
Read uncertain local OCR word candidates from this exact retained screenshot, never a new capture. Returns image-pixel boxes and uncalibrated engine scores, not exact text or action permission. Snapshot expires after 15 seconds or cache eviction; changed layout is refused. Optional Tesseract and the selected language must be installed. No input is sent.
Inputs: snapshot_id (required), limit, language.
desktop_open_browserA
Open a fresh owned Chromium with explicit temporary_session lifetime. Browser/profile and unsaved content are deleted on server disconnect, backend close or reconnect. Requires optional browser dependencies and configured executable; never downloads automatically or attaches existing profiles. Inspect text_fields for ordinary HTML fields and explicit password-only secret entry. Ordinary protected-field operations, contenteditable editors and frames are unsupported.
Inputs: url (required), lifetime (required).
desktop_pasteA
Paste through the shared CLIPBOARD when semantic typing is unavailable, automatically choosing independent or foreground input for the target. Chooses common app shortcut from window class, with optional override. Destination is unverified; inspect dialogs/read back. Terminals can execute pasted newlines.
Inputs: text (required), window_id (required), shortcut.
desktop_press_keysA
Send a deliberate chord, e.g. ctrl+s, ctrl+plus, ctrl+minus, Return, Tab, Escape or Down. Punctuation uses X11 names (plus, equal, bracketleft, slash); implicit Shift follows the current layout. count is 1–20 complete press/release repetitions, default 1. Revalidates target identity, focus and input state between repetitions; stops on the first failure and never retries. Automatically chooses independent input where supported or foreground control otherwise. Refuses held input on the selected devices and preserves their keyboard mapping. Foreground fallback can change human focus. Unavailable symbols return UNSUPPORTED_KEYMAP; text belongs in desktop_type. When a final companion receipt is available, progress reports fully dispatched, possibly partial and not-started repetitions. Dispatched count is not application completion.
Inputs: chord (required), window_id (required), count.
desktop_read_textARead-only
Read accessible text and representation metadata, preserving whitespace. limit counts Unicode code points (default 16000, maximum 1000000), not bytes. Opaque embedded objects are not exact logical plain text: check plain_text_verification_supported. Normalization reads the bounded full field; a smaller limit does not enable streaming. Protected fields are refused. Generic native readback reports composition known=false, active=null; pending preedit is not checked.
Inputs: element_id (required), limit.
desktop_reconnectA
Reconnect this MCP connection to a running XFCE session owned by this account after a desktop restart. Omit PID only when exactly one session exists. Validates display and bus before replacing the backend; failed validation preserves it. Returns BUSY during other operations, never restarts apps or replays input. All prior window, element and screenshot IDs expire; observe again. The selected display's pause state remains in force. Temporary owned browsers/profiles are closed and unsaved content is lost; browser_cleanup=unconfirmed reports cleanup that could not be proved.
Inputs: session_pid.
desktop_recordingA
Explicit temporary screen recording: start, status, stop or delete by ticket. Start records only this X11 display, no audio, at 10fps and at most 1280×720 for 1–60 seconds. Start is not completed-file verification; stop/status return a path only after decoding verifies completion. Files are private, bounded and deleted on backend close/reconnect/server death; explicitly copy elsewhere before closure to save durably. Stop/delete remain usable while paused. Optional local ffmpeg/ffprobe required.
Inputs: action (required), max_seconds, recording_id.
desktop_recover_inputA
Retry cleanup of this server's interrupted supervised input, without replaying keys/clicks or resuming a paused desktop. Uses each operation's original session; a replaced X server is left untouched. Unproven cleanup stays blocked. Returns pending_count and recovery proofs; observe again before acting. Returns BUSY if another operation is still running.
Inputs: no named parameters.
desktop_reportARead-only
Return a sanitized bug-report JSON: fixed environment/dependency versions, projected health and up to 32 recent operation IDs/methods/effects/timings, fixed error codes and semantic verbs from this MCP process. Excludes desktop content, paths, exceptions and action arguments. No files, uploads or replay. Supply synthetic repro steps separately; explicitly save/delete the returned report if needed.
Inputs: no named parameters.
desktop_scrollA
Scroll 1–20 wheel ticks at a point in the observed target, automatically choosing independent or foreground pointer input after validation. Read resulting state to confirm.
Inputs: x (required), y (required), direction (required), window_id (required), +2 more.
desktop_selectA
Select a text range using Unicode code-point offsets, or place the caret when equal; verify the result. Owned-browser HTML fields focus automatically after range validation. Native edits inside graphemes can be refused.
Inputs: element_id (required), end_offset (required), start_offset (required).
desktop_set_checkedA
Set a checkable control to the requested state; avoid a blind toggle when it already matches.
Inputs: checked (required), element_id (required).
desktop_set_expandedB
Expand or collapse a supported control and verify state. Reinspect newly exposed children.
Inputs: expanded (required), element_id (required).
desktop_set_valueA
Set a numeric control within its inspected range and verify its accessibility numeric value. Displayed formatting and application commit may differ; inspect/read both, then explicitly commit only when intended.
Inputs: value (required), element_id (required).
desktop_statusARead-only
Return recent operation outcomes after timeout/cancellation. This operation history excludes input text and screenshots.
Inputs: no named parameters.
desktop_typeA
Type into an editable element and verify exact readback. Native supported edits address the control in the background; native input chooses independent or foreground control automatically. Insert replaces the selection; replace changes the entire field. Preserves Unicode, LF and tabs without submitting. Owned HTML fields require complete grapheme boundaries. Exact readback does not confirm saving or application commit; inspect before an explicit commit or suggestion selection.
Inputs: text (required), element_id (required), mode.
desktop_type_secretA
Replace an observed protected field. Never reads back or echoes the value, uses no clipboard, and reports dispatched only. Requires native protected EditableText or an owned password input with known inactive composition. Owned input rejects LF/CR and declared maxlength overflow; submission is separate. Applications control their own masking, which can change during input.
Inputs: text (required), element_id (required).
desktop_waitARead-only
Wait up to timeout (0–10 seconds) for an observed condition. window_present/window_absent/window_active require only window_id, never a title in text; discover title matches with desktop_windows(query=...). text_equals/text_contains require only element_id and text. element_present/element_absent require window_id plus at least one name/role substring or states filter; they search native accessibility nodes, and absence requires complete coverage. pixels_stable requires window_id and optionally stable_for (0.1–10 seconds, at most timeout); it samples the client rectangle, not general application idleness. Omit parameters belonging to other condition types.
Inputs: condition (required), name, role, text, +5 more.
desktop_windowA
Manage one window. move uses frame x/y; resize uses client width/height; workspace requires its index. fullscreen requests WM fullscreen; restore exits fullscreen/maximization/minimization; raise changes stacking without activation. Other actions take no extra parameters. Close reports an owned blocking dialog without confirming it. Geometry actions return fresh same-generation client/frame bounds and WM state; move/resize distinguish matched from nonmatching requests without upgrading dispatched effects. Restore compares captured pre-maximize bounds when observed geometry, state and size hints remain consistent; otherwise comparison is unknown. No saved geometry is forced, and unobserved external changes cannot be ruled out.
Inputs: action (required), window_id (required), x, y, +3 more.
desktop_windowsARead-only
List window identities, titles, focus and client bounds, optionally filtering title/class. Returns counts, unavailable rows and next_offset when paginated. Each call is a fresh enumeration.
Inputs: limit, query, offset.
desktop_workspacesA
List workspaces, or switch to an existing index and verify the active workspace. Every dispatched switch expires this server’s screenshots, even when requesting the current workspace; observe again before using coordinates. External workspace changes between observations are not continuously tracked.
Inputs: workspace.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 36 tools
Most tools target clearly distinct operations, and the detailed descriptions resolve boundary cases such as desktop_select vs desktop_choose or desktop_inspect vs desktop_read_text. A few diagnostic/status tools (desktop_status, desktop_report, desktop_doctor) and singular/plural window tools could still be confused at first glance, but not to the point of causing frequent misselection.
All tool names share the desktop_ prefix and use lowercase snake_case, which provides a strong overall pattern. Most names are verb-led (desktop_click, desktop_set_value, desktop_open_browser), though a few noun-only names like desktop_windows, desktop_applications, and desktop_status are minor deviations from a strict verb_noun convention.
36 tools is well above the 25-tool threshold and feels heavy even for a desktop automation server. Each tool is individually meaningful, but the set could be consolidated or grouped more tightly without losing essential functionality.
The tool surface covers the full desktop automation lifecycle: discovery, observation, inspection, input, window management, verification, and diagnostics/recovery. There are no obvious dead ends; every interaction has a corresponding observation or inspection tool, and failure/health handling is well supported.
Maintenance
Related MCP Connectors
Make games with AI: one MCP server that connects your agent to Unity, Unreal and Godot editors.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Official MCP server for Agentwork — delegate tasks to AI agents with human-in-the-loop
Cloud desktops for AI agents: create a Linux computer, see its screen, click, type and run tasks.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA desktop automation MCP server that enables AI agents to interact with Linux environments through screenshots, window inspection, and input simulation. It provides tools for mouse control, keyboard input, and screen capture using xdotool and XDG Desktop Portals.MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server for reliable native macOS desktop control from AI agents, providing 72 tools for screenshots, mouse, keyboard, scroll, clipboard, window management, and more.MIT
- FlicenseNot gradedqualityCmaintenanceAn MCP server that gives a model eyes and hands on a Linux Wayland desktop, enabling screenshot capture, mouse/keyboard control, OCR, and icon detection via OmniParser.1-
- AlicenseNot gradedqualityCmaintenanceMCP server for controlling Linux desktops over Wayland, enabling AI agents to perform mouse, keyboard, window, and screenshot operations on Fedora KDE Plasma.AGPL 3.0