luda
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@ludatake a screenshot of the current desktop and describe what you see"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Luda
An experimental local Linux desktop MCP server and agent skill, with an explicit feature/acceptance catalog. Runs on the existing microsandbox XFCE/X11/KasmVNC desktop. No AIO/Cua worker, cloud service, model API or nested VM.
Status: working prototype, not production-qualified. A tool can return dispatched without the application doing what you intended. Exact text replacement and window activation have specific readback checks. Clipboard insertion does not claim destination verification.
Deliverables
345 acceptance cases in 34 feature areas, with machine-readable catalog.
src/luda: 14 typed MCP tools, session launcher, X11 geometry, isolated AT-SPI worker, clipboard input and screenshots.tests: native fixture, real application checks and actual stdio MCP integration.
Related MCP server: MacWright
Tools
Tool | Contract |
| Actual session/display/dependency/accessibility checks |
| Window identities, focus, client/frame geometry |
| Activate and verify exact window identity |
| Screenshot, image/native geometry and expiring snapshot ID |
| Bounded, window-scoped accessibility tree and expiring element IDs |
| Exact accessible text with explicit truncation |
| Full editable-text replacement with exact readback |
| Explicit clipboard shortcut; destination remains unverified |
| One intentional chord; not a text-typing substitute |
| Recent screenshot coordinates in active client bounds |
| Discrete wheel ticks at a validated point |
| Same-window drag, with button-release cleanup attempt |
| Request focus on an enabled/showing element |
| Advertised semantic action, not an assertion of app outcome |
Install in a guest
For Ubuntu 24.04 with an existing XFCE/X11 desktop:
sudo bash scripts/install.sh /opt/luda-0.1.0The script provisions explicit apt dependencies and installs hash-pinned Python runtime requirements. The Python project itself is built from this checkout. It does not replace the desktop, edit Codex configuration, or start a public network listener. Use a new versioned directory for upgrades; automated upgrade/rollback is not implemented.
For development with uv:
uv sync --frozen --extra test
.venv/bin/luda-session --user silo-desktop -- .venv/bin/ludaRun the launcher as root or the desktop account. It reads that account's XFCE environment and runs the server as that account. If multiple XFCE sessions exist, pass --session-pid. Paths supplied to the server should be absolute in an actual Codex configuration.
Codex remote-context configuration
The following is a configuration example for the guest execution context, using the install path above. This example has not been tested through the Mac app's fresh SSH onboarding flow. Verify which Codex configuration the selected remote context loads; configuring the Mac host to run a guest-only path is not sufficient.
[mcp_servers.luda]
command = "/opt/luda-0.1.0/.venv/bin/luda-session"
args = ["--user", "silo-desktop", "--", "/opt/luda-0.1.0/.venv/bin/luda"]
startup_timeout_sec = 20
tool_timeout_sec = 20Copy the included skills/luda folder into the skill directory of the Codex account executing in that guest, for example ~/.codex/skills/luda. Do not overwrite an existing skill of that name without inspecting it. A new task should then discover the skill and MCP tools. The current project deliberately does not modify an existing Codex account's global configuration.
Test
Tests create only owned fixtures/documents and terminate their own applications. Run them against a disposable agent desktop: they necessarily change focus and clipboard contents. Native tests require GTK3 Python typelibs and Mousepad; application tests also require XFCE Terminal and a Playwright-compatible Chromium.
python scripts/build_requirements.py
.venv/bin/python -m unittest discover -s tests -p 'test_*.py'
.venv/bin/luda-session -- .venv/bin/python tests/live_backend.py
.venv/bin/luda-session -- .venv/bin/python tests/live_mcp.py
.venv/bin/luda-session -- .venv/bin/python tests/live_apps.py --browser /absolute/path/to/chromiumUse absolute paths if launching from a different directory. artifacts/ is gitignored and stores local synthetic test evidence. The browser helper uses Playwright only for offline setup and independent readback; desktop input comes from this driver.
Known limits
The acceptance catalog is intentionally broader than this implementation. Missing capabilities include generalized state waits, window move/resize management, cross-window drag, verified universal insertion, human takeover, cancellation guarantees, automatic reconnection after X-server restart, richer clipboard types, Wayland and browser DOM control. Qt/Electron/Firefox and AMD64 are not qualified. X11 focus races and unmanaged overlay interception remain possible. The worker's accessibility mapping can refuse apps whose reported top-level bounds do not match X11 geometry.
Clipboard contents are overwritten, PRIMARY remains separate, and a terminal can execute pasted newlines. Protected fields are unsupported for semantic read/write. No private user documents are included in tests or logs. See the design and validation documents before treating this as a release component.
Available Tools
14 toolsdesktop_activateA
Activate a window from desktop_windows and verify focus. Observe again afterward.
| Name | Required | Description | Default |
|---|---|---|---|
| window_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the burden. It discloses that activation is followed by focus verification and advises re-observing afterward. It does not detail side effects like focus shifting from the previously active window or possible failure cases, but it does convey the core behavioral sequence.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: two short sentences that front-load the primary action and include the necessary follow-up guidance. Every word earns its place, with no filler or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter tool, the description is largely complete: it tells where to get the ID, what action to perform, and what to do after. The absence of an output schema means the return value is not described, but 'verify focus' hints at the expected outcome.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only provides a string window_id with no description. The tool description adds the key semantic meaning: the window_id comes from desktop_windows. For a single-parameter tool with 0% schema coverage, this is meaningful compensation even though it does not describe the exact format.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action: activate a window, and it specifies the source (desktop_windows) and the verification behavior (verify focus). This distinguishes it from related tools like desktop_windows (listing) and desktop_observe (observing).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies a clear workflow: get a window from desktop_windows, activate it, verify focus, then observe again afterward. However, it does not explicitly mention when not to use this tool or compare it to alternatives such as desktop_focus_element.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_clickA
Click screenshot-image coordinates inside the active target client. Rejects expired or changed layouts.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| count | No | ||
| button | No | left | |
| window_id | Yes | ||
| snapshot_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral burden. It usefully discloses the coordinate frame and the rejection of expired or changed layouts, but does not mention side effects, error/success behavior, or whether the target window must first be activated.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused sentence. The action and coordinate system are front-loaded, and the second clause adds an important operational detail without filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a coordinate-click tool, the description plus schema is minimally viable: it explains the key non-obvious facts of coordinate space and stale-layout rejection. However, without annotations or an output schema, it leaves gaps around how to obtain snapshot_id, what happens on success or failure, and whether the window must already be active.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It partially does: x/y are explained as screenshot-image coordinates, snapshot_id relates to layout freshness, and window_id maps to the active target client. Count and button semantics are left to the schema enums and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: clicking screenshot-image coordinates inside the active target client. This clearly distinguishes the tool from element-focused siblings like desktop_focus_element or desktop_invoke.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives useful context: use screenshot-derived coordinates and expect rejection when the layout snapshot is stale. It does not explicitly name alternative tools or when-not-to-use conditions, but the intended context is clear enough.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_doctorA
Check actual display access, desktop session, dependencies and accessibility availability.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden. It discloses that the tool checks multiple aspects (display access, session, dependencies, accessibility), which is useful, but it doesn't disclose what happens on failure, whether it returns a report, or whether it has side effects. The description is honest but thin on behavioral detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with the verb 'Check', and every word adds meaning. It lists the four check areas compactly without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 0-parameter diagnostic tool with no output schema, the description is reasonably complete: it tells the agent what the tool checks. However, it doesn't describe the return value or how to interpret results, which an agent would need to act on the diagnosis. Given the tool's diagnostic nature, this is a notable gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has 0 parameters, so the schema is trivially complete. The description adds meaning by explaining what the tool checks, which is the only semantic content an agent needs. Baseline 4 for 0 params is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('Check') and a clear resource ('display access, desktop session, dependencies and accessibility availability'). It distinguishes itself from sibling tools that perform actions (activate, click, drag) or observations (observe, inspect), though it doesn't explicitly name a sibling alternative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this is a diagnostic tool to run before other desktop operations, but it doesn't explicitly state when to use it vs alternatives. The context signals show 0 parameters and no annotations, so the description carries the burden; it gives a clear context but no exclusions or alternative routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_dragA
Drag between two observed points inside the same active window; always attempts button release. Cross-window drags are not supported yet.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| end_x | Yes | ||
| end_y | Yes | ||
| button | No | left | |
| window_id | Yes | ||
| snapshot_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full behavioral burden. It discloses that it 'always attempts button release' and that cross-window drags are not supported, which is useful. However, it does not mention what happens on failure, whether coordinates are relative to the window or screen, or if any prerequisites like a valid snapshot are required. These gaps are significant for a mutating tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, two sentences, with the core purpose stated first and the limitation following. No redundant phrasing or filler, every sentence adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 7 parameters, 0% schema coverage, and no output schema, the description is insufficient. It does not explain return values, error handling, coordinate origins, or how the snapshot and window IDs relate to the drag points. An agent would need additional context to invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for parameter meaning. The description implies that x,y and end_x,end_y are start and end points, but it does not explain their coordinate system (window-relative? screen?), the role of snapshot_id and window_id, or the optional button parameter. This leaves the agent to guess critical details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (drag) and the scope (between two observed points inside the same active window). It also distinguishes from sibling tools like desktop_click or desktop_scroll by specifying a drag operation, and it notes a limitation (cross-window drags not supported). This is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly restricts usage to same-window drags and notes that cross-window drags are unsupported, which guides an agent on when this tool is applicable. However, it does not explicitly name alternative tools or provide conditions for when to prefer this over desktop_click or desktop_scroll, leaving some inference required.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_enter_textA
Paste literal text at the current caret using the application's explicit shortcut. Replaces CLIPBOARD; target contents are NOT verified. Terminals may execute newlines. Observe dialogs/read back before proceeding.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| shortcut | Yes | ||
| window_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states that the clipboard is replaced, that target contents are NOT verified, that terminals may execute newlines, and recommends verifying via dialogs/read-back. This is unusually transparent for a tool description.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short, dense sentences cover the action, side effect, hazard, and recommended mitigation. There is no filler or repetition; the main action is front-loaded in the first sentence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 3-parameter tool with no annotations and no output schema, the description covers the core action, side effects, and key risks. However, window_id semantics are ambiguous, particularly whether prior activation via desktop_activate is required, and it does not describe return/error behavior. The 'current caret' phrasing leaves a gap between the specified window and the actual paste location.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description must clarify parameters. It explains 'text' as literal and 'shortcut' as the application's explicit paste shortcut, but window_id is only indirectly implied by 'current caret'. It does not state whether window_id must be activated first or how the caret is selected within that window.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Paste literal text at the current caret using the application's explicit shortcut', naming a specific verb, object, and mechanism. The 'literal text' and 'explicit shortcut' details distinguish it from sibling tools like desktop_set_text or desktop_press_keys, and the clipboard side effect is also mentioned.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear operational context: pasting at the caret, using the application's shortcut, and the warning that terminals may execute newlines. It also advises reading back before proceeding. However, it does not explicitly state when to prefer this tool over desktop_set_text or desktop_press_keys, nor does it list exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_focus_elementA
Request accessibility focus in the active window. Inspect to confirm focused state.
| Name | Required | Description | Default |
|---|---|---|---|
| element_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the behavioral burden. 'Request accessibility focus' implies a best-effort action rather than a guaranteed change, and 'Inspect to confirm focused state' explicitly warns that verification is needed. This is meaningful behavior disclosure for a focus tool, though it does not detail permissions or error conditions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences with no filler. The action is front-loaded, and the second sentence adds a practical verification step. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter tool with no output schema and no annotations, this description provides the active-window scope and a verification instruction. It does not explain where element_id comes from or possible failure modes, but sibling tool names and the concise workflow largely fill that gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides only the name and type of element_id, with 0% schema description coverage. The tool description does not explain what element_id refers to or how to obtain it, leaving the agent to infer from the tool name and sibling tools.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Request accessibility focus in the active window.' This clearly distinguishes the operation from sibling tools like desktop_click or desktop_invoke, and adding 'active window' defines the scope precisely.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives a clear follow-up workflow ('Inspect to confirm focused state'), which implies how to verify the result. However, it does not explicitly state when to use this tool versus alternatives like desktop_click, desktop_invoke, or desktop_activate, and no exclusions are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_inspectC
Return a bounded accessibility tree with opaque 60-second element IDs, roles, states and actions.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| window_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must carry the burden. It discloses key traits: the tree is bounded, element IDs are opaque and valid for 60 seconds. This is useful context. However, it does not mention that the operation is read-only, nor what happens on invalid window_id or when the tree exceeds the limit. It also doesn't state whether the returned IDs are intended for use with other desktop tools, which is a significant behavioral aspect.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that front-loads the core action and resource, followed by specific details on the returned data and ID characteristics. No filler or redundancy. It is concise while conveying multiple important points.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is part of a larger desktop automation suite, and the description mentions the expiring IDs but does not explicitly connect them to subsequent actions or tools. It also omits the effect of the limit parameter and how to obtain the window_id (e.g., via desktop_windows). While the core behavior is clear, these gaps make it incomplete for an agent to use effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description adds no meaning to the parameters. It doesn't explain that 'limit' controls the bound or how 'window_id' is obtained. The description's mention of 'bounded' is vague and does not connect to the limit parameter. The agent gets no help understanding the parameters beyond the schema's types and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool returns a bounded accessibility tree with specific content (roles, states, actions). The verb 'Return' and resource 'bounded accessibility tree' are specific and distinguish it from siblings like desktop_observe (likely for changes) or desktop_read_text. However, it does not explicitly name alternatives or contrast with them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is given on when to use this tool versus alternatives. It does not mention prerequisites (e.g., how to get window_id) or situations where another tool would be more appropriate. The description only states what it does, not the context of use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_invokeA
Invoke an exact action name returned by inspect. Completion means dispatch, not verified application outcome.
| Name | Required | Description | Default |
|---|---|---|---|
| action | Yes | ||
| element_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It transparently warns that completion means dispatch, not verified application outcome, which is a crucial non-obvious behavior. It does not discuss side effects or reversibility, but for an arbitrary action dispatcher, the dispatch caveat is the most important behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The entire description is one tightly worded sentence with no filler. It front-loads the core action and includes the most valuable caveat without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no output schema, the description covers the main runtime source for the action and the critical completion semantics. It could mention element_id provenance and return behavior, but the essential information for correct invocation is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema gives only bare parameter names with 0% description coverage. The description adds real meaning to 'action' by requiring an exact value returned by inspect, but it does not clarify how 'element_id' should be obtained or formatted, leaving half the parameter semantics implicit.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb-resource relationship: invoke an exact action name on an element identified by element_id. It distinguishes this generic dispatcher from concrete sibling tools like desktop_click or desktop_scroll, and ties the action to what inspect returns.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an important prerequisite: use an action name returned by inspect. However, it does not explicitly say when to use this tool instead of sibling actions like desktop_click or desktop_focus_element, nor does it describe when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_observeC
Return screenshot plus window layout and a 15-second snapshot ID. Coordinates are image pixels.
| Name | Required | Description | Default |
|---|---|---|---|
| max_width | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It does reveal key traits: it returns three artifacts and explains coordinate semantics. However, it leaves the meaning and lifetime of the '15-second snapshot ID' vague and says nothing about potential side effects or limitations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences with the core output list front-loaded and no filler. Every phrase earns its place, and the coordinate-system note is placed at the end as a useful clarification.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple observation tool with one optional parameter and no output schema, the description names the three returned artifacts and coordinate semantics, which is fairly complete. Gaps remain around max_width's effect and the purpose/lifetime of the snapshot ID, and no annotations help fill those gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The only parameter, max_width, is not explained anywhere; schema description coverage is 0%. The note about image-pixel coordinates does not clarify what max_width controls, so an agent must guess whether it limits screenshot width, layout dimensions, or something else.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action and resource: it returns a screenshot, window layout, and a 15-second snapshot ID. This makes its core purpose clear, though it does not explicitly contrast itself with siblings like desktop_inspect or desktop_windows.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is given about when to use desktop_observe versus alternatives such as desktop_inspect, desktop_windows, or desktop_read_text. The only contextual note, 'Coordinates are image pixels,' is about output interpretation, not usage conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_press_keysA
Send one deliberate chord, e.g. ctrl+s, ctrl+shift+v, Return, Tab, Escape. Requires target focus; never use this to type text.
| Name | Required | Description | Default |
|---|---|---|---|
| chord | Yes | ||
| window_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With zero annotations and no output schema, the description carries the full disclosure burden. It does add real behavioral constraints: exactly one chord at a time (no sequences/macros), focus is a prerequisite, and text entry is out of scope. But it stays silent on what happens when focus is absent (fail vs. auto-activate), error behavior, and any response/return signal, leaving meaningful operational gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences totaling roughly 17 words, with zero filler. Purpose is front-loaded, followed by format examples, a precondition, and a boundary clause — every piece earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no annotations and no output schema, the description covers purpose, chord value format, the focus requirement, and the text-entry exclusion. What's missing is operational chaining detail: how to obtain window_id and what to expect if the target window is not focused. These are minor gaps given the low complexity, but they would help an agent compose this with desktop_activate/desktop_windows.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for both parameters. It compensates strongly for chord by documenting the value format with modifier+key examples and named keys (Return, Tab, Escape). The window_id parameter is only implied as the 'target focus' and is not tied to how an agent should source it (e.g., from a desktop_windows call), which is the weaker half.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Send one deliberate chord') and gives concrete value examples (ctrl+s, ctrl+shift+v, Return, Tab, Escape) so an agent knows exactly what kind of input this tool accepts. The clause 'never use this to type text' distinguishes it from the text-entry siblings (desktop_enter_text, desktop_set_text) in the sibling list, making its purpose unmistakable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides an explicit precondition ('Requires target focus') and an explicit exclusion ('never use this to type text'), which tells the agent when not to reach for this tool. However, it never names the alternative tool for text entry (desktop_enter_text), so the positive when-to-use side is left mostly to inference from the siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_read_textB
Read exact accessible text, preserving whitespace. Protected fields are unsupported. Maximum 1 MB.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| element_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It does disclose useful behavioral constraints: exact whitespace preservation, protected fields being unsupported, and a 1 MB maximum. However, it does not explain what happens when a protected field is encountered, whether the call is purely read-only, or how failures surface.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short, information-dense sentences with no filler. The core behavior is front-loaded, and each sentence adds new, useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Reasonable for a simple read tool: it names the element identifier requirement and size limits. But it omits the return format, does not define 'protected fields', and relies on sibling tools to teach the agent how to discover element_id. Given the absence of a schema output model, these gaps keep it from being fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, and the description does not explain element_id or limit. 'Maximum 1 MB' hints at an overall size cap but does not clarify the units or role of limit. Most parameter meaning is left to the agent to infer from names alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific action ('Read') and resource ('exact accessible text'), and adds a useful qualifier about preserving whitespace. It is clear enough to distinguish it from write-oriented siblings like desktop_set_text, though it does not explicitly name alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is given about when to use this tool versus desktop_inspect, desktop_observe, or desktop_set_text. The agent is left to infer that this is for retrieving text, and must discover element IDs elsewhere without direction.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_scrollB
Scroll 1–20 wheel ticks at a point in the observed active target. Read resulting state to confirm.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| ticks | No | ||
| direction | Yes | ||
| window_id | Yes | ||
| snapshot_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full burden of behavioral disclosure. It reveals the action range and suggests verifying state afterward, but it does not disclose whether the operation is safe/reversible, what prerequisites exist, how errors manifest, or what the tool returns.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact, front-loaded with the action, and contains no filler. The second sentence adds a useful follow-up instruction without bloating the text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given six parameters, zero schema descriptions, no annotations, and no output schema, this description is under-specified. An agent would be left guessing about coordinate systems, scroll behavior limits, failure conditions, and whether it must call desktop_observe to see the result.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate for all six parameters. It maps roughly to 'ticks', 'point' (x/y), and 'observed active target' (window_id/snapshot_id), but it does not explain coordinate origin/units, the meaning of direction relative to screen/window, or how ticks interact with the direction parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Scroll 1–20 wheel ticks'), a target ('a point in the observed active target'), and clearly distinguishes itself from siblings like desktop_click, desktop_drag, and desktop_press_keys. This is immediately actionable and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool: on an observed active target that needs wheel scrolling, and it advises reading the resulting state afterward. However, it does not explicitly contrast with alternatives such as desktop_drag or desktop_invoke, nor does it state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_set_textA
Replace all text through accessibility and compare exact readback. Preserves LF, Tab and Unicode; rejects other control characters. Does not submit.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | ||
| element_id | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the behavioral disclosure burden and does so well: it notes the accessibility-based method, exact readback comparison, preservation of LF/Tab/Unicode, rejection of other control characters, and absence of form submission. This is substantially more transparent than a generic 'set text' statement, though it does not cover failure modes or prerequisites.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences with no filler. The main action is front-loaded, followed by value-adding behavioral constraints and a clear negative side-effect. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no output schema, the description covers the core behavior, character constraints, and what it does not do. It lacks explicit guidance on element_id provenance and error behavior, but the provided details are enough for an agent to invoke it correctly in most desktop automation flows.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate. It enriches the 'text' parameter by specifying allowed characters and what is rejected, but it leaves 'element_id' completely unaddressed beyond its schema title and name. Partial compensation only.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action—'Replace all text through accessibility'—and identifies the resource (text via accessibility). It clearly differentiates from siblings like desktop_enter_text and desktop_read_text by emphasizing replacement of the entire text value rather than insertion or reading.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'Replace all text' implies the use case of overwriting an element's entire text content, and 'Does not submit' provides a negative boundary. However, it does not explicitly say when to prefer this over desktop_enter_text or when it is not appropriate, leaving usage conditions largely implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
desktop_windowsA
List window identities, titles, process identity, focus and native client bounds.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Since there are no annotations, the description must carry the behavioral disclosure burden. 'List' strongly implies a read-only, non-mutating operation, and the returned data is stated. However, it does not disclose potential permission requirements, whether all or only visible windows are included, or any output formatting assumptions. There is no contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single tight sentence with no filler. The action and return content are front-loaded, and every word contributes useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a parameter-less read tool, the description is largely complete: it states the operation and the exact data returned. It lacks explicit scope and usage-boundary details, but the absence of parameters and output schema means an agent can still invoke it correctly based on this description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters and the schema is empty, so there are no parameter semantics to document. The baseline of 4 applies because nothing is lost by omitting parameter details.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Uses a specific verb ('List') and names the exact resource ('windows'), then enumerates the fields returned: identities, titles, process identity, focus, and native client bounds. It does not explicitly distinguish itself from observation siblings like desktop_observe or desktop_inspect, but the window-level scope is clear from the name and description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'List window...' implies use when the agent needs to enumerate open windows, but there is no explicit when-to-use or when-not-to-use guidance. It does not mention that desktop_observe or desktop_inspect should be used for UI element inspection, leaving usage context to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
14 tool updates
v0.1.0- First observed
desktop_activate - First observed
desktop_click - First observed
desktop_doctor - First observed
desktop_drag - First observed
desktop_enter_text - First observed
desktop_focus_element - First observed
desktop_inspect - First observed
desktop_invoke - First observed
desktop_observe - First observed
desktop_press_keys - First observed
desktop_read_text - First observed
desktop_scroll - First observed
desktop_set_text - First observed
desktop_windows
TDQS
Scored across 14 tools
Each tool targets a distinct desktop automation primitive: environment checks, window listing/activation, observation, accessibility inspection, text reading/writing/pasting, key chords, clicking, focusing, scrolling, dragging, and invoking actions. Even the input-related tools are clearly separated by mechanism and verification behavior.
All tools share the consistent desktop_ prefix and mostly use an imperative verb or verb_noun pattern (desktop_activate, desktop_read_text, desktop_press_keys). desktop_windows is the one noun-style outlier, and a few verbs like desktop_click and desktop_scroll lack an explicit object, but the overall convention is predictable.
Fourteen tools is well within the ideal range for a desktop automation server and each tool earns its place by covering a distinct interaction or observation need. The count feels appropriately scoped rather than bloated or thin.
The surface covers the core desktop workflow well: inspect, observe, activate, input, click, scroll, drag, and invoke. Minor gaps remain such as explicit window minimize/close/resize operations, mouse hover/right-click primitives, and direct clipboard access, but agents can typically work around these using the provided accessibility and keyboard tools.
Maintenance
Related MCP Connectors
MCP server for building and testing AI agents with multi-model experimentation and insights.
Official MCP server for Agentwork — delegate tasks to AI agents with human-in-the-loop
MCP server connecting AI agents to 100+ apps (Gmail, Slack, Notion, GitHub) via one-click OAuth.
Hosted MCP server connecting claude.ai, ChatGPT and other AI apps to your own computer
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceA desktop automation MCP server that enables AI agents to interact with Linux environments through screenshots, window inspection, and input simulation. It provides tools for mouse control, keyboard input, and screen capture using xdotool and XDG Desktop Portals.MIT
- AlicenseNot gradedqualityDmaintenanceAn MCP server for reliable native macOS desktop control from AI agents, providing 72 tools for screenshots, mouse, keyboard, scroll, clipboard, window management, and more.MIT
- FlicenseNot gradedqualityCmaintenanceAn MCP server that gives a model eyes and hands on a Linux Wayland desktop, enabling screenshot capture, mouse/keyboard control, OCR, and icon detection via OmniParser.1-
- AlicenseNot gradedqualityAmaintenanceMCP server for controlling Linux desktops over Wayland, enabling AI agents to perform mouse, keyboard, window, and screenshot operations on Fedora KDE Plasma.AGPL 3.0