claude-background-control
Supports interacting with Electron app content using real mouse clicks (CoreGraphics) when accessibility-based control is insufficient.
Provides control of the macOS GUI through accessibility APIs and fallback real input, enabling UI inspection, clicking, setting values, typing, keyboard shortcuts, menu navigation, scrolling, dragging, screenshots, and app activation.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@claude-background-controlfill the subject in Mail with 'Meeting notes'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
claude-background-control
An MCP server that lets Claude Code use your Mac's GUI: read what's on
screen, click things, and type. It uses only built-in macOS tools
(osascript, screencapture, sips), so there are no native npm
dependencies to break on install.
It goes through the accessibility tree first. That way it can press buttons and fill in text fields in background apps without moving your mouse or taking focus away from what you're doing. Real mouse and keyboard events are only the fallback.
Install
claude mcp add computer-control -- npx -y claude-background-controlRequires macOS and Node 20+ (development needs 22.18+ to run the TypeScript tests directly).
Related MCP server: Desktop Pilot MCP
Permissions (first run)
macOS has to allow the app that runs Claude Code (Terminal, iTerm, the Claude app, …):
System Settings → Privacy & Security → Accessibility: turn it on for that app. Every tool except
screenshotneeds this.System Settings → Privacy & Security → Screen & System Audio Recording: turn it on for that app. Needed for
screenshot.Restart that app.
If a permission is missing, the error names the exact app and path
macOS expects. That isn't always the app you see. Claude Code inside the
Claude desktop app, for example, needs the grant on
~/Library/Application Support/Claude/claude-code/<version>/claude.app,
which you add with + and Cmd+Shift+G. Because that path includes the
version number, a Claude Code update can require granting it again.
Tools
Tool | What it does |
| Lists UI elements of the app's windows with an id, role, label, value, and frame. Start here. |
| Presses a button or focuses a field via accessibility, with no mouse movement. Falls back to a real click. |
| Sets a text field's value directly. Works in background apps and with any language. |
| Types into the focused field by pasting, so Korean and other IME input comes through correctly. The whole clipboard (images, files, rich text) is restored afterwards. |
| Sends key codes like |
| Waits until a matching element appears, or disappears with |
| Real mouse click (CoreGraphics) for canvas or Electron content. |
| Walks the menu bar by item names, e.g. |
| Mouse-wheel scroll by lines, at a point or wherever the cursor is. |
| Left-button drag: select text, move things, resize. |
| PNG of the main screen, a region, or one app's front window, even if other windows cover it. Image pixels equal screen points. For a window, add the reported origin to get click coordinates. |
| Launches an app or brings it to the front. |
app is the process name (e.g. "TextEdit"). If you leave it out, the
frontmost app is used. When a label or role matches in several windows,
the frontmost window wins. If it's still ambiguous, you get a list of
ids to choose from.
Develop
npm install
npm test # pure-logic unit tests
npm run buildAfter changing src/mac.ts, run the manual GUI checklist in
scripts/smoke-test.md.
Limits
Main display only.
inspectdoesn't descend into leaf-like roles such as buttons and static text.With Stage Manager on, a background app's window is only a side-strip thumbnail, so
screenshot(app)warns and suggestsactivate_appfirst.
Design notes: docs/superpowers/specs/2026-09-30-computer-control-mcp-design.md
Available Tools
12 toolsactivate_appAIdempotent
Launch or bring an app to the front (needed before type/key, which go to the frontmost app).
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | App name, e.g. "TextEdit" |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare idempotent=true, destructive=false, readOnly=false and openWorld=false, so the safety profile is covered. The description adds real behavioral context beyond that: the tool mutates frontmost-app focus state, which downstream type/key calls depend on. It omits error behavior (e.g. app not found), which keeps it from a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence with the primary action front-loaded and the prerequisite rationale in parentheses. No filler, every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-parameter UI action with annotations covering safety and no output schema, the description supplies what an agent needs: what it does and why it must precede keyboard input. Minor gaps around failure modes and return behavior are acceptable given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Only one parameter exists and schema coverage is 100%, with the example 'TextEdit' documented in the schema itself. The description adds no further detail about the name argument, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Launch or bring an app to the front') with the name parameter implied. It also separates itself from the frontmost-app family by noting the ordering dependency on type/key, so an agent can distinguish it from sibling input tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states the condition under which this tool is needed ('needed before type/key'), which is actionable context for sequencing calls. It does not, however, describe alternatives for switching apps or when not to call it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
clickADestructive
Real mouse click at screen coordinates (points). Fallback for content without accessibility info.
| Name | Required | Description | Default |
|---|---|---|---|
| x | Yes | ||
| y | Yes | ||
| button | No | left | |
| clicks | No | 2 = double-click |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true, readOnlyHint=false and non-idempotency, so the safety profile is covered. The description adds that this is a real input injection at point coordinates, but says nothing about whether the target window must be focused, cursor side effects, or failure modes when coordinates land off-target.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, no filler, with the core action front-loaded and the fallback condition following immediately. Nothing could be cut without losing signal.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a four-parameter mutation tool with no output schema, the description covers the essentials (what, where, fallback role) but omits environmental prerequisites such as window focus or coordinate origin. Adequate, with clear room for more behavioral detail.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 25% (only 'clicks' is described), so x, y and button lean on the description for meaning. "Screen coordinates (points)" usefully pins down the unit for x/y, but button semantics and any coupling between clicks and button are left unstated.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and mechanism: a real (synthetic) mouse click at screen coordinates. It implicitly carves out a niche against the accessibility-based alternative, though it never names click_element directly. An agent can tell it apart from drag/scroll/type by the resource acted on.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Fallback for content without accessibility info" gives a clear condition for selecting this tool over the accessibility-driven click path. It does not name click_element explicitly or state exclusions (e.g., prefer click_element when it exists), so it falls short of a full when/when-not statement.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
click_elementADestructive
Press a button/link or focus a text field by id or label. Uses accessibility actions (no mouse movement) and falls back to a real click at the element center.
| Name | Required | Description | Default |
|---|---|---|---|
| id | No | Element id from inspect, e.g. "1.3.2" | |
| app | No | Process name, e.g. "TextEdit". Defaults to the frontmost app. Works on background apps. | |
| role | No | Role to match, e.g. "AXButton". Alone, it must match exactly one element. | |
| label | No | Element name/description/value to match (used when id is not given) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true and non-idempotent, and the description adds genuine behavioral context beyond them: accessibility actions without mouse movement with a fallback to a real click at the element center. It does not explain what makes the operation destructive or what state it changes, but the mechanism disclosure is substantive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the action and its target set, followed by the implementation/fallback note. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers mechanism and fallback for a 4-parameter tool with no output schema, and annotations carry the safety profile. However, it leaves the destructive flag unexplained and omits selection guidance against the many siblings in this UI-automation family, so it is adequate rather than complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents id, app, role, and label with examples. The description only restates the id-or-label addressing ('by id or label') and adds no semantics the schema lacks, which matches the baseline 3 when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (press/focus) and resource (button/link/text field) and the addressing mechanism (id or label). It is clear what the tool does, but it never distinguishes itself from the sibling 'click' or 'set_value', so an agent cannot tell from the text alone why it would pick this one.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use, when-not-to-use, or alternative is named. The presence of a sibling 'click' tool makes this omission material — the description gives no condition for choosing click_element over click, nor for using set_value instead of focusing a text field.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
dragBDestructive
Drag with the left mouse button from (x1, y1) to (x2, y2): select text, move items, resize.
| Name | Required | Description | Default |
|---|---|---|---|
| x1 | Yes | ||
| x2 | Yes | ||
| y1 | Yes | ||
| y2 | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true and non-idempotent/non-readOnly, so the safety profile is covered. The description adds genuinely new behavioral detail by specifying the left mouse button, which is not in the annotations. It does not disclose the coordinate frame (screen vs. window), whether intermediate move events are emitted, or any focus/activation requirement — significant gaps for a drag gesture.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One sentence, front-loaded with the action and gesture, followed by the parameter mapping and outcomes. No filler or repetition of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, 0% parameter coverage, and a destructive mutation, the description is missing the essentials an agent needs to call it reliably: coordinate origin/units, whether the target window must be activated first, and what the tool returns or how failure is signaled. The use-case list is helpful but not sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must carry parameter meaning, and it partially does: 'from (x1, y1) to (x2, y2)' establishes that the first pair is the start point and the second pair is the drop point. It still leaves the coordinate origin and units unspecified, so it does not fully compensate for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (drag) plus the resource and gesture (left mouse button from (x1,y1) to (x2,y2)), and enumerates intended outcomes (select text, move items, resize). It is clearly distinguishable from click/scroll/type siblings by naming the press-drag-release movement. It stops short of explicitly contrasting itself with click, but the gesture is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The three example outcomes (select text, move items, resize) imply when the tool applies, which is better than nothing. However, there is no explicit when-not guidance (e.g., use click for a single point, click_element for known targets) and no prerequisites such as needing the target window focused before dragging.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
inspectARead-onlyIdempotent
List UI elements (id, role, label, value, frame) of an app's windows via the accessibility tree. The preferred way to find what to click. Pass find to get only matching elements instead of the whole tree.
| Name | Required | Description | Default |
|---|---|---|---|
| app | No | Process name, e.g. "TextEdit". Defaults to the frontmost app. Works on background apps. | |
| find | No | Only elements whose role equals this or whose label/value contains it, e.g. "Save" or "AXTextField" | |
| maxDepth | No | ||
| maxNodes | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive/openWorld=false, so the safety profile is covered. The description adds real behavioral value on top: it discloses the data source (accessibility tree), the returned field set, and that an unfiltered call returns the 'whole tree' while find narrows it. It does not mention truncation or the maxNodes/maxDepth caps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, front-loaded with what it lists and where the data comes from, then the usage cue, then the parameter tip. Every sentence earns its place with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter, annotation-covered reader with no output schema, the description adequately covers purpose, source, and filtering. It omits the practical hazard that results are depth/node-capped, which an agent needs to know before trusting completeness of the listing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 50%: 'app' and 'find' are documented in the schema and the description reinforces find's filtering behavior. The two undocumented parameters (maxDepth, maxNodes) are never explained in either place, so the description only partially compensates for the coverage gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource: 'List UI elements ... of an app's windows via the accessibility tree', and it names the returned fields (id, role, label, value, frame). It also differentiates itself from visual siblings by calling itself 'the preferred way to find what to click'. Clear, though it never names an alternative tool explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'The preferred way to find what to click' gives strong, actionable context for when to reach for this tool (element discovery before click_element/drag). No explicit when-not-to-use or named alternative, which keeps it short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
keyADestructive
Press a key or combo in the frontmost app, e.g. "return", "cmd+s", "cmd+shift+t", "esc", "down". repeat presses it several times.
| Name | Required | Description | Default |
|---|---|---|---|
| combo | Yes | ||
| repeat | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true, readOnlyHint=false and idempotentHint=false, so the safety profile is covered structurally. The description adds the useful fact that input goes to the frontmost app, but never warns that arbitrary combos can destroy unsaved work (cmd+w, cmd+s) or that the target app must be focused first.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, core action front-loaded, examples and the repeat parameter explanation placed after. No filler or repetition of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter tool with no output schema, the description covers the action and both parameters adequately. However, for a destructive-annotated tool it lacks any caveat about irreversibility or the precondition that a target app be frontmost (a sibling, activate_app, exists for exactly that).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it largely does: it gives the accepted combo syntax by example and explains that "repeat presses it several times". It omits the repeat bounds (1-50), which the schema does carry via minimum/maximum.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: press a key or combo, scoped to the frontmost app, with concrete examples ("cmd+s", "esc"). It clearly differs from click/drag/scroll siblings, but does not distinguish itself from "type" or "menu", which an agent could easily conflate with keyboard input.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied by the shortcut examples ("cmd+s", "cmd+shift+t"), which suggest keyboard-shortcut invocation. There is no explicit when-to-use statement and no mention of the adjacent tools (type, menu, activate_app) that an agent might pick instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshotARead-onlyIdempotent
Capture as PNG: the main screen, a region (x, y, w, h), or one app's front window (app), even if other windows cover it. Image pixels equal screen points; for a window, add the reported origin to get screen coordinates for click.
| Name | Required | Description | Default |
|---|---|---|---|
| h | No | ||
| w | No | ||
| x | No | ||
| y | No | ||
| app | No | Process name, e.g. "TextEdit". Defaults to the frontmost app. Works on background apps. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish the safety profile (readOnly, idempotent, non-destructive, closed-world), so the bar is lower. The description adds real context beyond them: the output is PNG, captured windows are included even when occluded, and image pixels map 1:1 to screen points, with the offset rule for window captures. It omits anything about screen-recording permissions or where the image is returned, keeping it short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences with no filler. The capture modes come first and the coordinate-mapping caveat that an agent most needs before clicking comes second — well front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description is responsible for the return contract and does explain the image format and coordinate mapping. The one remaining gap is how the captured image is delivered (path vs inline data), which an agent would want to know, but everything needed to invoke it correctly is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is only 20% (just 'app'), so the description carries most of the semantic load. It names the region group '(x, y, w, h)', implies omission selects the main screen, and explains the app parameter's role as the front window target, which meaningfully compensates for the bare schema entries.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (Capture) plus resource (PNG) and enumerates the three capture scopes: main screen, region (x,y,w,h), or an app's front window. An agent immediately knows this produces an image and which modes exist, distinguishing it from the interaction siblings like click, drag, and type.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No alternative screenshot tool exists to route against, but the text does give useful workflow context: 'even if other windows cover it' and the instruction to add the reported origin to get screen coordinates for click. That is implied usage guidance rather than an explicit when/when-not statement.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrollB
Scroll with the mouse wheel, at (x, y) if given, else wherever the cursor is.
| Name | Required | Description | Default |
|---|---|---|---|
| x | No | ||
| y | No | ||
| amount | No | Lines to scroll | |
| direction | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare non-read-only, non-destructive, non-idempotent and closed-world, so the safety profile is covered. The description adds the useful default of scrolling 'wherever the cursor is' when x/y are omitted, but says nothing about whether scrolling animates, blocks, or how it interacts with focused windows.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with the core action first and the conditional coordinate rule second; nothing is wasted. It is terse to the point of omitting useful parameter detail, but structurally sound.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple UI action with no output schema, the description covers the essential mechanics adequately. It still leaves the meaning of direction and amount, and any interaction with the active window, to inference from the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 25% (only 'amount' is described), so the description must compensate, and it does explain the x/y targeting fallback. However it never mentions 'amount' (lines to scroll) or clarifies that direction accepts up/down/left/right, leaving half the semantics implicit.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and device ('Scroll with the mouse wheel') plus the coordinate targeting rule, which is more than the bare name. It doesn't explicitly distinguish itself from siblings like drag or click, but the action is unambiguous on its own.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No when-to-use guidance, no prerequisites, and no reference to alternatives such as drag (which might also move content). The only contextual hint is the cursor-position default, which describes mechanics rather than when to invoke the tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
set_valueADestructiveIdempotent
Set a text field's value directly through accessibility. Works in background apps and with any language, without touching the keyboard or clipboard.
| Name | Required | Description | Default |
|---|---|---|---|
| id | No | Element id from inspect, e.g. "1.3.2" | |
| app | No | Process name, e.g. "TextEdit". Defaults to the frontmost app. Works on background apps. | |
| role | No | Role to match, e.g. "AXButton". Alone, it must match exactly one element. | |
| label | No | Element name/description/value to match (used when id is not given) | |
| value | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover the safety profile (readOnly=false, destructive=true, idempotent=true). The description adds non-obvious behavioral context: works on background apps without keyboard or clipboard focus, and is language-agnostic. It could say more about the destructive overwrite nature, but the annotation plus this context is solid.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single tight sentence front-loading the core action, followed by the key differentiator. No waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-param, 1-required mutation tool with no output schema and annotations carrying the safety profile, the description conveys the essential differentiator and behavioral context. Minor gaps remain on element-targeting precedence, but nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 80%, so the schema already documents id, app, role, and label. The description adds no parameter-level meaning (no note on which of id/label/role takes precedence), so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: set a text field's value via accessibility. Clearly distinguishes from sibling 'type' by emphasizing direct accessibility-setting without keyboard/clipboard. Scope is a bit narrow ('text field') given the schema allows role/label matching of arbitrary elements.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage context (background apps, no keyboard/clipboard needed), which hints when to prefer it over type/key. But it never explicitly names alternatives or states when NOT to use it, nor the relationship with the 'inspect' tool to obtain ids.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
typeADestructive
Type text into the focused element of the frontmost app (via paste, so Korean and other IME text work; the clipboard, images included, is restored). Prefer set_value for plain text fields.
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare destructiveHint=true and readOnlyHint=false, so the mutation profile is covered. The description adds genuinely useful behavior beyond that: input goes through paste for IME (Korean) support, and the clipboard including images is restored afterward. It still doesn't say whether typing appends to or replaces existing field content, which matters for a destructive tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
One lead sentence with the action front-loaded, a parenthetical for mechanism details, and a short routing sentence. Zero wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With annotations covering the safety profile and no output schema to explain, the description covers mechanism, IME handling, clipboard side effects, and sibling routing. The remaining gap is focus/replacement semantics and what happens if no element is focused.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There is one required parameter (text) with 0% schema description coverage, so the description carries the burden. "Type text" makes the parameter's meaning inferable, but it adds nothing about newlines, special characters, or length handling beyond the schema's minLength: 1.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (type) and resource (focused element of the frontmost app), and the parenthetical explains the mechanism. It explicitly names the sibling it is not — set_value — so an agent can route between the two without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"Prefer set_value for plain text fields" gives a clear alternative and the condition that selects it, which is strong guidance. It stops short of full 5 because it doesn't address when typing is preferable to key, click, or other input siblings, nor any when-not conditions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
wait_forARead-onlyIdempotent
Wait until an element matching find (role or label, like inspect find) appears in the app, or disappears with gone=true. Use after actions that open dialogs, load pages, or show spinners.
| Name | Required | Description | Default |
|---|---|---|---|
| app | No | Process name, e.g. "TextEdit". Defaults to the frontmost app. Works on background apps. | |
| find | Yes | ||
| gone | No | ||
| timeout | No | Seconds |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/non-destructive, so the safety profile is covered. The description adds behavior annotations don't convey: the wait can be inverted with gone=true to wait for disappearance, and it is intended as a post-action synchronization primitive. It doesn't state what happens on timeout expiry, which would have pushed this higher.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero filler, front-loaded with the core behavior and the find reference before the usage cue. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description arguably should say what a successful wait yields (e.g., the matched element) and what timeout expiry means. It adequately covers the matching semantics and usage moment for a 4-param read-only tool whose annotations carry the safety story.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is only 50% (find and gone are undocumented in the schema), but the description compensates by defining find as a role-or-label matcher and gone as the disappearance switch. app and timeout are already explained in the schema, so no further detail is needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb ('wait until') plus the resource and matching semantics ('element matching find (role or label)'), and even names the sibling inspect as the syntax reference for find. An agent can distinguish this from click_element or inspect without reading any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives clear triggering context: 'Use after actions that open dialogs, load pages, or show spinners.' It says when to reach for this tool but names no alternative or exclusion (e.g., when not to wait, vs. polling with inspect).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
12 tool updates
v0.4.1- First observed
activate_app - First observed
click - First observed
click_element - First observed
drag - First observed
inspect - First observed
key - First observed
menu - First observed
screenshot - First observed
scroll - First observed
set_value - First observed
type - First observed
wait_for
TDQS
Scored across 12 tools
Most tools target clearly distinct actions (drag, key, scroll, inspect, screenshot, wait_for), and the descriptions actively steer between the overlapping pairs: type vs set_value (frontmost/paste vs background/accessibility) and click vs click_element (raw coordinates vs accessibility target). These pairs remain somewhat confusable at a glance, but the guidance resolves them, so only minor ambiguity remains.
Everything is snake_case, but the convention is mixed: some names are verbs (drag, type, click, scroll, inspect) while others are bare nouns (menu, key, screenshot) or noun-verbs (click_element, set_value). It stays readable, but there is no single predictable verb_noun pattern.
12 tools is well-scoped for a desktop GUI automation surface, with each tool covering a distinct primitive (input, navigation, inspection, capture) and none appearing redundant.
The set covers a broad automation lifecycle: launch/activate, inspect, click/type/key/set_value, scroll/drag, wait, and screenshot, with accessibility and coordinate fallbacks. Minor gaps exist (e.g. right-click/context menus, hover), but agents can work around them.
Maintenance
Related MCP Connectors
Use your Mac, Windows or Linux computer from ChatGPT, Claude or Codex: files, commands, documents.
Mac & Windows: let ChatGPT, Claude & Cursor use your email, calendar, iMessage, Teams, files.
Cloud Macs with Xcode for AI agents: run commands, build and test iOS apps, see and use the screen.
Run Apple Search Ads from Claude: performance, recommendations, keywords, budgets, launches.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables Claude to see and interact with any macOS application using natural language commands. Perfect for testing Mac applications, UI automation, and app development with AI assistance.33MIT
- AlicenseNot gradedqualityDmaintenanceEnables high-speed native macOS automation for Claude by interacting directly with the Accessibility API, AppleScript, and UI trees instead of using screenshots. It allows users to read app states, click elements, and type text semantically across any macOS application.3 npm12MIT
- AlicenseAqualityCmaintenanceEnables Claude to inspect and drive native macOS app UIs during development via an in-process view tree and screenshot renderer, without requiring screen recording permission.7MIT
- AlicenseNot gradedqualityBmaintenanceEnables controlling macOS apps in the background, letting Claude Code or Codex act on native app UI elements without foregrounding them.1MIT