Skip to main content
Glama
Sign In
claude-background-control

claude-background-control

by lhwlhw0829-cmd
OverviewSchemaSourceRelated ServersDiscussionsAdmin
TypeScript
Local

claude-background-control

An MCP server that lets Claude Code use your Mac's GUI: read what's on screen, click things, and type. It uses only built-in macOS tools (osascript, screencapture, sips), so there are no native npm dependencies to break on install.

It goes through the accessibility tree first. That way it can press buttons and fill in text fields in background apps without moving your mouse or taking focus away from what you're doing. Real mouse and keyboard events are only the fallback.

Install

claude mcp add computer-control -- npx -y claude-background-control

Requires macOS and Node 20+ (development needs 22.18+ to run the TypeScript tests directly).

Related MCP server: Desktop Pilot MCP

Permissions (first run)

macOS has to allow the app that runs Claude Code (Terminal, iTerm, the Claude app, …):

  1. System Settings → Privacy & Security → Accessibility: turn it on for that app. Every tool except screenshot needs this.

  2. System Settings → Privacy & Security → Screen & System Audio Recording: turn it on for that app. Needed for screenshot.

  3. Restart that app.

If a permission is missing, the error names the exact app and path macOS expects. That isn't always the app you see. Claude Code inside the Claude desktop app, for example, needs the grant on ~/Library/Application Support/Claude/claude-code/<version>/claude.app, which you add with + and Cmd+Shift+G. Because that path includes the version number, a Claude Code update can require granting it again.

Tools

Tool

What it does

inspect(app?, find?)

Lists UI elements of the app's windows with an id, role, label, value, and frame. Start here. find returns only matching elements (by role or label).

click_element(id | label | role, app?)

Presses a button or focuses a field via accessibility, with no mouse movement. Falls back to a real click.

set_value(id | label | role, value, app?)

Sets a text field's value directly. Works in background apps and with any language.

type(text)

Types into the focused field by pasting, so Korean and other IME input comes through correctly. The whole clipboard (images, files, rich text) is restored afterwards.

key(combo, repeat?)

Sends key codes like cmd+s or return, so shortcuts work even when a Korean input source is active.

wait_for(find, app?, gone?, timeout?)

Waits until a matching element appears, or disappears with gone. Use after actions that open dialogs or load content.

click(x, y, button?, clicks?)

Real mouse click (CoreGraphics) for canvas or Electron content.

menu(path?, app?)

Walks the menu bar by item names, e.g. ["File", "Save…"]. A path that ends on a menu lists its items; one that ends on an item clicks it. Works on background apps.

scroll(direction, amount?, x?, y?)

Mouse-wheel scroll by lines, at a point or wherever the cursor is.

drag(x1, y1, x2, y2)

Left-button drag: select text, move things, resize.

screenshot(app? | x, y, w, h)

PNG of the main screen, a region, or one app's front window, even if other windows cover it. Image pixels equal screen points. For a window, add the reported origin to get click coordinates.

activate_app(name)

Launches an app or brings it to the front.

app is the process name (e.g. "TextEdit"). If you leave it out, the frontmost app is used. When a label or role matches in several windows, the frontmost window wins. If it's still ambiguous, you get a list of ids to choose from.

Develop

npm install
npm test        # pure-logic unit tests
npm run build

After changing src/mac.ts, run the manual GUI checklist in scripts/smoke-test.md.

Limits

  • Main display only.

  • inspect doesn't descend into leaf-like roles such as buttons and static text.

  • With Stage Manager on, a background app's window is only a side-strip thumbnail, so screenshot(app) warns and suggests activate_app first.

Design notes: docs/superpowers/specs/2026-09-30-computer-control-mcp-design.md

Available Tools

12 tools
activate_appA
Idempotent

Launch or bring an app to the front (needed before type/key, which go to the frontmost app).

Inputs: name (required).

clickA
Destructive

Real mouse click at screen coordinates (points). Fallback for content without accessibility info.

Inputs: x (required), y (required), button, clicks.

click_elementA
Destructive

Press a button/link or focus a text field by id or label. Uses accessibility actions (no mouse movement) and falls back to a real click at the element center.

Inputs: id, app, role, label.

dragB
Destructive

Drag with the left mouse button from (x1, y1) to (x2, y2): select text, move items, resize.

Inputs: x1 (required), x2 (required), y1 (required), y2 (required).

inspectA
Read-onlyIdempotent

List UI elements (id, role, label, value, frame) of an app's windows via the accessibility tree. The preferred way to find what to click. Pass find to get only matching elements instead of the whole tree.

Inputs: app, find, maxDepth, maxNodes.

keyA
Destructive

Press a key or combo in the frontmost app, e.g. "return", "cmd+s", "cmd+shift+t", "esc", "down". repeat presses it several times.

Inputs: combo (required), repeat.

menuA
Destructive

Use an app's menu bar by item names, e.g. path ["File", "Save…"]. A path ending on a menu (or empty) lists its items; ending on an item clicks it. Works on background apps. Names are localized ("..." matches "…").

Inputs: app, path.

screenshotA
Read-onlyIdempotent

Capture as PNG: the main screen, a region (x, y, w, h), or one app's front window (app), even if other windows cover it. Image pixels equal screen points; for a window, add the reported origin to get screen coordinates for click.

Inputs: h, w, x, y, +1 more.

scrollB

Scroll with the mouse wheel, at (x, y) if given, else wherever the cursor is.

Inputs: direction (required), x, y, amount.

set_valueA
DestructiveIdempotent

Set a text field's value directly through accessibility. Works in background apps and with any language, without touching the keyboard or clipboard.

Inputs: value (required), id, app, role, +1 more.

typeA
Destructive

Type text into the focused element of the frontmost app (via paste, so Korean and other IME text work; the clipboard, images included, is restored). Prefer set_value for plain text fields.

Inputs: text (required).

wait_forA
Read-onlyIdempotent

Wait until an element matching find (role or label, like inspect find) appears in the app, or disappears with gone=true. Use after actions that open dialogs, load pages, or show spinners.

Inputs: find (required), app, gone, timeout.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Deploy Server
    One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

    TDQS

    A3.7/5.0

    Scored across 12 tools

    Disambiguation4/5

    Most tools target clearly distinct actions (drag, key, scroll, inspect, screenshot, wait_for), and the descriptions actively steer between the overlapping pairs: type vs set_value (frontmost/paste vs background/accessibility) and click vs click_element (raw coordinates vs accessibility target). These pairs remain somewhat confusable at a glance, but the guidance resolves them, so only minor ambiguity remains.

    Naming Consistency3/5

    Everything is snake_case, but the convention is mixed: some names are verbs (drag, type, click, scroll, inspect) while others are bare nouns (menu, key, screenshot) or noun-verbs (click_element, set_value). It stays readable, but there is no single predictable verb_noun pattern.

    Tool Count5/5

    12 tools is well-scoped for a desktop GUI automation surface, with each tool covering a distinct primitive (input, navigation, inspection, capture) and none appearing redundant.

    Completeness4/5

    The set covers a broad automation lifecycle: launch/activate, inspect, click/type/key/set_value, scroll/drag, wait, and screenshot, with accessibility and coordinate fallbacks. Minor gaps exist (e.g. right-click/context menus, hover), but agents can work around them.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Resources

    Related MCP Connectors

    Related MCP Servers

    • Desktop Pilot MCP

      VersoXBT
      A
      license
      Not graded
      quality
      D
      maintenance
      Enables high-speed native macOS automation for Claude by interacting directly with the Accessibility API, AppleScript, and UI trees instead of using screenshots. It allows users to read app states, click elements, and type text semantically across any macOS application.
      12 npm
      12
      MIT

    • macos-ui-mcp

      thangduonghuu
      A
      license
      A
      quality
      C
      maintenance
      Enables Claude to inspect and drive native macOS app UIs during development via an in-process view tree and screenshot renderer, without requiring screen recording permission.
      7
      MIT

    • cua-jev

      mimo-3
      A
      license
      Not graded
      quality
      B
      maintenance
      Enables controlling macOS apps in the background, letting Claude Code or Codex act on native app UI elements without foregrounding them.
      1
      MIT